Using Object Storage for Uploads (S3 & Clones)

It’d be great if you could add Wasabi as well…

1 Like

I used wasabi for backups for a while. As far as configuration goes, it “just worked”, so you could give it a try if you want.

But with some frequency the backups failed silently and backups stayed on the local machine, filling up the disk. I looked at wasabi and Discourse errors and never found an explanation that would make it possible for either end to “fix” anything.

I don’t recommend it, as I’m not sure that it’s “great.”

2 Likes

Thanks!
I will check how it goes this time.
The default backup frequency is 7 days between backups and up to 5 backups.
I’ll share how it goes.

1 Like

I was doing daily backups; I don’t remember how many I was trying to keep, but the local hard drive had room for only a couple.

2 Likes

Which storage would you personally recommend?
I don’t think AWS and Azure are affordable for personal projects.
Not sure about Azure, but AWS looks confusing and unpredictable…

1 Like

I hoped that wasabi would be good for backups at least. Backblaze s3 is affordable. I don’t think I’ve used it for uploads, but it works for backups (I think). I think the only issue with backblaze is that (when I last tested it) you had to use a global key (so I couldn’t use it for clients who could see the key). I think someone recently posted a fix for that (something about “legacy” something, somewhere). For a personal project, that’s what I’d try next, I think. (And if you’re on Digital Ocean, spaces is OK, I think.)

3 Likes

I’m on DO, but cost-wise Blackblaze/Wasabi is cheaper.
What did you use for uploads?

1 Like

Sorry, I can’t get.
Why do we have to specify settings in app.yml, as we can enter this information within from Discourse->Settings ?

1 Like

Because of the way that it handles building assets when the container is built, I think. It is quite confusing that the behavior is different when the settings are in the environment variables and the database, but that’s the way it works. It’s also a better way too handle them, as it means you can build and restore a new site from the command line.

3 Likes

Where should I set this? I’m not finding it in Discourse admin settings?

1 Like

It’s set in the example configuration block for Vultr in the OP. Copy and paste into your app.yml and adjust the necessary fields.

2 Likes

Thanks, I missed this bit:

1 Like

This isn’t going smoothly for me, sorry to reply so many times.

If I do the above, should I ignore the admin panel settings for s3 and backups?

Or should admin panel settings be configured as well?

1 Like

You just need to follow this guide here in the OP and you will get a functional object storage configuration.

3 Likes

Setting those variables makes them unavailable from the UX. You must set them as described here rather than the ux.

2 Likes

I think it’s (mostly?) working now, but is this content security policy script src safe?

I’m using AWS for two S3 containers (for uploads & backups), and two CloudFront CDNs (for files & assets). When I use my own CNAMEs for the CloudFront CDNs, I get a bunch of script-src network errors in my browser when loading Discourse. No more errors after adding those entires to my CSP.

1 Like

Are those the urls you put in the env variables described in the OP? And are they https?

1 Like

Yes, I don’t have script-src warnings when I put the two d23whatever.cloudfront.net URLs in the env variables. When I put my custom URLs, i.e. community-cdn.mydomain and files-cdn.mydomain, in the env variables, that’s the time I get these script-src warnings. And apparently the stripe js is still giving me this warning even though it’s in my content security policy script src.

2 Likes

I set up S3 Uploads and object storage as outlined here in the OP, but without a CDN.

For the DISCOURSE_S3_CDN_URL variable, I have this:
https://my-bucket-uploads.s3.dualstack.us-west-2.amazonaws.com

All seems fine, including backups, however, in the console this error shows up when a reply to a post is started:

The request url in the error is actually a string of two urls which seems like the cause?

https://mydiscourse.com/t/uploads-test-for-s3/79/https://my-bucket-uploads.s3.dualstack.us-west-2.amazonaws.com/assets/markdown-it-bundle-a7328b73d3e7b030770eab70f10bdb0af655b3d8fa929bc49f1ad04c4cdaa198.br.js

1 Like

A CDN is mandatory for it to work correctly.

4 Likes