Configure an S3 compatible object storage provider for uploads

Thanks for the report Richie. I also have had AWS S3 image storage running for several years and came to this post via the console message. But the description at the top doesn’t say anything about the case that you already had S3 and just need a CDN.

For the record here is what I did:

  1. Went to AWS console, under Network and Content Delivery picked Cloudfront
  2. Clicked the Create distribution button
  3. Filled out the fairly obvious form, the only thing you really need to do on it is to pick your AWS S3 bucket where the images are from the drop-down menu.
  4. Waited a bit for the Cloudfront configuration to finish…
  5. A <gibberish>.cloudfront.net domain showed up in the “Domain Name” column of the Cloudfront Distributions list.
  6. I copied and pasted that domain into the s3 cdn url field in my site admin File settings.
  7. I did some tests:
    a. I made a new post with an image upload and indeed it was on cloudfront.
    b. I hit Rebuild HTML on some random existing image posts and saw they also rebuilt with cloudfront.net images.
  8. Since all looked good I went in and ran a rebake, which took several hours as I have around half a million posts now:
./launcher enter app
# rake posts:rebake
  1. All seems to be working fine. It put a ton of jobs in the sidekiq queue, one per post it looks like, which are going to take a few days to clear but it is chunking though them now.
16 Likes

Are you sure that is the case? This site here uses assets from CDN and we didn’t have to purge the cache. It’s also a EmberCli change that shouldn’t affect production :thinking:

5 Likes

Oh those damn optimizers. I would tell you to disable that if possible as Discourse already ships optimal configurations for each asset. Those optimizers are great when you are hosting a black box web software from the 2000’s, but they fail hard on modern stuff. Also even the big name optimizer from Cloudflare often breaks Discourse, so I have no hopes for others. They may work one day, and break the next and all your visitors are left with a blank page. All that for zero benefits.

5 Likes

Any chance you enabled secure_uploads in the site settings?

Also looks like this was reported and fixed today because of Discourse compatibility :

5 Likes

Is there a way to disable this warning from showing up on my admin dashboard?

The server is configured to upload files to S3, but there is no S3 CDN configured.

I had issues setting up an S3 CDN, but it doesn’t cost me an arm and a leg so I am fine with just using S3 directly. But what I’d love to see is for this notification to go away because I am fully aware of the consequences.

1 Like

Hi.

I just want give update. We will able setup backup using GCS now. I post on other thread too. I hope it will help other people who painful search this solution.

How to do that?
Enable default config for backup(or you can setting form admin dashboard).

DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
DISCOURSE_BACKUP_LOCATION: s3

Then set bucket permission as Storage Legacy Object Owner:

  1. Go to your project in Google Cloud Console
  2. Select Storage
  3. Select your bucket
  4. Go to the permissions tab
  5. Add new permission, fill your service account email with your account. for roles, select Storage Legacy Object Owner
  6. Save and done.

Sorry for double post, I just want share this good update.
Thank you

5 Likes

It’d be great if you could add Wasabi as well…

1 Like

I used wasabi for backups for a while. As far as configuration goes, it “just worked”, so you could give it a try if you want.

But with some frequency the backups failed silently and backups stayed on the local machine, filling up the disk. I looked at wasabi and Discourse errors and never found an explanation that would make it possible for either end to “fix” anything.

I don’t recommend it, as I’m not sure that it’s “great.”

2 Likes

Thanks!
I will check how it goes this time.
The default backup frequency is 7 days between backups and up to 5 backups.
I’ll share how it goes.

1 Like

I was doing daily backups; I don’t remember how many I was trying to keep, but the local hard drive had room for only a couple.

2 Likes

Which storage would you personally recommend?
I don’t think AWS and Azure are affordable for personal projects.
Not sure about Azure, but AWS looks confusing and unpredictable…

1 Like

I hoped that wasabi would be good for backups at least. Backblaze s3 is affordable. I don’t think I’ve used it for uploads, but it works for backups (I think). I think the only issue with backblaze is that (when I last tested it) you had to use a global key (so I couldn’t use it for clients who could see the key). I think someone recently posted a fix for that (something about “legacy” something, somewhere). For a personal project, that’s what I’d try next, I think. (And if you’re on Digital Ocean, spaces is OK, I think.)

3 Likes

I’m on DO, but cost-wise Blackblaze/Wasabi is cheaper.
What did you use for uploads?

1 Like

Sorry, I can’t get.
Why do we have to specify settings in app.yml, as we can enter this information within from Discourse->Settings ?

1 Like

Because of the way that it handles building assets when the container is built, I think. It is quite confusing that the behavior is different when the settings are in the environment variables and the database, but that’s the way it works. It’s also a better way too handle them, as it means you can build and restore a new site from the command line.

3 Likes

Where should I set this? I’m not finding it in Discourse admin settings?

1 Like

It’s set in the example configuration block for Vultr in the OP. Copy and paste into your app.yml and adjust the necessary fields.

2 Likes

Thanks, I missed this bit:

1 Like

This isn’t going smoothly for me, sorry to reply so many times.

If I do the above, should I ignore the admin panel settings for s3 and backups?

Or should admin panel settings be configured as well?

1 Like

You just need to follow this guide here in the OP and you will get a functional object storage configuration.

3 Likes