Configure an S3 compatible object storage provider for uploads

Thank you, also I would to add that you can use any of these tools to copy from cloud to cloud, especially to/from S3 compatible object storage, for example, Rclone, Shargate, Gs Richcopy360 and GoodSync. all of these are compatible with similar clouds

1 Like

We have just discovered a problem, Cloudflare R2 doesn’t allow public-read from the S3 endpoint URL, rather only the custom domain or a random r2.dev domain.
(Pre-signed downloads work, just no direct public access is supported.)
But discourse only uses the CDN URL for embedded images, and not direct downloads, which use the S3 endpoint URL.
Is there a way to make it use the CDN URL for all files, or force the use of a presigned url?

Related:

The workaround mentioned in that post works, add ?dl=1 fixes it, because its forces discourse to use a presigned S3 URL.

1 Like

Fixed in 2023-03-16 now R2 working with Discourse like charm with free plan

3 Likes

I also see this with some frequency (every several months) even though my Discourse is running in AWS Lightsail and I’m uploading to AWS S3. So I’m not sure it’s wasabi’s fault.

Would it be possible to catch this error and alert the admin? I do check the disk space and remove old backups when I upgrade but sometimes that’s too late and the forum goes down for no disk space.

1 Like

I am fairly certain that the issue was that automatic OS reboots for security updates were happening while the backup was running. Make sure that you schedule your OS reboots and your backups at different times. It was after I’d moved that site from wasabi that I came up with this explanation, but I’m pretty sure that’s what it was.

2 Likes

uptime says it’s been up for 300 days so I don’t think that’s the problem. But along similar lines, I had Discourse backups scheduled at 2:00 am and Lightsail snapshots at 2:30 am, so maybe the upload sometimes isn’t complete and the snapshot messes with it. I’ve separated the two operations by an hour- we’ll see if it makes a difference.

Regardless I think it’s reasonable to warn admins if the upload fails, for whatever reason.

2 Likes

I think that it’s time that you do kernel upgrades and reboot. :slight_smile:

Could you be running out of ram?

After implementing remote Backblaze backups, I see this error in my dashboard:

The server is configured to upload files to S3, but there is no S3 CDN configured. This can lead to expensive S3 costs and slower site performance. See “Using Object Storage for Uploads” to learn more.

I didn’t configure uploading of files, I only configured backups via this config:

DISCOURSE_S3_REGION: "s3.us-west-00X"
DISCOURSE_S3_INSTALL_CORS_RULE: false
DISCOURSE_S3_ENDPOINT: https://s3.us-west-00X.backblazeb2.com
DISCOURSE_S3_ACCESS_KEY_ID: mykeyid
DISCOURSE_S3_SECRET_ACCESS_KEY: myaccesskey
DISCOURSE_S3_BUCKET: community-forum
DISCOURSE_S3_BACKUP_BUCKET: community-forum/backups
DISCOURSE_BACKUP_LOCATION: s3

Did I do something wrong?

Something seems misconfigured, I notice when I try uploading a file to a post, I received this error:

Unsupported value for canned acl 'public-read’

Any assistance would be appreciated.

1 Like

Remove this if you don’t want uploads to go to s3.

3 Likes

You saved the day brother. :+1:t3: Thanks so much!

2 Likes

Did that seem to work?

1 Like

It has happened once in the past month since I separated the two processes by an hour, so it didn’t “fix” it, and it doesn’t happen often enough to say whether it helped.

On the bright side, I noticed there is a backup status section on the admin page that shows available disk space, which saves me from constantly opening a terminal and doing a df just to check for stuck backups. I customized the text to remind myself that I expect around 80 GB free.

image

1 Like

That’s a good idea.

I saw the image before I read that you had customized the text and was wondering what logic was at play to determine that was “good”!

1 Like

I just couldn’t get this to work with Scaleway using the Bitnami Discourse image.
The env variables were set but clearly weren’t being read/applied correctly (or at all?).

So I’ve set the S3 variables in the admin panel and set the region directly in the rails console (still hoping that this just becomes a text field):
SiteSetting.s3_region="fr-par"

It gave me a validation error, but I just commented out the validation check before updating the setting, then put it in again.

1 Like

The Bitnami image isn’t packaged by us and don’t follow our recommendations. Everything documented here is only tested against the official install.

4 Likes

This has been solved by enabling “s3 use cdn url for all uploads”, an option recently added by discourse.
since we were using R2 before, we needed to use discourse remap to manually replace the broken links, and synced s3 files just in case, and then we rebaked all posts.