Using Object Storage for Uploads (S3 & Clones)

Thanks!
I will check how it goes this time.
The default backup frequency is 7 days between backups and up to 5 backups.
I’ll share how it goes.

1 Like

I was doing daily backups; I don’t remember how many I was trying to keep, but the local hard drive had room for only a couple.

2 Likes

Which storage would you personally recommend?
I don’t think AWS and Azure are affordable for personal projects.
Not sure about Azure, but AWS looks confusing and unpredictable…

1 Like

I hoped that wasabi would be good for backups at least. Backblaze s3 is affordable. I don’t think I’ve used it for uploads, but it works for backups (I think). I think the only issue with backblaze is that (when I last tested it) you had to use a global key (so I couldn’t use it for clients who could see the key). I think someone recently posted a fix for that (something about “legacy” something, somewhere). For a personal project, that’s what I’d try next, I think. (And if you’re on Digital Ocean, spaces is OK, I think.)

3 Likes

I’m on DO, but cost-wise Blackblaze/Wasabi is cheaper.
What did you use for uploads?

1 Like

Sorry, I can’t get.
Why do we have to specify settings in app.yml, as we can enter this information within from Discourse->Settings ?

1 Like

Because of the way that it handles building assets when the container is built, I think. It is quite confusing that the behavior is different when the settings are in the environment variables and the database, but that’s the way it works. It’s also a better way too handle them, as it means you can build and restore a new site from the command line.

3 Likes

Where should I set this? I’m not finding it in Discourse admin settings?

1 Like

It’s set in the example configuration block for Vultr in the OP. Copy and paste into your app.yml and adjust the necessary fields.

2 Likes

Thanks, I missed this bit:

1 Like

This isn’t going smoothly for me, sorry to reply so many times.

If I do the above, should I ignore the admin panel settings for s3 and backups?

Or should admin panel settings be configured as well?

1 Like

You just need to follow this guide here in the OP and you will get a functional object storage configuration.

3 Likes

Setting those variables makes them unavailable from the UX. You must set them as described here rather than the ux.

2 Likes

I think it’s (mostly?) working now, but is this content security policy script src safe?

I’m using AWS for two S3 containers (for uploads & backups), and two CloudFront CDNs (for files & assets). When I use my own CNAMEs for the CloudFront CDNs, I get a bunch of script-src network errors in my browser when loading Discourse. No more errors after adding those entires to my CSP.

1 Like

Are those the urls you put in the env variables described in the OP? And are they https?

1 Like

Yes, I don’t have script-src warnings when I put the two d23whatever.cloudfront.net URLs in the env variables. When I put my custom URLs, i.e. community-cdn.mydomain and files-cdn.mydomain, in the env variables, that’s the time I get these script-src warnings. And apparently the stripe js is still giving me this warning even though it’s in my content security policy script src.

2 Likes

I set up S3 Uploads and object storage as outlined here in the OP, but without a CDN.

For the DISCOURSE_S3_CDN_URL variable, I have this:
https://my-bucket-uploads.s3.dualstack.us-west-2.amazonaws.com

All seems fine, including backups, however, in the console this error shows up when a reply to a post is started:

The request url in the error is actually a string of two urls which seems like the cause?

https://mydiscourse.com/t/uploads-test-for-s3/79/https://my-bucket-uploads.s3.dualstack.us-west-2.amazonaws.com/assets/markdown-it-bundle-a7328b73d3e7b030770eab70f10bdb0af655b3d8fa929bc49f1ad04c4cdaa198.br.js

2 Likes

A CDN is mandatory for it to work correctly.

4 Likes

I’m also in this situation with an object store configured (minio) but no CDN. Is it a use case that could be supported ?

From what I’m seeing so far in my tests there is only the markdown-it-bundle js file that is having issues as its pointing to the wrong URL - DISCOURSE_HOSTNAME/DISCOURSE_S3_CDN_URL/assets/markdown-it-bundle-HASH.br.js

It actually looks lit a bug for this one, if I set DISCOURSE_CDN_URL variable, it still points to the wrong URL in this form DISCOURSE_HOSTNAME/DISCOURSE_CDN_URL/assets/markdown-it-bundle-HASH.br.js

it should point to DISCOURSE_S3_CDN_URL/assets/markdown-it-bundle-HASH.br.js

Other js assets are pointing to the right URL ’

I guess from what you are saying I will have other issues that I have not identified yet. Maybe you can give me more info on what could go wrong ?

If I undestand it well, js assets are on the object store, stylesheets should be on a CDN. WIthout a CDN could the stylesheets be delivered by the app as usual ? (from what i’m seeing its the case)

Thanks for your help

3 Likes

That is not a supported use case per the OP:

1 Like