Configure an S3 compatible object storage provider for uploads

[V1ktor (Viktor N.)]

I wanted to add one thing for Backblaze, as I just set this up and this might save others some time:

Master application key is not compatible with S3 API. You must create new application key (source).

And I wanted to ask 3 questions, to clarify some things:

1. Is it normal to have a lot of missing .map files? They all seem to be from brotli_asset folder. They are neither on the server nor object storage.
2. I’ve seen reports that DISCOURSE_S3_BUCKET was deprecated and DISCOURSE_S3_UPLOADS_BUCKET should be used. Which is the correct one?
3. Is it necessary to add DISCOURSE_ENABLE_S3_UPLOADS: true? I’ve seen this mentioned in other topics.

Thanks.

[Falco]

@V1ktor

Is it normal to have a lot of missing .map files? They all seem to be from brotli_asset folder. They are neither on the server nor object storage.

Yes, that is a know bug in our asset pipeline that will be solved by the ongoing ember-cli migration.

@V1ktor

I’ve seen reports that DISCOURSE_S3_BUCKET was deprecated and DISCOURSE_S3_UPLOADS_BUCKET should be used. Which is the correct one?

Warning is correct, gotta update the wiki guide here in the OP.

@V1ktor

Is it necessary to add DISCOURSE_ENABLE_S3_UPLOADS: true ? I’ve seen this mentioned in other topics.

Not mandatory at the moment, because I’m pretty sure it gets overwritten by the USE_S3 ENV, but I would have to dig into the codebase for a definite answer on that.

[pfaffman (Jay Pfaffman)]

@Falco

Not mandatory at the moment,

I’m working on a multisite instance where I tried to restore a database from another instance and when I restored database-only the main page rendered json that said that it was required. But that’s lkely an edge case. I was always confused by the DISCOURSE_S3_BUCKET env variable…

[cookieman768]

@cookieman768

Hey all, I’m pretty new to S3, so I’m not entirely sure how to phrase this, but I’ll try my best. So, I just switched to using S3 for uploads and backups and I have been using Discourse Connect in order to allow for logins on other parts of my site, but now profile images don’t work. I believe this has to do with CORS policies, but I’m not sure where I could configure it. I would ideally want to whitelist it for forum.domain.tld and domain.tld - or a wildcard on all subdomains would work too. Is this something I would set in Discourse, or where exactly? I’m using Vultr object storage if that makes a difference.

I’m still curious about this if anyone has any insight, also I just had another question come up.

If I were wanting to change the domain of my Discourse installation, how would that impact Object Storage access policies? Would I need to change rules, or would that be taken care of for me by Discourse?

[TheDarkWizard]

@Falco

Have you seen this? https://blog.cloudflare.com/introducing-r2-object-storage/

I’ve already signed up for a test, looking forward to testing it.

[pfaffman (Jay Pfaffman)]

I don’t know anyone who’s seen it.

I signed up for that test long ago, back in October. It doesn’t seem to be an actual product.

[TheDarkWizard]

Interesting.

I got an email about it roughly 2 weeks ago in regards about signing up for the test, it’s the only reason why I learned about it. I don’t follow the cloudflare blog. Hopefully it doesn’t get shoved into the back like the railgun though Argo is just so much better.

[pfaffman (Jay Pfaffman)]

@TheDarkWizard

I got an email about it roughly 2 weeks ago in regards about signing up for the test,

Maybe I missed a memo! Or maybe I got it and just don’t remember. (The most recent email I see with r2 in it says “Thank you for your interest in the Cloudflare R2 Storage early access program. We will be in touch regarding your request.” last November…)

[TheDarkWizard]

Fair enough. I just went back to the blog, and also noticed the date for last September. Considering how recent the email asking me to sign up (I did do so), I thought it was a more recent product announcement/offering. I don’t really keep up with cloudflare news, to be honest. Heres to hoping it marteralizes and that it’s actually good, their scope and pitch about the fees could be really enticing.

[Falco]

I signed up as soon as it was announced and I’m on the same boat

[TheDarkWizard]

I was so excited that I thought I brought something to Meta that people had barely heard about yet xD

[pfaffman (Jay Pfaffman)]

There does exist documentation: https://developers.cloudflare.com/r2/get-started/

It looks very unpleasant to configure.

[TheDarkWizard]

I guess they misunderstood what they meant when they said that it was supposed to be an easy drop in replacement for amazon s3 and other s3 compatible services.

[itsbhanusharma (Bhanu Sharma)]

Given what they did with tunnels, I guess they’ll introduce a GUI once the product gets enough traction. Let’s wait & see how it grows.

[pfaffman (Jay Pfaffman)]

@Falco

after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

Could I submit a PR for a new template called something like web.upload-assets.template.yml that includes that stanza so that then people could just include it? It’s a bit fussy to paste it in to the yml.

And if that’s OK, could I also submit a PR including it commented out standalone.yml and web_only.yml so that it can easily be included?

[luizjr]

I’m having difficulties configuring Oracle’s object storage.
I didn’t see any guides here.

[Falco]

There is no Oracle Cloud guide as it’s not that popular option. If you are experimenting with it, try starting with the DigitalOcean settings and going from there.

When you get it to work you can update the wiki in the OP with the guide.

[luizjr]

I did the configuration and it didn’t work, besides that I didn’t understand the logic of the CDN

[Falco]

Walking off the beaten path of already tested services is not for the faint of the heart.

I strongly recommend using AWS S3, which is what we use internally.

[pfaffman (Jay Pfaffman)]

@luizjr

I didn’t understand the logic of the CDN

Then you likely didn’t do the configuration correctly. If you included the details of your setup (minus the keys) we could offer help, and test that the CDNs were correct. Also, an easy thing to overlook is the bit at the top about pushing assets to S3. Which is also hard to understand.