# Configure an S3 compatible object storage provider for uploads
**URL:** https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916
**Category:** Self-Hosting
**Tags:** cdn, configuring, how-to, reference
**Created:** [April 22, 2020, 10:37pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916 "2020-04-22T22:37:37Z")
**Posts on this page:** 20
**Page:** 9
### Author:  [@fearlessfrog](https://meta.discourse.org/u/fearlessfrog)
#### Post date: [December 14, 2022, 10:22pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/392 "2022-12-14T22:22:34Z")
Had to park this for now, as it looked like was going to work but then there is something odd going on with R2 in terms of content encoding with the assets either on uploading and not setting the header or something else. It’ll choke on a ‘Invalid or unexpected token’ given the gz asset of something like `browser-detect-7af298cd000a967d2bdc01b04807eda2924a388584ea38ad84919b726283c2ed.gz.js`. The `rake s3:upload_assets` seems to be working but the files aren’t being read correctly on the browser side.
I don’t really get why with AWS S3 it is fine using the local server URL for assets (they don’t exist on our existing S3 bucket for uploads) but for R2 use it wants to use DISCOURSE\_S3\_CDN\_URL for assets only. If I could force the assets to be from the server URL this would probably all work.
EDIT: Chatting on the CF, this seems to be the issue, and as of today why R2 can’t be used with Discourse without some changes. I could script something in the post hook step to remove the gz assets but I feel I’m already ‘off the path’ far enough for one day:
> Files that you gzip are not currently handled correctly by R2. You have to upload uncompressed files. Cloudflare has transparent compression, they pick identity, gzip, or Brotli based on what the client can handle. This is a difference from S3.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [December 15, 2022, 11:16am UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/393 "2022-12-15T11:16:53Z")
Nice work! And that’s a clear message from cloudflare about why it won’t work. Thanks very much. I’ll copy that into the OP soon.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [December 16, 2022, 3:05pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/394 "2022-12-16T15:05:27Z")
Thanks again! I updated the OP:
> [@Falco](#):
>
> Cloudflare’s offering is incompatible. In testing, @fearlessfrog filed a ticket with Cloudflare and in December 2022 they said:
>
> > [@](#):
> >
> > Files that you gzip are not currently handled correctly by R2. You have to upload uncompressed files. Cloudflare has transparent compression, they pick identity, gzip, or Brotli based on what the client can handle. This is a difference from S3.
---
### Author:  [@saulshanabrook](https://meta.discourse.org/u/saulshanabrook)
#### Post date: [December 23, 2022, 5:39pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/395 "2022-12-23T17:39:44Z")
Thank you for putting together this guide! I have had some success using Minio.
For anyone else who is trying to set it up locally with Docker Compose, you can tell Docker to add a hostname alias so that it works as a subdomain, like this:
```yaml
minio:
image: minio/minio
command: server --console-address :9001 /data
ports:
- "9000:9000"
- "9001:9001"
volumes:
- ./data/minio:/data
environment:
MINIO_DOMAIN: minio.mydomain.com
networks:
default:
aliases:
- assets.minio.mydomain.com
```
In this case, you would set `DISCOURSE_S3_ENDPOINT=http://minio.mydomain.com:9000`, `DISCOURSE_S3_CDN_URL=//assets.minio.mydomain.com:9000`, and set your local `/etc/hosts/` file to point the subdomain to localhost.
This works mostly fine, but I did notice that Discourse [is unable to download files from an address that doesn’t have port `80` or `443`](https://github.com/discourse/discourse/blob/505aec123fc04bf9a6e743e94820edd2b7c1722e/lib/final_destination.rb#L361), so uploading an image will work, but then when it attempts to download it to resize it, it will fail.
I was thinking it might be good to mention that in the Minio section or in summary, that the `DISCOURSE_S3_CDN_URL` must be on port 80 or 443.
---
### Author:  [@fearlessfrog](https://meta.discourse.org/u/fearlessfrog)
#### Post date: [December 28, 2022, 8:17pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/396 "2022-12-28T20:17:38Z")
> [@Falco](#):
>
> Only problem is that their CDN offering is [awfully broken](https://docs.digitalocean.com/products/spaces/how-to/set-file-metadata/), so you need to use a different CDN for the files.
Hey @Falco - Is this referring to the way the `Content-Encoding: gzip` header works with their Spaces CDN? That sounds similar to Cloudflare R2, in that the asset locations is made to be the same as the uploads CDN, so the gzip breaks? Here’s [what happens with R2 today](https://gist.github.com/csuhta/0001d1bb74200412bc1d7f9e11ec4ea5).
It might be worth considering a toggle for that behavior, i.e. serve assets from origin rather than always DISCOURSE\_S3\_CDN\_URL? I’ll happily go look to see how to do this, if it would be considered as a potential config change.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [December 28, 2022, 8:31pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/397 "2022-12-28T20:31:26Z")
> [@fearlessfrog](#):
>
> It might be worth considering a toggle for that behavior, i.e. serve assets from origin rather than always DISCOURSE\_S3\_CDN\_URL?
That’s what should happen if you omit configuring `DISCOURSE_S3_CDN_URL` but since it’s a weird corner case, and a potential expensive mistake, it’s not a common configuration.
---
### Author:  [@fearlessfrog](https://meta.discourse.org/u/fearlessfrog)
#### Post date: [December 28, 2022, 8:38pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/398 "2022-12-28T20:38:58Z")
Yep, I can understand that. A new GlobalSetting bool S3\_ORIGIN\_ASSETS (or S3\_BROKEN\_PROXY\_FUDGE 🙂) entry around about here, sort of like for how the [test scripts aren’t compressed](https://github.com/discourse/discourse/blob/main/app/helpers/application_helper.rb#L117) would allow Digital Ocean Spaces and Cloudflare R2 storage and CDN to work with Discourse out of the box though, which is a nice feature add for not much effort? Maybe for future consideration anyway. 😻
---
### Author:  [@fearlessfrog](https://meta.discourse.org/u/fearlessfrog)
#### Post date: [January 5, 2023, 3:20am UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/399 "2023-01-05T03:20:11Z")
Oh, I saw on the 3.0.beta release notes there’s something added. I’ll give it a go, unless I misunderstand what it’s for? It might allow Cloudflare R2 and Digital Ocean Spaces to be used with their CDNs doing that weird stuff with gzip.
[https://github.com/discourse/discourse/pull/19284](https://github.com/discourse/discourse/pull/19284)
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [January 5, 2023, 5:12am UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/400 "2023-01-05T05:12:13Z")
No, that’s unrelated.
---
### Author:  [@fearlessfrog](https://meta.discourse.org/u/fearlessfrog)
#### Post date: [January 6, 2023, 2:04am UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/401 "2023-01-06T02:04:48Z")
The setting allowed me to specify the local site as the origin, to get around the need for the js assets to be on the S3 site (in this case Cloudflare or Digital Ocean Spaces with CDN enabled). Thanks to @david for the change, even if that wasn’t the intention.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [January 6, 2023, 4:55am UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/402 "2023-01-06T04:55:27Z")
Do you enter the site url for the asset cdn? Clever!
---
### Author:  [@satonotdead](https://meta.discourse.org/u/satonotdead)
#### Post date: [January 31, 2023, 6:03pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/404 "2023-01-31T18:03:24Z")
Hi folks, anybody knows if that could be related with Discourse?
That’s the XML of the files that we tried to upload to our previously ‘working with Discourse’ S3 storage:
```plaintext
InvalidArgument
Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.
Authorization
null
ID
ID
```
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [January 31, 2023, 7:30pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/405 "2023-01-31T19:30:52Z")
Are you using AWS? Something else?
Is that bucket configured with server side encryption?
It could be that a library got updated and is behaving differently.
---
### Author:  [@satonotdead](https://meta.discourse.org/u/satonotdead)
#### Post date: [January 31, 2023, 10:02pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/406 "2023-01-31T22:02:58Z")
Thanks, I double-checked and it seems to work with auto configuration but not managing my own keys from S3 management.
Do you know if can be possible within Discourse?
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [February 19, 2023, 1:34pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/407 "2023-02-19T13:34:51Z")
3 posts were split to a new topic: [Why run UpdatePostUploadsSecureStatus even when secure uploads is disabled?](https://meta.discourse.org/t/why-run-updatepostuploadssecurestatus-even-when-secure-uploads-is-disabled/255612)
---
### Author:  [@aosus](https://meta.discourse.org/u/aosus)
#### Post date: [April 6, 2023, 3:19am UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/408 "2023-04-06T03:19:25Z")
> [@Falco](#):
>
> > [@](#):
> >
> > Files that you gzip are not currently handled correctly by R2. You have to upload uncompressed files. Cloudflare has transparent compression, they pick identity, gzip, or Brotli based on what the client can handle. This is a difference from S3.
this seems to have been fixed recently.
In the [2023-3-16 changelog](https://developers.cloudflare.com/r2/reference/changelog/#2023-03-16) it lists bug fix for gzip files handling.
We are running our discourse forum at [discourse.aosus.org](http://discourse.aosus.org) with R2 right now(haven’t run migrate\_to\_s3 yet), and it seems to be OK!, no noticeable issues so far.
```plaintext
DISCOURSE_USE_S3: true
DISCOURSE_S3_REGION: "us-east-1" #alias to auto
#DISCOURSE_S3_INSTALL_CORS_RULE: true #it should be supported
DISCOURSE_S3_ENDPOINT: S3_API_URL
DISCOURSE_S3_ACCESS_KEY_ID: xxx
DISCOURSE_S3_SECRET_ACCESS_KEY: xxxx
DISCOURSE_S3_CDN_URL: your cdn url
DISCOURSE_S3_BUCKET: BUCKET_NAME
```
is there a way to specify a separate hosts for backups?, it would be great if its possible to leave R2 just for CDN stuff.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 6, 2023, 4:41am UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/409 "2023-04-06T04:41:54Z")
> [@aosus](#):
>
> is there a way to specify a separate hosts for backups?
There is not. It seems unlikely to me that this will change.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [April 26, 2023, 6:49pm UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/410 "2023-04-26T18:49:05Z")
23 posts were split to a new topic: [Troubles configuring Object Storage](https://meta.discourse.org/t/troubles-configuring-object-storage/262996)
---
### Author:  [@Hyan](https://meta.discourse.org/u/Hyan)
#### Post date: [April 22, 2023, 6:50am UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/413 "2023-04-22T06:50:24Z")
> [@pfaffman](#):
>
> If you use the ENV variables (which you should) those settings are no longer visible in the Web UI.
It’s wired that the settings in ENV do not reflect in admin UI. Does overriding happen? Will new settings of S3 in admin UI override those in environment?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 22, 2023, 10:24am UTC](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916/414 "2023-04-22T10:24:39Z")
> [@Hyan](#):
>
> Does overriding happen?
Yes. Env variables override values on the database and are hidden from the UX.
[Previous page](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916.md?page=8)
[Next page](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916.md?page=10)