Using Object Storage for Uploads (S3 & Clones)

I just went over some common Object Storage providers so I can attest if they work or not with Discourse.

Provider Service Name Works with Discourse?
Amazon AWS S3 Yes
Digital Ocean Spaces Yes
Linode Object Storage Yes
Google Cloud Storage Yes
Scaleway Object Storage Yes
Vultr Object Storage Yes
BackBlaze Cloud Storage Yes

If you got a different service working, please add it to this wiki.

Configuration

In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the hooks section:

  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

When using object storage, you also need a CDN to serve what gets stored in the bucket. I used StackPath CDN in my testing, and other than needing to set Dynamic Caching By Header: Accept-Encoding in their configuration it works ok.

DISCOURSE_CDN_URL is a CDN that points to you Discourse hostname and caches requests. It will be used mainly for pullable assets: CSS and other theme assets.

DISCOURSE_S3_CDN_URL is a CDN that points to your object storage bucket and caches requests. It will be mainly used for pushable assets: JS, images and user uploads.

We recommend those being different and for admins to set both.

In the following examples https://falcoland-files-cdn.falco.dev is a CDN configured to serve the files under the bucket. The bucket name was set to falcoland-files in my examples.

Choose your provider from the list below and add these settings to the env section of your app.yml file, adjusting the values accordingly:

AWS S3

What we officially support and use internally. Their CDN offering Cloudfront also works to front the bucket files. See Setting up file and image uploads to S3 for how to configure the permissions properly.

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-west-1
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Digital Ocean Spaces

DO offering is good and works out of the box. Only problem is that their CDN offering is awfully broken, so you need to use a different CDN for the files.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: whatever
  DISCOURSE_S3_ENDPOINT: https://nyc3.digitaloceanspaces.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Linode Object Storage

An extra configuration parameter, HTTP_CONTINUE_TIMEOUT, is required for Linode.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-east-1
  DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT: 0
  DISCOURSE_S3_ENDPOINT: https://us-east-1.linodeobjects.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

GCP Storage

Listing files is broken, so you need an extra ENV to skip that so assets can work. Also skip CORS and configure it manually.

:warning: Since you can’t list files you won’t be able to list backups, and automatic backups will fail, we don’t recommend using it for backups. :warning:

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-east1
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  FORCE_S3_UPLOADS: 1
  DISCOURSE_S3_ENDPOINT: https://storage.googleapis.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  #DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  #DISCOURSE_BACKUP_LOCATION: s3

Scaleway Object Storage

Scaleway offering is also very good, and everything works fine.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: fr-par
  DISCOURSE_S3_ENDPOINT: https://s3.fr-par.scw.cloud
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Vultr Object Storage

An extra configuration parameter, HTTP_CONTINUE_TIMEOUT, is required for Vultr.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: whatever
  DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT: 0
  DISCOURSE_S3_ENDPOINT: https://ewr1.vultrobjects.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

Backblaze B2 Cloud Storage

You need to skip CORS and configure it manually.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: "us-west-002"
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  DISCOURSE_S3_CONFIGURE_TOMBSTONE_POLICY: false
  DISCOURSE_S3_ENDPOINT: https://s3.us-west-002.backblazeb2.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3
39 Likes
Using Scaleway s3-compatible object storage
Defining DISCOURSE_S3_CDN_URL links to assets in S3 CDN URL
Setting up backup and image uploads to DigitalOcean Spaces
Setting up file and image uploads to S3
Migrate from AWS to Digital Ocean with 2 containers, spaces and 2 CDNs
Extend S3 configuration for other s3 API compatible cloud storage solutions
How to Setup BackBlaze S3 with BunnyCDN
What are the right settings to use S3 bucket (with non-Amazon URL)?
Setting up backup and image uploads to Backblaze B2
Migrating uploaded files from DO to S3
Discourse as a closed wiki
Imgur images broken
Admin role conflates server admin and board admin
Use WebTorrent to load media objects
Hosting Optimization with Digital Ocean
Hosting Optimization with Digital Ocean
Theme modifiers: A brief introduction
Which free storage for many images? also to be used for thumbnails etc
Disk usage spike during backup, Discourse crashed hard :-(
Migrate from AWS to Digital Ocean with 2 containers, spaces and 2 CDNs
Restore Failure - S3 (compatible) backup
Restore Failure - S3 (compatible) backup
Digitalocean block storage VS amazon S3
Digitalocean block storage VS amazon S3
Setting up backup and image uploads to DigitalOcean Spaces
Custom emoji don't use CDN for S3 stored assets in a few pages
Admin upgrade page doesn't load with a CDN
Install Discourse for Production Environment on Windows Server
Running Discourse on Azure Web Sites vs. Azure VM?
How to turn off S3 storage?
Access Denied error message when trying to upload images
What are the right settings to use S3 bucket (with non-Amazon URL)?
REQ: Support S3 backup to a service like Backblaze
REQ: Support S3 backup to a service like Backblaze
Topic List Previews
Overwrite meta og:image image source to use externally public loaded images on topics?
How to store uploads with multiple web_only servers?
Setting up backup and image uploads to DigitalOcean Spaces
Comparing Discourse for Teams with Discourse
Looking for doc to connect discourse with digital ocean spaces
Looking for doc to connect discourse with digital ocean spaces
Looking for doc to connect discourse with digital ocean spaces
403 Error with digital ocean cdn
Can not access backup page and related error when restoring using GCP Object Storage
NoMethodError downcase s3_bucket_name absolute_base_url
What should I enter in the S3 CDN settings if I don't have a CDN?
Backing up files in Object Storage
Moving uploads and backups to DigitalOcean Block Storage
Configure automatic backups for Discourse
Setting up backup and image uploads to Backblaze B2
S3 OVH Object Storage
SMF2 Conversion and Rake to S3 Help
What causes rake uploads:fix_relative_upload_links
Running 2 hosts behind haproxy fails with random 404s
Site Blank After Rebuild
Migrate_to_S3 Fails on Rebake
Digital Ocean Spaces don’t implement the AWS S3 API for the CORS rule
Extend S3 configuration for other S3 API compatible services
Upload assets to S3 after in-browser upgrade