Using Object Storage for Uploads (S3 & Clones)

I just went over some common Object Storage providers so I can attest if they work or not with Discourse.

Provider Service Name Works with Discourse?
Amazon AWS S3 Yes
Digital Ocean Spaces Yes
Linode Object Storage Yes
Google Cloud Storage Yes
Scaleway Object Storage Yes
Vultr Object Storage Yes
BackBlaze Cloud Storage Yes

If you got a different service working, please add it to this wiki.

Configuration

In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the hooks section:

  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

When using object storage, you also need a CDN to serve what gets stored in the bucket. I used StackPath CDN in my testing, and other than needing to set Dynamic Caching By Header: Accept-Encoding in their configuration it works ok.

DISCOURSE_CDN_URL is a CDN that points to you Discourse hostname and caches requests. It will be used mainly for pullable assets: CSS and other theme assets.

DISCOURSE_S3_CDN_URL is a CDN that points to your object storage bucket and caches requests. It will be mainly used for pushable assets: JS, images and user uploads.

We recommend those being different and for admins to set both.

In the following examples https://falcoland-files-cdn.falco.dev is a CDN configured to serve the files under the bucket. The bucket name was set to falcoland-files in my examples.

AWS S3

What we officially support and use internally. Their CDN offering Cloudfront also works to front the bucket files.

In order to use it add this to the env section of your app.yml file, adjusting the values accordingly:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-west-1
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Digital Ocean Spaces

DO offering is good and works out of the box. Only problem is that their CDN offering is awfully broken, so you need to use a different CDN for the files.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: whatever
  DISCOURSE_S3_ENDPOINT: https://nyc3.digitaloceanspaces.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Linode Object Storage

An extra configuration parameter, HTTP_CONTINUE_TIMEOUT, is required for Linode.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-east-1
  DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT: 0
  DISCOURSE_S3_ENDPOINT: https://us-east-1.linodeobjects.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

GCP Storage

Listing files is broken, so you need an extra ENV to skip that. Also skip CORS and configure it manually.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-east1
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  FORCE_S3_UPLOADS: 1
  DISCOURSE_S3_ENDPOINT: https://storage.googleapis.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

Scaleway Object Storage

Scaleway offering is also very good, and everything works fine.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: fr-par
  DISCOURSE_S3_ENDPOINT: https://s3.fr-par.scw.cloud
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Vultr Object Storage

An extra configuration parameter, HTTP_CONTINUE_TIMEOUT, is required for Vultr.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: whatever
  DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT: 0
  DISCOURSE_S3_ENDPOINT: https://ewr1.vultrobjects.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

Backblaze B2 Cloud Storage

You need to skip CORS and configure it manually.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: "us-west-002"
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  DISCOURSE_S3_CONFIGURE_TOMBSTONE_POLICY: false
  DISCOURSE_S3_ENDPOINT: https://s3.us-west-002.backblazeb2.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3
33 Likes
Using Scaleway s3-compatible object storage
Defining DISCOURSE_S3_CDN_URL links to assets in S3 CDN URL
Setting up backup and image uploads to DigitalOcean Spaces
Extend S3 configuration for other s3 API compatible cloud storage solutions
Migrating uploaded files from DO to S3
Discourse as a closed wiki
Imgur images broken
Admin role conflates server admin and board admin
Migrate from AWS to Digital Ocean with 2 containers, spaces and 2 CDNs
Use WebTorrent to load media objects
Hosting Optimization with Digital Ocean
Hosting Optimization with Digital Ocean
Theme modifiers: A brief introduction
Which free storage for many images? also to be used for thumbnails etc
Disk usage spike during backup, Discourse crashed hard :-(
Migrate from AWS to Digital Ocean with 2 containers, spaces and 2 CDNs
Restore Failure - S3 (compatible) backup
Restore Failure - S3 (compatible) backup
Digitalocean block storage VS amazon S3
Digitalocean block storage VS amazon S3
Setting up backup and image uploads to Backblaze B2
Setting up backup and image uploads to DigitalOcean Spaces
How to Setup BackBlaze S3 with BunnyCDN
Custom emoji don't use CDN for S3 stored assets in a few pages
Admin upgrade page doesn't load with a CDN
Install Discourse for Production Environment on Windows Server
Running Discourse on Azure Web Sites vs. Azure VM?
How to turn off S3 storage?
Access Denied error message when trying to upload images
What are the right settings to use S3 bucket (with non-Amazon URL)?
What are the right settings to use S3 bucket (with non-Amazon URL)?
REQ: Support S3 backup to a service like Backblaze
Setting up backup and image uploads to Backblaze B2
REQ: Support S3 backup to a service like Backblaze
SMF2 Conversion and Rake to S3 Help
Running 2 hosts behind haproxy fails with random 404s
Site Blank After Rebuild
Migrate_to_S3 Fails on Rebake
Digital Ocean Spaces don’t implement the AWS S3 API for the CORS rule
Extend S3 configuration for other S3 API compatible services
Upload assets to S3 after in-browser upgrade

Just have a question if anyone wants to chime in it would be appreciated. :slight_smile: If I’m changing from storing images locally without a s3 cdn url => s3 with a cdn url (following these instructions). After rebuilding the app, I just need to run these two commands afterwards:

rake posts:rebake followed by rake uploads:migrate_to_s3

Is that correct? Or is rebake alone sufficient?

I tried running just rake uploads:migrate_to_s3 but it uploaded a rather small amount of images (I don’t think it was enough for all the original images). So I think the rebake needs to be done first due to the CDN url. Now that I’m running a rebake I’m seeing a lot of images being uploaded to s3.

1 Like

You need to run both the rebake and migrate commands.

2 Likes

One thing I’ve noticed is this error in my logs which may be specific to Backblaze B2. Does anyone else have this error? Functionally, everything works for me, so I’m not exactly sure what this means. This error appears daily during midnight. I’m not sure if it’s related to backups since my backups work just fine.

Backtrace
aws-sdk-core-3.99.1/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
aws-sdk-s3-1.66.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
aws-sdk-s3-1.66.0/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
aws-sdk-s3-1.66.0/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
aws-sdk-core-3.99.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
aws-sdk-core-3.99.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
aws-sdk-core-3.99.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
aws-sdk-core-3.99.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
aws-sdk-core-3.99.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'
aws-sdk-core-3.99.1/lib/seahorse/client/request.rb:70:in `send_request'
aws-sdk-s3-1.66.0/lib/aws-sdk-s3/client.rb:7930:in `put_bucket_lifecycle_configuration'
/var/www/discourse/lib/s3_helper.rb:176:in `update_lifecycle'
/var/www/discourse/lib/s3_helper.rb:186:in `update_tombstone_lifecycle'
/var/www/discourse/lib/file_store/s3_store.rb:142:in `purge_tombstone'
/var/www/discourse/app/jobs/scheduled/purge_deleted_uploads.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'
rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/app/jobs/base.rb:221:in `block in perform'
/var/www/discourse/app/jobs/base.rb:217:in `each'
/var/www/discourse/app/jobs/base.rb:217:in `perform'
/var/www/discourse/app/jobs/base.rb:279:in `perform'
mini_scheduler-0.12.2/lib/mini_scheduler/manager.rb:86:in `process_queue'
mini_scheduler-0.12.2/lib/mini_scheduler/manager.rb:36:in `block (2 levels) in initialize'
1 Like

Oh looks like they are missing yet another API. Updated the OP to disable this routine.

2 Likes

So should we still be using Backblaze B2 if it doesn’t support yet that API?

1 Like

The only S3 service that have all the S3 API is S3 themselves, everyone using a clone is settling for something not as complete. OP says:

That said, tombstone policy isn’t a deal breaker. You can go and manually clean it when you find yourself over the backblaze max storage for your budget.

2 Likes

Hello,

I use B2 also. One question the manual way is ./launcher cleanup ? It will purge the tombstone?

Thanks :slight_smile:

No, that is not it.

You will need to use a S3 tool, like aws-cli or s3cmd and use a command to remove files from a specific folder.

3 Likes

Hello Moksh,

I follow your instructions and it works fine I upload assets to BackBlaze B2. But now I have to disable because it crashed the admin/upgrade page so delete these from app.yml.

So now the assets and images on B2. I set up BackBlaze B2 on Discourse admin/settings. Now the new images goes to B2 but what happening with assets? It will regenerate locally after rebuild or should I download it from B2?

2 Likes

Hello Don,

I am also a newbie to Discourse and I had shared the variables what worked for me. I would ask a more experienced to pitch in to answer your query.

Thanks!

3 Likes

I don’t see DISCOURSE_CDN_URL in any of the examples in your OP. Could you please explain when this is necessary or desirable to use?

2 Likes

It is explained in the text you quoted. You must have two, each ones handles a part of the assets.

4 Likes

Thank you. After coming up with a blank screen after not including DISCOURSE_CDN_URL, I understood. Thanks for your guide.

1 Like

@Falco, I’m preparing to edit some errors in this wiki topic, and I want to clarify something. Those instructions involve using the admin panel to set up storing files in S3 via a CDN. The admin panel does not seem to have an equivalent to DISCOURSE_CDN_URL. Does Discourse handle page assets differently if uploads are set up in the admin panel vs in app.yml?

1 Like

I’m testing this with DO Spaces. Everything works as expected, however, when I try to upload a backup file I get the following error. Any thoughts on how to correct this?

Access to XMLHttpRequest at 'https://360velo.sfo2.digitaloceanspaces.com/backups/default/360-velo-forum-2019-09-22-162048-v20190315055432.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXXX from origin 'https://forum.360velo.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

I have been digging around this forum and this seems to be caused by http / https conflicts. My site is on https. I played around with the CORS setting at DO to no avail. Here is what that looks like now.

2 Likes

With GCP restoring from a backup seems to be (unfortunately) broken.

It appears to be attempting to list the files that were uploaded to GCP storage despite my app.yml containing FORCE_S3_UPLOADS: 1

Restoring uploads, this may take a while...
Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
...... => 6761 files
 - Listing S3 files
EXCEPTION: A header or query you provided requested a function that is not implemented.
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.99.1/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.99.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.99.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.99.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.99.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.99.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.99.1/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/client.rb:6675:in `list_objects_v2'
/var/www/discourse/lib/file_store/to_s3_migration.rb:190:in `block in migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:189:in `loop'
/var/www/discourse/lib/file_store/to_s3_migration.rb:189:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/file_store/s3_store.rb:238:in `copy_from'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:48:in `restore_uploads'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:30:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:60:in `run'
script/discourse:143:in `restore'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
script/discourse:284:in `<top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Trying to rollback...
Rolling back...
Cleaning stuff up...
Dropping functions from the discourse_functions schema...
Removing tmp '/var/www/discourse/tmp/restores/default/2020-09-10-010517' directory...
Unpausing sidekiq...
Marking restore as finished...
Notifying 'system' of the end of the restore...
Finished!
[FAILED]
Restore done.

Although I managed to restore by killing all db connections and manually importing the dump.

3 Likes

In my tests GCP was one of the worst in compatibility indeed.

3 Likes

@Falco Sorry to bump this inquiry, but I’m still struggling to solve this problem. Any ideas on why the backup uploads would be failing with this error message?

UPDATE: I changed the Allowed Headers to * from Origin https://forum.360velo.com at DO and can now upload backups without errors. I tried it with Allowed Headers of: ‘Access-Control-Allow-Origin’ but that did not work. The new question is is it OK to Allow Headers * as shown in the screen shot?

1 Like

Hey @Falco, I am having a strange issue with S3. After updating configuration like the post, I did ./launcher rebuild app and got error:
INFO – : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets
/root is not writable.
Bundler will use `/tmp/bundler20200925-3644-xfgx0d3644’ as your home directory temporarily.
rake aborted!
Aws::S3::Errors::AccessDenied: Access Denied

I tried aws cli on the EC2 host, uploading is working fine. I read a post saying we can try remove the bucket and discourse will create one automatically. It didn’t work and the error was S3: NoSuchBucket. So seems there is no blocker talking to S3… Is there anything I missed? Thanks!

1 Like