Using Object Storage for Uploads (S3 Clones)

I just went over some common Object Storage providers so I can attest if they work or not with Discourse.

Provider Service Name Works with Discourse?
Amazon AWS S3 Yes
Digital Ocean Spaces Yes
Linode Object Storage Yes
Google Cloud Storage Yes
Scaleway Object Storage Yes
Vultr Object Storage Yes
BackBlaze Cloud Storage Yes

If you got a different service working, please add it to this wiki.

Configuration

In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the hooks section:

  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

When using object storage, you also need a CDN to serve what gets stored in the bucket. I used StackPath CDN in my testing, and other than needing to set Dynamic Caching By Header: Accept-Encoding in their configuration it works ok.

DISCOURSE_CDN_URL is a CDN that points to you Discourse hostname and caches requests. It will be used mainly for pullable assets: CSS and other theme assets.

DISCOURSE_S3_CDN_URL is a CDN that points to your object storage bucket and caches requests. It will be mainly used for pushable assets: JS, images and user uploads.

We recommend those being different and for admins to set both.

In the following examples https://falcoland-files-cdn.falco.dev is a CDN configured to serve the files under the bucket. The bucket name was set to falcoland-files in my examples.

AWS S3

What we officially support and use internally. Their CDN offering Cloudfront also works to front the bucket files.

In order to use it add this to the env section of your app.yml file, adjusting the values accordingly:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-west-1
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Digital Ocean Spaces

DO offering is good and works out of the box. Only problem is that their CDN offering is awfully broken, so you need to use a different CDN for the files.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: whatever
  DISCOURSE_S3_ENDPOINT: https://nyc3.digitaloceanspaces.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Linode Object Storage

An extra configuration parameter, HTTP_CONTINUE_TIMEOUT, is required for Linode.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-east-1
  DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT: 0
  DISCOURSE_S3_ENDPOINT: https://us-east-1.linodeobjects.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

GCP Storage

Listing files is broken, so you need an extra ENV to skip that. Also skip CORS and configure it manually.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-east1
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  FORCE_S3_UPLOADS: 1
  DISCOURSE_S3_ENDPOINT: https://storage.googleapis.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

Scaleway Object Storage

Scaleway offering is also very good, and everything works fine.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: fr-par
  DISCOURSE_S3_ENDPOINT: https://s3.fr-par.scw.cloud
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Vultr Object Storage

An extra configuration parameter, HTTP_CONTINUE_TIMEOUT, is required for Vultr.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: whatever
  DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT: 0
  DISCOURSE_S3_ENDPOINT: https://ewr1.vultrobjects.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

Backblaze B2 Cloud Storage

You need to skip CORS and configure it manually.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: "us-west-002"
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  DISCOURSE_S3_ENDPOINT: https://s3.us-west-002.backblazeb2.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3
31 Likes

@Falco can you share any specifics on this CDN brokenness?

I’m currently testing the migration to DO S3+CDN and haven’t experienced any issues so far.

How would one go about diagnosing the potential CDN issues?

2 Likes

It’s what is documented here:

Warning

Due to a known issue, file metadata headers like Content-Encoding are not passed through the CDN. Metadata headers are correctly set when fetching content directly from the origin.

A missing content-encoding means all our brotli encoded assets are broken.

That means you can use their Object Storage offering, but must shop for a caching CDN in another provider.

3 Likes

@falco, please add your response to the DO section; this is the same question I asked, so we ought to capture it. :slight_smile:

1 Like

It is already there.

3 Likes

I figured we want the “why”, since it was asked twice.

Something like, “…as documented Content-Encoding headers are not set correctly.”

Since the install is for DO, I presume this is a common gotcha.

2 Likes

In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the hooks section:

So I use:

cd /var/discourse
nano containers/app.yml

Then I’m at the plugin area, where it says “hooks”. Where do I put this code? Like this?

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-spoiler-alert.git
          - git clone https://github.com/discourse/discourse-akismet.git
          - git clone https://github.com/discourse/discourse-adplugin.git
          - git clone https://github.com/discourse/discourse-patreon.git
          - git clone https://github.com/discourse/discourse-sitemap.git

after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

## Any custom commands to run after building
1 Like

I did this yesterday and it seems that you need to have not only DISCOURSE_S3_BUCKET defined, but also DISCOURSE_S3_UPLOAD_BUCKET. It’s confusing (to me, anyway) that both of these variables exist.

1 Like

What was the exact error without it defined? The rebuild failed?

1 Like

No. The rebuild worked, but it wouldn’t run:

                                                                                                                                         
root@abednegoat-voicecomputer:/# tail -f /shared/log/rails/production.log                                                                         
ActionController::RoutingError (No route matches [GET] "/memberlist.php")                                                                         
config/initializers/100-quiet_logger.rb:19:in `call'                                                                                                      
config/initializers/100-silence_logger.rb:31:in `call'                                                                                                     
lib/middleware/enforce_hostname.rb:22:in `call'                                                                                                            
lib/middleware/request_tracker.rb:176:in `call'                                                                                                     
  Rendered exceptions/_not_found_topics.html.erb (Duration: 18.6ms | Allocations: 4451)                                                              
  Rendering exceptions/not_found.html.erb within layouts/no_ember                                                                                    
  Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 0.2ms | Allocations: 144)                                               
  Rendered layouts/_head.html.erb (Duration: 1.9ms | Allocations: 377)                                                                        
Failed to handle exception in exception app middleware : s3_upload_bucket                                                                     
Started GET "/" for 71.89.242.121 at 2020-07-23 22:08:53 +0000                                                                   
Processing by ListController#latest as HTML                                                                                                     
Completed 500 Internal Server Error in 3ms (ActiveRecord: 0.0ms | Allocations: 790)                                                              
Discourse::SiteSettingMissing (s3_upload_bucket)                                                                                                 
lib/file_store/s3_store.rb:188:in `s3_bucket'                                                        
lib/file_store/s3_store.rb:17:in `initialize'                         
lib/discourse.rb:657:in `new'                                                                                                                    
lib/discourse.rb:657:in `store'                                                                                                                   
lib/global_path.rb:14:in `upload_cdn_path'                                                                                                        
lib/global_path.rb:29:in `full_cdn_url'           
1 Like

The only commit I can see that would trigger this change is

1 Like

Well, that doesn’t look like the culprit. But it makes sense that S3 uploads would not work without DISCOURSE_S3_UPLOAD_BUCKET being defined. I don’t see how the OP could have worked (but I’m still very confused about DISCOURSE_S3_UPLOAD_BUCKET and DISCOURSE_S3_BUCKET

1 Like

Is S3 really needed long-term (let’s say 5-10 years)? What is the advantage of using S3 over the default? The Amazon one is only free for the first 12 months. Is there any free S3 solution?

1 Like

What do you mean by that?

It is not mandatory, hence it being a site setting and not a default in Discourse. But it is very useful and a great experience for both admins and users. And there are dozens of providers.

Every request to a static assets goes to a dedicated CDN/Service combo and doesn’t hit your server leaving it to only serve dynamic requests. Which means everything is faster and closer to the user.

Discourse backups switch to being just the database and are also faster and smaller.

Object Storage plus the Discourse feature of downloading images means your forum images will never break. 30 years from now they will still be loading. And everything is under your control.

Using Object Storage may not make sense for toy/small communities with zero budget. It is not about being cheaper, but better.

If cost is a blocker stick to local uploads.

4 Likes

You gotta dig deeper.

The SiteSetting.Upload.s3_upload_bucket.blank? you quote is defined as:

So as long as the Global is defined it didn’t count as empty, which may have changed due to the new validation. Or a typo on your app.yml :stuck_out_tongue:

2 Likes

Well, that’s certainly possible, as my eyes kept getting confused about the UPLOADS over the course of at least an hour. I may try again on a clean site later to see if I’m just crazy. This stuff has been giving me fits since last December at least.

2 Likes

Cheapest I found was backblaze. It’s free up to 10 gigs and then very cheap after that. Plus you get free data transfer if use cloudflare because they have a partnership with them.

3 Likes

Upon rebuilding after adding the hook, I got this:

Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 18708 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets"]}
1 Like

You gotta scroll up and copy the error backtrace for us to debug it.

1 Like

This is what I added in my yml after the plugins:

  after_assets_precompile:
        - exec:
            cd: $home
            cmd:
              - sudo -E -u discourse bundle exec rake s3:upload_assets

This is the error I get after rebuild:

166:M 24 Jul 2020 21:44:58.524 # Redis is now ready to exit, bye bye...
2020-07-24 21:44:58.525 UTC [49] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 18717 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets"]}
a1ddd688e98b6f81fdeddc5e78d5bcb21f90e7e53d7105e4bc3f80c5b04d7dcf
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
root@duelistsunite:/var/www/docker-sites/discourse# ./discourse-doctor
DISCOURSE DOCTOR Sat 25 Jul 2020 12:50:26 AM CEST
OS: Linux duelistsunite 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Note that my rebuild works and the site works as long as I don’t add the above code in the yml.

1 Like