Using Object Storage for Uploads (S3 & Clones)

I just went over some common Object Storage providers so I can attest if they work or not with Discourse.

Provider Service Name Works with Discourse?
Amazon AWS S3 Yes
Digital Ocean Spaces Yes
Linode Object Storage Yes
Google Cloud Storage Yes
Scaleway Object Storage Yes
Vultr Object Storage Yes
BackBlaze Cloud Storage Yes
Self-hosted MinIO Yes

If you got a different service working, please add it to this wiki.

Configuration

In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the hooks section:

  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

When using object storage, you also need a CDN to serve what gets stored in the bucket. I used StackPath CDN in my testing, and other than needing to set Dynamic Caching By Header: Accept-Encoding in their configuration it works ok.

DISCOURSE_CDN_URL is a CDN that points to you Discourse hostname and caches requests. It will be used mainly for pullable assets: CSS and other theme assets.

DISCOURSE_S3_CDN_URL is a CDN that points to your object storage bucket and caches requests. It will be mainly used for pushable assets: JS, images and user uploads.

We recommend those being different and for admins to set both.

In the following examples https://falcoland-files-cdn.falco.dev is a CDN configured to serve the files under the bucket. The bucket name was set to falcoland-files in my examples.

Choose your provider from the list below and add these settings to the env section of your app.yml file, adjusting the values accordingly:

AWS S3

What we officially support and use internally. Their CDN offering Cloudfront also works to front the bucket files. See Setting up file and image uploads to S3 for how to configure the permissions properly.

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-west-1
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Digital Ocean Spaces

DO offering is good and works out of the box. Only problem is that their CDN offering is awfully broken, so you need to use a different CDN for the files.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: whatever
  DISCOURSE_S3_ENDPOINT: https://nyc3.digitaloceanspaces.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Linode Object Storage

An extra configuration parameter, HTTP_CONTINUE_TIMEOUT, is required for Linode.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-east-1
  DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT: 0
  DISCOURSE_S3_ENDPOINT: https://us-east-1.linodeobjects.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

GCP Storage

Listing files is broken, so you need an extra ENV to skip that so assets can work. Also skip CORS and configure it manually.

:warning: Since you can’t list files you won’t be able to list backups, and automatic backups will fail, we don’t recommend using it for backups. :warning:

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: us-east1
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  FORCE_S3_UPLOADS: 1
  DISCOURSE_S3_ENDPOINT: https://storage.googleapis.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  #DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  #DISCOURSE_BACKUP_LOCATION: s3

Scaleway Object Storage

Scaleway offering is also very good, and everything works fine.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: fr-par
  DISCOURSE_S3_ENDPOINT: https://s3.fr-par.scw.cloud
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backups
  DISCOURSE_BACKUP_LOCATION: s3

Vultr Object Storage

An extra configuration parameter, HTTP_CONTINUE_TIMEOUT, is required for Vultr.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: whatever
  DISCOURSE_S3_HTTP_CONTINUE_TIMEOUT: 0
  DISCOURSE_S3_ENDPOINT: https://ewr1.vultrobjects.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

Backblaze B2 Cloud Storage

You need to skip CORS and configure it manually.

Example configuration:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: "us-west-002"
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  DISCOURSE_S3_CONFIGURE_TOMBSTONE_POLICY: false
  DISCOURSE_S3_ENDPOINT: https://s3.us-west-002.backblazeb2.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://falcoland-files-cdn.falco.dev
  DISCOURSE_S3_BUCKET: falcoland-files
  DISCOURSE_S3_BACKUP_BUCKET: falcoland-files/backup
  DISCOURSE_BACKUP_LOCATION: s3

MinIO Storage Server

There are a few caveats and requirements you need to ensure are met before you can use MinIO storage server as an alternative to S3:

  1. You have a fully configured MinIO server instance
  2. You have Domain Support enabled in the MinIO configuration
  3. You have DNS configuration properly set up for MinIO so that bucket subdomains properly resolve to the MinIO server and the MinIO server is configured with a base domain (in this case, minio.example.com)
  4. The bucket discourse-data exists on the MinIO server and has a “public” policy set on it
  5. Your S3 CDN URL points to a properly configured CDN pointing to the bucket and cache requests, as stated earlier in this document.
  6. Your CDNs are configured to actually use a “Host” header of the core S3 URL - for example, discourse-data.minio.example.com when it fetches data - otherwise it can cause CORB problems.

Assuming the caveats and prerequisites above are met, an example configuration would be something like this:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: anything
  DISCOURSE_S3_ENDPOINT: https://minio.example.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://discourse-data-cdn.example.com
  DISCOURSE_S3_BUCKET: discourse-data
  DISCOURSE_S3_BACKUP_BUCKET: discourse-backups
  DISCOURSE_BACKUP_LOCATION: s3
  DISCOURSE_S3_INSTALL_CORS_RULE: false

CORS is still going to be enabled on MinIO even if the rule is not installed by the app rebuilder - by default, it seems, CORS is enabled on all HTTP verbs in MinIO, and MinIO does not support BucketCORS (S3 API) as a result

43 Likes
Setting up file and image uploads to S3
Defining DISCOURSE_S3_CDN_URL links to assets in S3 CDN URL
Using Scaleway s3-compatible object storage
Setting up backup and image uploads to DigitalOcean Spaces
Migrate from AWS to Digital Ocean with 2 containers, spaces and 2 CDNs
Extend S3 configuration for other s3 API compatible cloud storage solutions
How to Setup BackBlaze S3 with BunnyCDN
Setting up backup and image uploads to Backblaze B2
What are the right settings to use S3 bucket (with non-Amazon URL)?
Can not access backup page and related error when restoring using GCP Object Storage
Theme modifiers: A brief introduction
Which free storage for many images? also to be used for thumbnails etc
Disk usage spike during backup, Discourse crashed hard :-(
Migrate from AWS to Digital Ocean with 2 containers, spaces and 2 CDNs
Restore Failure - S3 (compatible) backup
Restore Failure - S3 (compatible) backup
Digitalocean block storage VS amazon S3
Digitalocean block storage VS amazon S3
Setting up backup and image uploads to DigitalOcean Spaces
Custom emoji don't use CDN for S3 stored assets in a few pages
Admin upgrade page doesn't load with a CDN
Install Discourse for Production Environment on Windows Server
Running Discourse on Azure Web Sites vs. Azure VM?
How to turn off S3 storage?
Access Denied error message when trying to upload images
What are the right settings to use S3 bucket (with non-Amazon URL)?
REQ: Support S3 backup to a service like Backblaze
Setting up backup and image uploads to Backblaze B2
Topic List Previews
Overwrite meta og:image image source to use externally public loaded images on topics?
How to store uploads with multiple web_only servers?
Setting up backup and image uploads to DigitalOcean Spaces
Comparing Discourse for Teams with Discourse
Looking for doc to connect discourse with digital ocean spaces
Looking for doc to connect discourse with digital ocean spaces
Looking for doc to connect discourse with digital ocean spaces
403 Error with digital ocean cdn
NoMethodError downcase s3_bucket_name absolute_base_url
What should I enter in the S3 CDN settings if I don't have a CDN?
Backing up files in Object Storage
Moving uploads and backups to DigitalOcean Block Storage
Configure automatic backups for Discourse
S3 OVH Object Storage
REQ: Support S3 backup to a service like Backblaze
SMF2 Conversion and Rake to S3 Help
What causes rake uploads:fix_relative_upload_links
Running 2 hosts behind haproxy fails with random 404s
Site Blank After Rebuild
Migrate_to_S3 Fails on Rebake
Downloads coming from S3 even with DISCOURSE_S3_CDN_URL set
Basic How-To for Using MinIO storage server run by you for your Discourse Instance
Digital Ocean Spaces don’t implement the AWS S3 API for the CORS rule
Extend S3 configuration for other S3 API compatible services
Decoupled Discourse Application - Managed Redis, Managed Postgres, and DIgital Ocean Volume with Discourse
Digital ocean CDN setup
Migrate_to_S3 Fails on Rebake
High Availability 3 Server setup
Upload assets to S3 after in-browser upgrade
How might we better structure #howto?
Migrating uploaded files from DO to S3
Discourse as a closed wiki
Virus scanning of uploaded files
Imgur images broken
Admin role conflates server admin and board admin
Use WebTorrent to load media objects
Hosting Optimization with Digital Ocean
Hosting Optimization with Digital Ocean

Regarding this:

And the tip in the dashboard:

If Secure Media is enabled, I think that it is not necessarily the best idea to use CDN.

3 Likes

You still want it to serve your site JS assets.

2 Likes

Thanks a lot for this detailed guide, and the effort you put in supporting people, it feels really good :slight_smile:

I tried to setup the minio part. I think I’ve got everything correct, but if I’m here :slight_smile:

When I inspect the index page I see something like this:

<link rel="preload" href="https://discourse-data.minio.example.com/browser-detect.br.js" as="script">

Here is my config in term of env:

  DISCOURSE_USE_S3: true
  DISCOURSE_S3_REGION: anything
  DISCOURSE_S3_ENDPOINT: https://minio.example.com
  DISCOURSE_S3_ACCESS_KEY_ID: myaccesskey
  DISCOURSE_S3_SECRET_ACCESS_KEY: mysecretkey
  DISCOURSE_S3_CDN_URL: https://discourse-data.minio.example.com
  DISCOURSE_S3_BUCKET: discourse-data
  DISCOURSE_S3_BACKUP_BUCKET: discourse-backups
  DISCOURSE_BACKUP_LOCATION: s3
  DISCOURSE_S3_INSTALL_CORS_RULE: false

In term of cdn, I just use the bucket url, as I don’t see why it wouldn’t work.

If I inspect the bucket, there is no browser-detect.br.js at the root of the bucket.
I do have: assets/browser-detect-115ab5953de1b5bb122bfb26b757f5391dd8d1d2aef2b81baf7b59aee99d9f34.br.js
And I can curl this asset, and the response has the right header:

curl https://discourse-data.minio.example.com
/assets/browser-detect-115ab5953de1b5bb122bfb26b757f5391dd8d1d2aef2b81baf7b59aee99d9f34.br.js -v -H "Origin: http://forum.example.com"
...
< access-control-allow-origin: *
...

So indeed, I don’t have a CDN, nor an S3_CDN, but I don’t think I would need one (I mean I understand that people need CDN, in my case, I just want minio).

I tried with stable and beta, without luck…

I also searched the code to try to understand where this asset path was set, without much success.

I did another test, with a different setup, and I got it working, except on the avatar side, when uploading, it would add https: to the beginning of the URL :confused:

If you have ideas about what I was missing, I’d love to try :slight_smile:

1 Like

I’ve never run into what you’re seeing, but I think your configuration is wrong.

Per your config you DO have a CDN configured:

DISCOURSE_S3_CDN_URL: https://discourse-data.minio.example.com

Leave this unset! The system automatically configures the S3 URLs for assets on the S3, and you shouldn’t use the CDN URL as your bucket URL, unless you’ve got a CDN at play.

When the assets get rebaked during the app rebuild process, it should put the URLs proper based on my tests.

3 Likes

I’d been using S3 for uploads with my Discourse install since early 2016 or therabouts and only started getting the prompt to configure a CDN for S3 for the past couple of months. I can confirm that my images are still being sent to S3, so the current behaviour is working, and I’d just like to dismiss this new notification without causing any other headaches – can a maintainer confirm that if I retrieve the CDN URL from my AWS console and plug it into Discourse, that I won’t need to rebake posts, given that everything seems to be working fine for now and I don’t want the current behaviour to change?

1 Like

Hi. I want to activate Digitalocean CDN in my forum, but I just couldn’t manage it.

The site is hosted on digitaocean. I want to use Digitalocen spaces.

My configuration is as follows:
In the containers / app.yml file, I add the following at the end of the env: section:

   DISCOURSE_USE_S3: true
   DISCOURSE_S3_REGION: fra1
   DISCOURSE_S3_ENDPOINT: https://fra1.digitaloceanspaces.com
   DISCOURSE_S3_ACCESS_KEY_ID: key
   DISCOURSE_S3_SECRET_ACCESS_KEY: secretkey
   DISCOURSE_S3_CDN_URL: https://depo.domain.com
   DISCOURSE_S3_BUCKET: resimler
   DISCOURSE_S3_BACKUP_BUCKET: forum/backup
   DISCOURSE_BACKUP_LOCATION: s3

then again in the containers / app.yml file, I add the following under hooks:

hooks:
   after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
   after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - sudo -E -u discourse git clone https://github.com/discourse/discourse-sitemap.git

I’m running the latest ./launcher rebuild app command and the error I get is:


What am I missing?

1 Like

As said in the image, scroll up to see the actual error message.

3 Likes

i guess this

 I, [2021-02-09T19:10:01.426447 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets
    `/root` is not writable.
    Bundler will use `/tmp/bundler20210209-4893-zwwb94893' as your home directory temporarily.
    rake aborted!
    Aws::S3::Errors::NoSuchKey: Aws::S3::Errors::NoSuchKey
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.109.2/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/plugins/dualstack.rb:30:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/plugins/accelerate.rb:47:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.109.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.109.2/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.109.2/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.109.2/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.109.2/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.109.2/lib/seahorse/client/plugins/response_target.rb:24:in `call'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-core-3.109.2/lib/seahorse/client/request.rb:72:in `send_request'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/aws-sdk-s3-1.83.2/lib/aws-sdk-s3/client.rb:8425:in `put_bucket_cors'
    /var/www/discourse/lib/s3_helper.rb:124:in `ensure_cors!'
    /var/www/discourse/lib/tasks/s3.rake:191:in `block in <main>'
    /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
    /usr/local/bin/bundle:23:in `load'
    /usr/local/bin/bundle:23:in `<main>'
    Tasks: TOP => s3:upload_assets
    (See full trace by running task with --trace)
1 Like

Maybe because the bucket name is kdn-depo and you put resimler for DISCOURSE_S3_BUCKET ?

2 Likes

I think it worked: D

I, [2021-02-09T19: 35: 01.488916 # 1] INFO -:> cd / var / www / discourse && sudo -E -u discourse bundle exec rake s3: upload_assets
`/ root` is not writable.
Bundler will use `/ tmp / bundler20210209-4885-175z71e4885 'as your home directory temporarily.
I, [2021-02-09T19: 35: 19.823643 # 1] INFO -: installing CORS rule

I guess I need to make a setting in dpsces for CORS. The site opens but nothing.

is this configuration correct?

1 Like

I think that’s because it’s broken.

3 Likes

Hi @Falco - Have any thoughts on what could be causing all avatar images (optimized) to not work (displaying default grey avatar icon) after migrating to an s3 clone?

I can confirm that:

  • Original avatars migrated successfully and correct paths are reflected in the database for those uploads.
  • New avatar uploads go to s3 successfully, but are not displayed either (optimized).
  • Other user uploads migrated successfully and display well, such as post images, profile card background and profile background. New uploads of these types work well.
  • I have other instances with the same configs, using the same s3 clone from the beginning and they work really well.

Also, do you know if there is a way to disable multipart uploads? I’m getting an issue uploading larger backup files as described here.

Thanks for providing this guide, it’s been very helpful.

1 Like

Hey! Nice thing something moves here.
Uploading files to the local disk is simple, but bad solution. Adding minio with simple nginx cache will add little memory footprint, but will abstract uploads totally to external service (nice to have in 12 factor app).

Is there are any plans adding minio to self-hosted solutions as a default?

1 Like

Our avatar code path is quite complex so this sounds plausible. However debugging this without reproducible steps is impossible.

There is no way to disable that. Multipart upload should be working, may try contacting scale way support and ask for a fix?

3 Likes

No such plans. Anyone who wants to use it can install and configure following this guide.

3 Likes

Thanks @Falco!

So just to add to this, when comparing the instances with working avatars to the instance with non-working avatars. I can see that srcset=“path/to/optimized/image” is missing on all post images (both migrated and new) I’ve checked on the non-working avatars instance.

So the issue seems to be with all optimized images (migrated and new uploads), not just the avatars. Just the non-avatar images are displaying still because the originals are being used.


Regarding reproducible steps:

  • Configured s3 uploads as described in this post on a new instance (using the same settings on other instances without issue that didn’t go through a migration).

  • Commented out two s3 backups settings (app.yml).

  • Uploaded backup (backup of a Discourse instance that was storing images locally) to the appropriate folder on the new server and restored it.

  • After successfully restoring the backup, uncommented the two s3 backup settings in the app.yml and rebuilt.

  • Attempted to run:
    rake uploads:migrate_to_s3
    Received this error:

Some uploads were not migrated to the new scheme. Running the migration, this may take a while...
rake aborted!
FileStore::ToS3MigrationError: Some uploads could not be migrated to the new scheme. You need to fix this manually.
  • Set
    SiteSetting.migrate_to_new_scheme = true

  • Waited until this returned false:
    SiteSetting.migrate_to_new_scheme

  • Found uploads with issues:
    Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '/uploads/default/original/_X/%'").to_a

  • Deleted the uploads with issues (the local image files didn’t exist for these records):
    Upload.find(upload_id).destroy

  • Ran:
    rake uploads:migrate_to_s3

Received error:

Updating the URLs in the database...
Warning: no type cast defined for type "name" with oid 19. Please cast this type explicitly to TEXT to be safe for future changes.
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
123456 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 3 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:131:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:86:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:357:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:245:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:224:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:76:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:86:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:222:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:218:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)
  • Tried to find the 3 posts with issues unsuccessfully.

  • Ran:
    rake posts:rebake_uncooked_posts

  • Received some errors while rebake was running, but did not seem critical or stop the rebake:
    Failed to report error: "\x8B" from ASCII-8BIT to UTF-8 3 Job exception: 403 Error:
    With unreadable characters underneath.

  • Migration appears to have been successful from all that I’ve checked (image files moved to s3, s3 upload paths, s3 avatar upload paths, s3 cdn url), the only issue seems to be the generating and displaying of optimized images for both migrated and new uploads.

2 Likes

Hi everyone,

I’ve been using S3 storage for a number of years now without a CDN.

Following the advice given to me in another thread I have today setup CloudFront CDN.

Before I add the CDN URL to my control panel and rebake 230,000+ posts only to find out I’ve got a CloudFront setting wrong and break everything, can someone confirm this is the expected behaviour for me please? :bowing_man:t2:

Currently, this is an example URL that for an image that a user has uploaded:

https://greyarrows.s3.dualstack.eu-west-2.amazonaws.com/original/3X/8/3/8335cab232f512f4a979c7f0c8562e149c01b212.png

Which displays:

My CloudFront “Domain Name” is: d1q8cepst0v8xp.cloudfront.net

If I manually edit my example URL above and replace the existing S3 part of the domain name with the domain name of my CloudFront domain name, I get:

https://d1q8cepst0v8xp.cloudfront.net/original/3X/8/3/8335cab232f512f4a979c7f0c8562e149c01b212.png

And sure enough, the image still loads correctly:

Therefore, am I correct in thinking I simply need to add a S3 CDN URL of d1q8cepst0v8xp.cloudfront.net` to my Discourse control panel, rebake all posts and just sit back and wait of the magic to happen?

Thanks in advance, CDN is all new to me and I don’t have a development environment in which to safely test this :grimacing:

4 Likes

I also have the s3 configure tombstone policy setting enabled:

Will this be an issue? As I’m now using a CDN instead? Or are things in the background still checking the original S3 bucket, rather than a CDN URL?

I’m naturally thinking the latter, but again I can’t afford to kill of hundreds of thousands of my users photo uploads :scream:

:blush:

2 Likes

The answer is yes.

To test this theory, before rebaking hundreds of thousands of posts, I did the following sanity checks:

  • Uploaded an image
  • Changed the S3 CDN URL setting
  • Rebuilt the HTML on my test post (via the UI)
  • Refreshed the page in the browser
  • Checked the Network tab of the browser console to confirm the image was being pulled via cloudfront
  • Uploaded a new test image to a new post
  • Checked the Network tab of the browser console to confirm the image was being pulled via cloudfront

I’m now rebaking all posts as we speak :+1:t2:

8 Likes