Set up file and image uploads to S3

So, you want to use S3 to handle image uploads? Here’s the definitive guide:

S3 registration

Head over to https://aws.amazon.com/free/ and click on Create a Free Account

During the create account process, make sure you provide payment information, otherwise you won’t be able to use S3. There’s no registration fee, you will only be charged for what you use, if you exceed the AWS Free Usage Tier.

Bucket

Go to S3 and click on Create bucket, then fill out the Bucket name. Remember this name because we’ll need it for the next step.

Name of your bucket

Select a Region. You should enter the location (eg. “EU (Frankfurt)”) that is nearest to your users for better performance.

Scroll down a little until you get to the Permissions panel.

  • When you set up the permissions, make sure that you allow public ACLs, otherwise uploads will fail. Uncheck Block all public access and check the bottom two checkboxes. You’ll also have to acknowledge that your settings may make the bucket public in the warning at the top.

User creation

Creating a policy

Sign in to AWS Management Console and search for the “IAM” service to access the AWS Identity and Access Management (IAM) console which enables you to manage access to your AWS resources.

First, click Policies in the sidebar. Then, click on Create Policy and choose the JSON tab:

image

Use the following piece of code as a template for your policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
               "s3:List*",
               "s3:Get*",
               "s3:AbortMultipartUpload",
               "s3:DeleteObject",
               "s3:PutObject",
               "s3:PutObjectAcl",
               "s3:PutObjectVersionAcl",
               "s3:PutLifecycleConfiguration",
               "s3:CreateBucket",
               "s3:PutBucketCORS"
      ],
      "Resource": [
        "arn:aws:s3:::=BUCKET=",
        "arn:aws:s3:::=BUCKET=/*"
      ]
    },
    {
       "Effect": "Allow",
       "Action": [
           "s3:ListAllMyBuckets",
           "s3:HeadBucket"
       ],
       "Resource": "*"
    }
  ]
}

:warning: Make sure that these two lines contain the actual name of your bucket. :blush:

image

:star2: If you also intend to do S3 backups, you can include your backup bucket here too like this:

image

Then click on Review Policy and fill out the Name field. For example, you can call it s3-discourse-policy

Then click on Create Policy at the bottom.

Creating a user account

Now that you’ve created the right policy, you’re ready to create the user account. Click on the Users link on the left side of the IAM console and then the Add user button.

Type in a descriptive user name and make sure the “Programmatic access” checkbox is checked.

Setting permissions

The next step in the process is to configure the user’s permissions. Click on the button that says Next: Permissions and then click on «Attach existing policies directly»:
image

Now search for the policy name you created in the previous step (in our case it’s s3-discourse-policy). When you find it, select the checkbox and click on Next: Tags, then Next: Review, then finally Create user.

Here’s the critical step: Make sure you either download the credentials (Download .csv) or you copy and paste somewhere safe both Access key ID and Secret access key values. We will need them in the next step.

Discourse configuration

Now that you’ve properly set up S3, the final step is to configure your Discourse forum. These instructions should work, but the preferred method is to use environment variables and a CDN as described in https://meta.discourse.org/t/using-object-storage-for-uploads-s3-clones/148916.

Make sure you’re logged in with an administrator account and go the Settings section in the admin panel.

Type in “S3” in the textbox on the right to display only the relevant settings:

You will need to:

  • Check the “enable s3 uploads” checkbox to activate the feature
  • Paste in both “Access Key Id” and “Secret Access Key” in their respective text fields
  • Enter =BUCKET= in the “s3 upload bucket” field

You need to append a prefix to the bucket name if you want to use the same bucket for uploads and backups.

Examples of valid bucket settings
  1. Different buckets

    • s3_upload_bucket: =BUCKET=
    • s3_backup_bucket: =BACKUPS=
  2. Different prefixes

    • s3_upload_bucket: =BUCKET=/uploads
    • s3_backup_bucket: =BUCKET=/backups
  3. Prefix for backups

    • s3_upload_bucket: =BUCKET=
    • s3_backup_bucket: =BUCKET=/backups

The “s3_region” setting is optional and defaults to “US East (N. Virginia)”. You should enter the location (eg. “EU (Frankfurt)”) that is nearest to your users for better performance. If you created the bucket manually, you’ll need to select the region you selected during the creation process.

Enjoy

That’s it. From now on, all your images will be uploaded to and served from S3.

Backups

Do you want store backups of your Discourse forum on S3 as well? Take a look at Configure automatic backups for Discourse.

Frequently Asked Questions

I reused the same bucket for uploads and backups and now backups aren’t working. What should I do?

Name of your bucket for backups

The easiest solution is to append a path to the s3_backup_bucket. Here’s an example of how your settings should look afterwards.

  • s3_upload_bucket: =BACKUPS=
  • s3_backup_bucket: =BACKUPS=/backups

You can use the S3 Console to move existing backups into the new folder.

Do I really need to use separate buckets for uploads and backups?

No, you don’t, but it’s usually the easiest way to set-up. Essentially you need to either use two different buckets or a prefix for the backup bucket. For example, the following combinations will work:

  1. Different buckets

    • s3_upload_bucket: =BUCKET=
    • s3_backup_bucket: =BACKUPS=
  2. Different prefixes

    • s3_upload_bucket: =BUCKET=/uploads
    • s3_backup_bucket: =BUCKET=/backups
  3. Prefix for backups (not recommended unless you previously reused the same bucket – see above question)

    • s3_upload_bucket: =BACKUPS=
    • s3_backup_bucket: =BACKUPS=/backups

I've enabled S3 uploads in my Discourse instance (which has been going for a while); what do I do with the existing local uploads?

To migrate your existing uploads to S3, you can do a couple of rake tasks. To perform this, you need SSH access, root permissions, and have entered the discourse app (as per Administrative Bulk Operations). Oh, and you have to set some environmental variables in app.yml. Not for the faint-hearted.

Once you have done all that you are ready for the rake tasks:

rake uploads:migrate_to_s3
rake posts:rebake

Once these are done (and the uploads are working well) you no longer need to include uploads in your backups. And as a bonus, you will be able to Restore a backup from command line in the event of catastrophe (just keep a copy of app.yml somewhere).

One-way door

Unlike many configuration decisions in Discourse, note that using S3 is a “one-way door;” that is, a move that cannot easily be reversed. There is no safe or maintained way to move files out of S3 to be in the local uploads. In particular, the migrate_to_s3 moves more than just post images to S3; files for which there is no reverse path. (For more details, see Migrate_from_s3 problems)

93 Likes
Configure automatic backups for Discourse
Configure automatic backups for Discourse
After setting up S3 - Access denied
S3 region vs. Discourse region
IAM and bucket policy for S3 access
Optimize images before uploading?
Images are disappearing off of s3!
Setting up SSL with my domain name and Discourse instance
Upload img/content via image/content host
Configure automatic backups for Discourse
Access Denied error message when trying to upload images
Setting up s3: what happens to previous uploads?
Configure an S3 compatible object storage provider for uploads
Downloading remote images disabled due to disk space
Extend S3 configuration for other S3 API compatible services
Would it be worth resizing uploaded images (to save space)?
Uploading Images stalls and does not translate to img src tag
Policy and permission issues with s3 buckets?
File Reference and Deletion, will it really be deleted?
Backups failing, and admin page inaccessible
S3: Failed to optimize image: no images defined
S3 Backup ... suspect access issue
Backups have started failing due to server time being wrong
Awareness for path dependencies when setting up a discourse forum
Hosting the pictures on external storage
Broken Images and Their S3 URLs
Image upload error: The bucket does not allow ACL's
Can I increase the 1024000 kb limit on attachment?
How to develop discourse in a team?
Configure automatic backups for Discourse
Automatic backup not working
Automatic backup not working
How to run rake tasks?
S3 Bucket objects restricted access policy as per Discourses groups
Can't upload PDF to S3
Broken Images and Their S3 URLs
Problem with Backblaze for backup- Failed to list backups from S3: Signature validation failed
AWS S3 Object Ownership
Error in rebuilding using minio as object store
Issues with changing File/Image upload location to S3 Server from local storage
Configure an S3 compatible object storage provider for uploads
MKJ's Opinionated Discourse Deployment Configuration
S3 bucket policy current example
How might we better structure #howto?
[SOLVED] My forum stop working... could not do Rebuild App
Please explain how hosting photos within the forum works. Does that get crazy expensive?
Problems with Patreon Login, Force HTTPS, and S3 CDN (three) Issues
Cron task to sync local backups to DigitalOcean Spaces
Where to Upload a File
Configure an S3 compatible object storage provider for uploads
Configure automatic backups for Discourse
Help restoring - system hung at midnight
S3 backups for multisite show up in root directory of bucket
Upload objects to private S3 is not working
Placeholder Forms
Configure an S3 compatible object storage provider for uploads
Upload objects to private S3 is not working
Best Practices for Backups
Configure an S3 compatible object storage provider for uploads
Digital file repository in forum
S3 migrating certs- how does this affect Discourse sites?
Backups failing, and admin page inaccessible
Policy and permission issues with s3 buckets?
S3 Backup ... suspect access issue
Policy and permission issues with s3 buckets?
More descriptive warning when recategorising a topic into a category with moderator approval
Configure an S3 compatible object storage provider for uploads
Policy and permission issues with s3 buckets?
Install Discourse on Amazon Web Services (AWS) with Cloudflare
Using s3: "backup with uploads" settings
After setting up S3 - Access denied
Image uploads path in posts will not change after rebake and remap
Incorrect S3 warning in admin menu

Close! ‘upload’ has an ‘s’. So it is:

rake uploads:migrate_to_s3

This seems to need all of the S3 stats as enviromental variables in containers/app.yml as per Best Practices for Backups, with the addition of the uploads bucket. Scary stuff for a relatively green admin like me but I took a deep breath and have done so.

In order to make this work, I’ve put this in the env section of containers/app.yml:

  DISCOURSE_S3_ACCESS_KEY_ID: 'key'
  DISCOURSE_S3_SECRET_ACCESS_KEY: 'secret'
  DISCOURSE_BACKUP_LOCATION: 's3'
  DISCOURSE_ENABLE_S3_UPLOADS: true
  DISCOURSE_S3_BUCKET: 'my-uploads-bucket'
  DISCOURSE_S3_BACKUP_BUCKET: 'my-backup-bucket'
  DISCOURSE_S3_REGION: 'us-west-1'

This should be in the OP, eh? done

5 Likes

8 posts were merged into an existing topic: Configure automatic backups for Discourse

Getting some errors setting this up. Seems some parts of this template are outdated?

2 Likes

Unfortunately this does not work. I have 10 full backs ups and I can’t restore any of them, they’re all giving me the same errors:

Reconnecting to the database...
Reloading site settings...
Disabling outgoing emails for non-staff users...
Disabling readonly mode...
Clearing category cache...
Reloading translations...
Remapping uploads...
Restoring uploads, this may take a while...
Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
 => 5 files
 - Listing S3 files
.. => 1911 files
 - Syncing files to S3
.....
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
285 posts were flagged for a rebake
EXCEPTION: 509 of 1823 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:373:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:66:in `migrate'
/var/www/discourse/lib/file_store/s3_store.rb:328:in `copy_from'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:62:in `restore_uploads'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:44:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:61:in `run'
script/discourse:149:in `restore'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
script/discourse:290:in `<top (required)>'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/cli.rb:483:in `exec'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/cli.rb:31:in `dispatch'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/cli.rb:25:in `start'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/exe/bundle:48:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.13/exe/bundle:36:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Trying to rollback...

Why is this happening? I’ve seen many folks report the issue but there’s no solution that I’ve come across as yet. Backup and restore is a very fundamental need.

2 Likes

@zogstrip the instructions on the links above seem to be missing. I think they were linked to a post which was moved to a different topic, so they don’t point to anything at this time. Can you please outline what needs to be done to complete the migration of local uploads to S3?

Once that migration is compete my understanding is that now we can do a complete backup/restore from S3?

2 Likes

The link is/was pointing to this post:

I’ll check it over in more detail later and see if anything needs adding to the OP. :+1:

3 Likes

the policy is not working, might want to update it

Hi @GripenANM :slight_smile:

Could you give some details on which bit didn’t work for you, and what you think needs changing?

the headBucket in the policy Action is giving me errors, and if I change it to list bucket it is giving me permission error when running the migration

It seems the IAM JSON referenced here isn’t accepted as is by AWS. I get the following error:

Ln 27, Col 11 Invalid Action: The action s3:HeadBucket does not exist. Did you mean s3:ListBucket? The API called HeadBucket authorizes against the IAM action s3:ListBucket. Learn more

I think where it reads "s3:HeadBucket" it should read instead "s3:ListBucket", so the new JSON config would be:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
               "s3:List*",
               "s3:Get*",
               "s3:AbortMultipartUpload",
               "s3:DeleteObject",
               "s3:PutObject",
               "s3:PutObjectAcl",
               "s3:PutObjectVersionAcl",
               "s3:PutLifecycleConfiguration",
               "s3:CreateBucket",
               "s3:PutBucketCORS"
      ],
      "Resource": [
        "arn:aws:s3:::your-uploads-bucket",
        "arn:aws:s3:::your-uploads-bucket/*"
      ]
    },
    {
       "Effect": "Allow",
       "Action": [
           "s3:ListAllMyBuckets",
           "s3:ListBucket"
       ],
       "Resource": "*"
    }
  ]
}

I am not particularly familiar with AWS so I would appreciate some validation here :slight_smile:

Thanks!

2 Likes

@Falco does that sound right to you? Should we update the OP accordingly?

2 Likes