Setting up file and image uploads to S3

So, you want to use S3 to handle image uploads? Here’s the definitive guide:

S3 registration

Head over to https://aws.amazon.com/free/ and click on Create a Free Account

During the create account process, make sure you provide payment information, otherwise you won’t be able to use S3. There’s no registration fee, you will only be charged for what you use, if you exceed the AWS Free Usage Tier.

Bucket

Go to S3 and click on Create bucket, then fill out the Bucket name. Remember this name because we’ll need it for the next step.

Name of your bucket

Select a Region. You should enter the location (eg. “EU (Frankfurt)”) that is nearest to your users for better performance.

Click on Next, and Next again until you get to the Permissions panel.

  • When you set up the permissions, make sure that you allow public ACLs, otherwise uploads will fail. Uncheck Block all public access and check the bottom two checkboxes. You’ll also have to acknowledge that your settings may make the bucket public in the warning at the top.

User creation

Creating a policy

Sign in to AWS Management Console and search for the “IAM” service to access the AWS Identity and Access Management (IAM) console which enables you to manage access to your AWS resources.

First, click Policies in the sidebar. Then, click on Create Policy and choose the JSON tab:

image

Use the following piece of code as a template for your policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
               "s3:List*",
               "s3:Get*",
               "s3:AbortMultipartUpload",
               "s3:DeleteObject",
               "s3:PutObject",
               "s3:PutObjectAcl",
               "s3:PutObjectVersionAcl",
               "s3:PutLifecycleConfiguration",
               "s3:CreateBucket",
               "s3:PutBucketCORS"
      ],
      "Resource": [
        "arn:aws:s3:::=BUCKET=",
        "arn:aws:s3:::=BUCKET=/*"
      ]
    },
    {
       "Effect": "Allow",
       "Action": [
           "s3:ListAllMyBuckets",
           "s3:HeadBucket"
       ],
       "Resource": "*"
    }
  ]
}

:warning: Make sure that these two lines contain the actual name of your bucket. :blush:

image

Then click on Review Policy and fill out the Name field. For example, you can call it s3-discourse-policy

Then click on Create Policy at the bottom.

Creating a user account

Now that you’ve created the right policy, you’re ready to create the user account. Click on the Users link on the left side of the IAM console and then the Add user button.

Type in a descriptive user name and make sure the “Programmatic access” checkbox is checked.

Setting permissions

The next step in the process is to configure the user’s permissions. Click on the button that says Next: Permissions and then click on «Attach existing policies directly»:
image

Now search for the policy name you created in the previous step (in our case it’s s3-discourse-policy). When you find it, select the checkbox and click on Next: Tags, then Next: Review, then finally Create user.

Here’s the critical step: Make sure you either download the credentials (Download .csv) or you copy and paste somewhere safe both Access key ID and Secret access key values. We will need them in the next step.

Discourse configuration

Now that you’ve properly set up S3, the final step is to configure your Discourse forum. Make sure you’re logged in with an administrator account and go the Settings section in the admin panel.

Type in “S3” in the textbox on the right to display only the relevant settings:

You will need to:

  • Check the “enable s3 uploads” checkbox to activate the feature
  • Paste in both “Access Key Id” and “Secret Access Key” in their respective text fields
  • Enter =BUCKET= in the “s3 upload bucket” field

You need to append a prefix to the bucket name if you want to use the same bucket for uploads and backups.

Examples of valid bucket settings
  1. Different buckets

    • s3_upload_bucket: =BUCKET=
    • s3_backup_bucket: =BACKUPS=
  2. Different prefixes

    • s3_upload_bucket: =BUCKET=/uploads
    • s3_backup_bucket: =BUCKET=/backups
  3. Prefix for backups

    • s3_upload_bucket: =BUCKET=
    • s3_backup_bucket: =BUCKET=/backups

The “s3_region” setting is optional and defaults to “US East (N. Virginia)”. You should enter the location (eg. “EU (Frankfurt)”) that is nearest to your users for better performance. If you created the bucket manually, you’ll need to select the region you selected during the creation process.

Enjoy

That’s it. From now on, all your images will be uploaded to and served from S3.

Backups

Do you want store backups of your Discourse forum on S3 as well? Take a look at Configure automatic backups for Discourse.

Frequently Asked Questions

I reused the same bucket for uploads and backups and now backups aren’t working. What should I do?

Name of your bucket for backups

The easiest solution is to append a path to the s3_backup_bucket. Here’s an example of how your settings should look afterwards.

  • s3_upload_bucket: =BACKUPS=
  • s3_backup_bucket: =BACKUPS=/backups

You can use the S3 Console to move existing backups into the new folder.

Do I really need to use separate buckets for uploads and backups?

No, you don’t, but it’s usually the easiest way to set-up. Essentially you need to either use two different buckets or a prefix for the backup bucket. For example, the following combinations will work:

  1. Different buckets

    • s3_upload_bucket: =BUCKET=
    • s3_backup_bucket: =BACKUPS=
  2. Different prefixes

    • s3_upload_bucket: =BUCKET=/uploads
    • s3_backup_bucket: =BUCKET=/backups
  3. Prefix for backups (not recommended unless you previously reused the same bucket – see above question)

    • s3_upload_bucket: =BACKUPS=
    • s3_backup_bucket: =BACKUPS=/backups
76 Likes
Configure automatic backups for Discourse
Configure automatic backups for Discourse
S3 region vs. Discourse region
After setting up S3 - Access denied
S3 Uploads / IAM user / backups questions
Optimize images before uploading?
Images are disappearing off of s3!
Install Discourse on Amazon WS with Cloudflare
Extend S3 configuration for other S3 API compatible services
Configure automatic backups for Discourse
IAM and bucket policy for S3 access
Setting up SSL with my domain name and Discourse instance
Downloading remote images disabled due to disk space
Would it be worth resizing uploaded images (to save space)?
Upload img/content via image/content host
Uploading Images stalls and does not translate to img src tag
Strange issues with s3
File Reference and Deletion, will it really be deleted?
Backups failing, and admin page inaccessible
Data is missing after restore
S3: Failed to optimize image: no images defined
Unable to download backup (v2.2.0.beta5)
S3 Backup ... suspect access issue
Backups have started failing due to server time being wrong
Awareness for path dependencies when setting up a discourse forum
Hosting the pictures on external storage
Setting up s3: what happens to previous uploads?
Broken Images and Their S3 URLs
Discourse Placeholder Forms theme component
Automatic backup not working
Automatic backup not working
Access Denied error message when trying to upload images
Broken Images and Their S3 URLs
Please explain how hosting photos within the forum works. Does that get crazy expensive?
Where to Upload a File
Configure automatic backups for Discourse
Best Practices for Backups
Digital file repository in forum
Backups failing, and admin page inaccessible
Strange issues with s3
S3 Backup ... suspect access issue
Strange issues with s3
Strange issues with s3
After setting up S3 - Access denied
Using Object Storage for Uploads (S3 & Clones)
Image uploads path in posts will not change after rebake and remap

Note that the free usage tier only lasts for 12 months, so either don’t forget about it and leave it active, or create an alert in AWS to have it notify you when it charges you money.

6 Likes

Hitting the like button wasn’t enough, I just have to saw this is awesome. Over at http://axisandallies.org/forums (which is down right now :disappointed:) people have uploaded tens of thousands of files and it’s unmanageable with old style forums.

So thank you thank you thank you.

1 Like

Just curious, what makes it unmanageable?

1 Like

On SMF there’s a bunch of reasons:

  • All files are uploaded into a single directory
  • Files cannot be moved via the OS without breaking things
  • Filenames are generated

To me this seems like a sensible constraint, only other 2 options are

  • Store file hashes and run recovery jobs that figure out where files really are based on a hash and a full scan of the filesystem.
  • Store attachements in the db, which is a world of pain.
3 Likes

I agree, I probably wasn’t specific enough. SMF doesn’t really allow you to create your own directory structure. If you write a script to move a file and update the database, it still won’t find it. There’s a bunch of hardcoded crap in the code.

Our design should be safe for moving the install, our tables really should only store relative locations.

1 Like

I followed these instructions to the letter and I’m getting “Sorry, there was an error uploading that file. Please try again.” every time.

Anyone got any ideas?

Have you got any errors in the logs?

1 Like

There are some errors in production_errors.log, Amazon returns 403 “The request signature we calculated does not match the signature you provided. Check your key and signing method.” It sounds like I put in the keys wrong, but I’ve checked that…

What’s your S3 user policy?

1 Like

This:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::discourse-4mix",
        "arn:aws:s3:::discourse-4mix/*"
      ]
    }
  ]
}

Sidenote: how do you get syntax highlighting on code blocks (like in the op?)

Did you figure it out?

The highlighting engine is unfortunately not smart enought to detect that your code block is javascript. You can force it using GitHub’s fenced code blocks.

1 Like

I have the same “signature” problem.

Edit: I just changed the keys, and it went away. Maybe I had a trailing space? @haiku would you check yours?

1 Like

Faced a problem below…

Installed latest discourse today. Setup S3 using the guide above but when creating new topic with image upload, it still loads from local /uploads folder.

So I checked Amazon S3 console, there was no bucket created.
Tried creating bucket manually, but same issue.
Checked keys such that there is no trailing space.
Checked logs, there was no errors relating to S3.

How can I debug this?

Discourse Version: 0.9.8.11
Git Version: a1b501c3fba126a3bc1705bea69a6397196b396e

UPDATED:
It is working now, guess it was some sort of delay/cache?

Does anyone have any suggestions about how to migrate file system stored images into S3?

If I need to write a script to make it happen that’s cool (and I’d be happy to share it), just figured I’d ask here before diving into it.

8 Likes

I would also like some kind of howto for this.

Might want to add to the howto a recommendation to not use dots in the bucket name. This is allowed by AWS but prevents you from referencing the bucket under SSL.

I came a cropper over this one earlier:
https://meta.discourse.org/t/s3-backups-fail-with-sslerror/14667

5 Likes

Very good point, done in the code and here too. We’ve had 5-6 reports of bucket problems due to periods in the name, something we definitely want to avoid in the future.

4 Likes