Setting up file and image uploads to S3


(Régis Hanol) #1

So, you want to use S3 to handle image uploads? Here’s the definitive guide:

S3 registration

Head over to https://aws.amazon.com/free/ and click on Create a Free Account

During the create account process, make sure you provide payment information, otherwise you won’t be able to use S3. There’s no registration fee, you will only be charged for what you use, if you exceed the AWS Free Usage Tier.

Bucket

You don’t need to create the S3 buckets manually. Discourse will automagically create them for you if they do not exist. :wink:

However, if you really want to create the S3 buckets yourself, please pay attention to the following notes:

  • The bucket name should not contain periods as this will cause huge HTTPS problems for you.

  • When you set up the permissions, make sure that you allow public ACLs, otherwise uploads will fail.
    image

User creation

Creating a user account

Sign in to AWS Management Console and search for the “IAM” service to access the AWS Identity and Access Management (IAM) console which enables you to manage access to your AWS resources.

We need to create a user account, so click on the Users link on the left side and then the Add user button. Type in a descriptive user name and make sure the “Programmatic access” checkbox is checked.

Here’s the critical step: Make sure you either download the credentials or you copy and paste somewhere safe both Access key ID and Secret access key values. We will need them later.

Setting permissions

Once the user is created, we need to configure the user’s permission. Select the user you’ve just created in the upper panel, click on the Permissions tab in the lower panel and then click the Add inline policy link.

Click on the JSON tab and use the following piece of code as a template for your policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
               "s3:List*",
               "s3:Get*",
               "s3:AbortMultipartUpload",
               "s3:DeleteObject",
               "s3:PutObject",
               "s3:PutObjectAcl",
               "s3:PutObjectVersionAcl",
               "s3:PutLifecycleConfiguration",
               "s3:CreateBucket",
               "s3:PutBucketCORS"
      ],
      "Resource": [
        "arn:aws:s3:::name-of-your-bucket",
        "arn:aws:s3:::name-of-your-bucket/*"
      ]
    },
    {
       "Effect": "Allow",
       "Action": [
           "s3:ListAllMyBuckets",
           "s3:HeadBucket"
       ],
       "Resource": "*"
    }
  ]
}

First, some warnings about your bucket name:

Make sure you replace both occurrences of “name-of-your-bucket” with the name of the bucket you will use for your Discourse instance before applying the policy.

Discourse configuration

Now that you’ve properly set up S3, the final step is to configure your Discourse forum. Make sure you’re logged in with an administrator account and go the Settings section in the admin panel.

Type in “S3” in the textbox on the right to display only the relevant settings:

You will need to:

  • Check the “enable s3 uploads” checkbox to activate the feature
  • Paste in both “Access Key Id” and “Secret Access Key” in their respective text fields
  • Enter the name of the bucket you’ve authorized in the “s3 upload bucket

You need to append a prefix to the bucket name if you want to use the same bucket for uploads and backups.

Examples of valid bucket settings
  1. Different buckets

    • s3_upload_bucket: name-of-your-upload-bucket
    • s3_backup_bucket: name-of-your-backup-bucket
  2. Different prefixes

    • s3_upload_bucket: name-of-your-bucket/uploads
    • s3_backup_bucket: name-of-your-bucket/backups
  3. Prefix for backups

    • s3_upload_bucket: name-of-your-bucket
    • s3_backup_bucket: name-of-your-bucket/backups

The “s3_region” setting is optional and defaults to “US East (N. Virginia)”. You should enter the location (eg. “EU (Frankfurt)”) that is nearest to your users for better performance. If you created the bucket manually, you’ll need to select the region you selected during the creation process.

Enjoy

That’s it. From now on, all your images will be uploaded to and served from S3.

Backups

Do you want store backups of your Discourse forum on S3 as well? Take a look at Configure automatic backups for Discourse.

Frequently Asked Questions

I reused the same bucket for uploads and backups and now backups aren’t working. What should I do?

The easiest solution is to append a path to the s3_backup_bucket. Here’s an example of how your settings should look afterwards.

  • s3_upload_bucket: my-bucket
  • s3_backup_bucket: my-bucket/backups

You can use the S3 Console to move existing backups into the new folder.


Configure automatic backups for Discourse
S3 region vs. Discourse region
Minimum S3 IAM Role Actions?
Migration of system stored images to S3 after configuration change
Configure automatic backups for Discourse
Configure automatic backups for Discourse
How does i use some cloud file system in discourse?
S3 Uploads / IAM user / backups questions
IAM and bucket policy for S3 access
After setting up S3 - Access denied
Downloading remote images disabled due to disk space
Backups failing, and admin page inaccessible
Unable to download backup (v2.2.0.beta5)
Awareness for path dependencies when setting up a discourse forum
Setting up SSL with my domain name and Discourse instance
Would it be worth resizing uploaded images (to save space)?
File Reference and Deletion, will it really be deleted?
Uploading Images stalls and does not translate to img src tag
Strange issues with s3
Strange issues with s3
S3 Backup ... suspect access issue
Strange issues with s3
S3 Backup ... suspect access issue
Images are disappearing off of s3!
Extend S3 configuration for other S3 API compatible services
After setting up S3 - Access denied
Upload img/content via image/content host
Strange issues with s3
(ComputerDruid) #2

Note that the free usage tier only lasts for 12 months, so either don’t forget about it and leave it active, or create an alert in AWS to have it notify you when it charges you money.


(Dave Jensen) #3

Hitting the like button wasn’t enough, I just have to saw this is awesome. Over at Axis and Allies .org Forums: Axis and Allies.org Boards - Index (which is down right now :disappointed:) people have uploaded tens of thousands of files and it’s unmanageable with old style forums.

So thank you thank you thank you.


(Régis Hanol) #4

Just curious, what makes it unmanageable?


(Dave Jensen) #5

On SMF there’s a bunch of reasons:

  • All files are uploaded into a single directory
  • Files cannot be moved via the OS without breaking things
  • Filenames are generated

(Sam Saffron) #6

To me this seems like a sensible constraint, only other 2 options are

  • Store file hashes and run recovery jobs that figure out where files really are based on a hash and a full scan of the filesystem.
  • Store attachements in the db, which is a world of pain.

(Dave Jensen) #7

I agree, I probably wasn’t specific enough. SMF doesn’t really allow you to create your own directory structure. If you write a script to move a file and update the database, it still won’t find it. There’s a bunch of hardcoded crap in the code.


(Sam Saffron) #8

Our design should be safe for moving the install, our tables really should only store relative locations.


(Rikki Tooley) #9

I followed these instructions to the letter and I’m getting “Sorry, there was an error uploading that file. Please try again.” every time.

Anyone got any ideas?


(Régis Hanol) #10

Have you got any errors in the logs?


(Rikki Tooley) #11

There are some errors in production_errors.log, Amazon returns 403 “The request signature we calculated does not match the signature you provided. Check your key and signing method.” It sounds like I put in the keys wrong, but I’ve checked that…


(Régis Hanol) #12

What’s your S3 user policy?


(Rikki Tooley) #13

This:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::discourse-4mix",
        "arn:aws:s3:::discourse-4mix/*"
      ]
    }
  ]
}

Sidenote: how do you get syntax highlighting on code blocks (like in the op?)


(Régis Hanol) #14

Did you figure it out?

The highlighting engine is unfortunately not smart enought to detect that your code block is javascript. You can force it using GitHub’s fenced code blocks.


(Jonathan Allard) #15

I have the same “signature” problem.

Edit: I just changed the keys, and it went away. Maybe I had a trailing space? @haiku would you check yours?


(Lgkh) #17

Faced a problem below…

Installed latest discourse today. Setup S3 using the guide above but when creating new topic with image upload, it still loads from local /uploads folder.

So I checked Amazon S3 console, there was no bucket created.
Tried creating bucket manually, but same issue.
Checked keys such that there is no trailing space.
Checked logs, there was no errors relating to S3.

How can I debug this?

Discourse Version: 0.9.8.11
Git Version: a1b501c3fba126a3bc1705bea69a6397196b396e

UPDATED:
It is working now, guess it was some sort of delay/cache?


(Joegoggins) #18

Does anyone have any suggestions about how to migrate file system stored images into S3?

If I need to write a script to make it happen that’s cool (and I’d be happy to share it), just figured I’d ask here before diving into it.


(Phil Nelson) #19

I would also like some kind of howto for this.


(Geoff Bowers) #20

Might want to add to the howto a recommendation to not use dots in the bucket name. This is allowed by AWS but prevents you from referencing the bucket under SSL.

I came a cropper over this one earlier:


(Jeff Atwood) #21

Very good point, done in the code and here too. We’ve had 5-6 reports of bucket problems due to periods in the name, something we definitely want to avoid in the future.