Setting up file and image uploads to S3


(Régis Hanol) #1

So, you want to use S3 to handle image uploads? Here’s the definitive guide:

S3 registration

Head over to Amazon Simple Storage Service (S3) — Cloud Storage — AWS and click on .

During the create account process, make sure you provide payment information, otherwise you won’t be able to use S3. There’s no registration fee, you will only be charged for what you use, if you exceed the free usage tier.

User creation

Creating a user account

Sign in to AWS Management Console and click on to access the AWS Identity and Access Management (IAM) console which enables you to manage access to your AWS resources.

We need to create a user account, so click on the Users link on the left handside and then the button. Type in a descriptive user name and make sure the “Generate an access key for each User” checkbox is checked.

Here’s the critical step: make sure you either download the credentials or you copy and paste somewhere safe both Access Key ID and Secret Access Key values. We will need them later.

Setting permissions

Once the user is created, we need to set him permission. Select the user you’ve just created in the upper panel, click on the Permissions tab in the lower panel and then click the button.

In the “Manage User Permissions” popup, select the radio button and click the select button to manually enter the permission.

Type in a descriptive name for the policy and use the following piece of code as a template for your policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::name-of-your-bucket",
        "arn:aws:s3:::name-of-your-bucket/*"
      ]
    }
  ]
}

First, some warnings about your bucket name:

Make sure you change both references to “name-of-your-bucket” with the name of the bucket you will use for your Discourse instance before applying the policy.

Discourse configuration

Now that you’ve properly set up S3, the final step is to configure your Discourse forum. Make sure you’re logged in with an administrator account and go the Settings section in the admin panel.

Type in “S3” in the textbox on the right to display only the relevant settings:

You will need to:

  • Check the “enable_s3_uploads” checkbox to activate the feature
  • Paste in both “Access Key Id” and “Secret Access Key” in their respective text fields
  • Enter the name of the bucket you’ve authorized in the “s3 upload bucket

The “region” setting is optional and defaults to “us-east-1”. You should enter the location (eg. eu-west-1, sa-east-1, etc…) that is nearest to your users for better performances.

Enjoy

That’s it. From now on, all your images will be uploaded to and served from S3.

Note how you did not have to create your S3 bucket? That’s because Discourse will automagically create it for you if it doesn’t already exists. :wink:


Configure automatic backups for Discourse
S3 region vs. Discourse region
Install Discourse on Amazon WS with Cloudflare
Minimum S3 IAM Role Actions?
Configure automatic backups for Discourse
Downloading remote images disabled due to disk space
Migration of system stored images to S3 after configuration change
How does i use some cloud file system in discourse?
Awareness for path dependencies when setting up a discourse forum
S3 Uploads / IAM user / backups questions
Uploading Images stalls and does not translate to img src tag
Setting up SSL with my domain name and Discourse instance
Would it be worth resizing uploaded images (to save space)?
File Reference and Deletion, will it really be deleted?
(ComputerDruid) #2

Note that the free usage tier only lasts for 12 months, so either don’t forget about it and leave it active, or create an alert in AWS to have it notify you when it charges you money.


(Dave Jensen) #3

Hitting the like button wasn’t enough, I just have to saw this is awesome. Over at Axis and Allies .org Forums: Axis and Allies.org Boards - Index (which is down right now :disappointed:) people have uploaded tens of thousands of files and it’s unmanageable with old style forums.

So thank you thank you thank you.


(Régis Hanol) #4

Just curious, what makes it unmanageable?


(Dave Jensen) #5

On SMF there’s a bunch of reasons:

  • All files are uploaded into a single directory
  • Files cannot be moved via the OS without breaking things
  • Filenames are generated

(Sam Saffron) #6

To me this seems like a sensible constraint, only other 2 options are

  • Store file hashes and run recovery jobs that figure out where files really are based on a hash and a full scan of the filesystem.
  • Store attachements in the db, which is a world of pain.

(Dave Jensen) #7

I agree, I probably wasn’t specific enough. SMF doesn’t really allow you to create your own directory structure. If you write a script to move a file and update the database, it still won’t find it. There’s a bunch of hardcoded crap in the code.


(Sam Saffron) #8

Our design should be safe for moving the install, our tables really should only store relative locations.


(Rikki Tooley) #9

I followed these instructions to the letter and I’m getting “Sorry, there was an error uploading that file. Please try again.” every time.

Anyone got any ideas?


(Régis Hanol) #10

Have you got any errors in the logs?


(Rikki Tooley) #11

There are some errors in production_errors.log, Amazon returns 403 “The request signature we calculated does not match the signature you provided. Check your key and signing method.” It sounds like I put in the keys wrong, but I’ve checked that…


(Régis Hanol) #12

What’s your S3 user policy?


(Rikki Tooley) #13

This:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::discourse-4mix",
        "arn:aws:s3:::discourse-4mix/*"
      ]
    }
  ]
}

Sidenote: how do you get syntax highlighting on code blocks (like in the op?)


(Régis Hanol) #14

Did you figure it out?

The highlighting engine is unfortunately not smart enought to detect that your code block is javascript. You can force it using GitHub’s fenced code blocks.


(Jonathan Allard) #15

I have the same “signature” problem.

Edit: I just changed the keys, and it went away. Maybe I had a trailing space? @haiku would you check yours?


(Lgkh) #17

Faced a problem below…

Installed latest discourse today. Setup S3 using the guide above but when creating new topic with image upload, it still loads from local /uploads folder.

So I checked Amazon S3 console, there was no bucket created.
Tried creating bucket manually, but same issue.
Checked keys such that there is no trailing space.
Checked logs, there was no errors relating to S3.

How can I debug this?

Discourse Version: 0.9.8.11
Git Version: a1b501c3fba126a3bc1705bea69a6397196b396e

UPDATED:
It is working now, guess it was some sort of delay/cache?


(Joegoggins) #18

Does anyone have any suggestions about how to migrate file system stored images into S3?

If I need to write a script to make it happen that’s cool (and I’d be happy to share it), just figured I’d ask here before diving into it.


(Phil Nelson) #19

I would also like some kind of howto for this.


(Geoff Bowers) #20

Might want to add to the howto a recommendation to not use dots in the bucket name. This is allowed by AWS but prevents you from referencing the bucket under SSL.

I came a cropper over this one earlier:


(Jeff Atwood) #21

Very good point, done in the code and here too. We’ve had 5-6 reports of bucket problems due to periods in the name, something we definitely want to avoid in the future.