Setting up file and image uploads to S3

So, you want to use S3 to handle image uploads? Here’s the definitive guide:

S3 registration

Head over to https://aws.amazon.com/free/ and click on Create a Free Account

During the create account process, make sure you provide payment information, otherwise you won’t be able to use S3. There’s no registration fee, you will only be charged for what you use, if you exceed the AWS Free Usage Tier.

Bucket

Go to S3 and click on Create bucket, then fill out the Bucket name. Remember this name because we’ll need it for the next step.

Name of your bucket

Select a Region. You should enter the location (eg. “EU (Frankfurt)”) that is nearest to your users for better performance.

Scroll down a little until you get to the Permissions panel.

  • When you set up the permissions, make sure that you allow public ACLs, otherwise uploads will fail. Uncheck Block all public access and check the bottom two checkboxes. You’ll also have to acknowledge that your settings may make the bucket public in the warning at the top.

User creation

Creating a policy

Sign in to AWS Management Console and search for the “IAM” service to access the AWS Identity and Access Management (IAM) console which enables you to manage access to your AWS resources.

First, click Policies in the sidebar. Then, click on Create Policy and choose the JSON tab:

image

Use the following piece of code as a template for your policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
               "s3:List*",
               "s3:Get*",
               "s3:AbortMultipartUpload",
               "s3:DeleteObject",
               "s3:PutObject",
               "s3:PutObjectAcl",
               "s3:PutObjectVersionAcl",
               "s3:PutLifecycleConfiguration",
               "s3:CreateBucket",
               "s3:PutBucketCORS"
      ],
      "Resource": [
        "arn:aws:s3:::name-of-your-uploads-bucket",
        "arn:aws:s3:::name-of-your-uploads-bucket/*"
      ]
    },
    {
       "Effect": "Allow",
       "Action": [
           "s3:ListAllMyBuckets",
           "s3:HeadBucket"
       ],
       "Resource": "*"
    }
  ]
}

:warning: Make sure that these two lines contain the actual name of your bucket. :blush:

image

:star2: If you also intend to do S3 backups, you can include your backup bucket here too like this:

image

Then click on Review Policy and fill out the Name field. For example, you can call it s3-discourse-policy

Then click on Create Policy at the bottom.

Creating a user account

Now that you’ve created the right policy, you’re ready to create the user account. Click on the Users link on the left side of the IAM console and then the Add user button.

Type in a descriptive user name and make sure the “Programmatic access” checkbox is checked.

Setting permissions

The next step in the process is to configure the user’s permissions. Click on the button that says Next: Permissions and then click on «Attach existing policies directly»:
image

Now search for the policy name you created in the previous step (in our case it’s s3-discourse-policy). When you find it, select the checkbox and click on Next: Tags, then Next: Review, then finally Create user.

Here’s the critical step: Make sure you either download the credentials (Download .csv) or you copy and paste somewhere safe both Access key ID and Secret access key values. We will need them in the next step.

Discourse configuration

Now that you’ve properly set up S3, the final step is to configure your Discourse forum. These instructions should work, but the preferred method is to use environment variables and a CDN as described in Using Object Storage for Uploads (S3 & Clones).

Make sure you’re logged in with an administrator account and go the Settings section in the admin panel.

Type in “S3” in the textbox on the right to display only the relevant settings:

You will need to:

  • Check the “enable s3 uploads” checkbox to activate the feature
  • Paste in both “Access Key Id” and “Secret Access Key” in their respective text fields
  • Enter =BUCKET= in the “s3 upload bucket” field

You need to append a prefix to the bucket name if you want to use the same bucket for uploads and backups.

Examples of valid bucket settings
  1. Different buckets

    • s3_upload_bucket: =BUCKET=
    • s3_backup_bucket: =BACKUPS=
  2. Different prefixes

    • s3_upload_bucket: =BUCKET=/uploads
    • s3_backup_bucket: =BUCKET=/backups
  3. Prefix for backups

    • s3_upload_bucket: =BUCKET=
    • s3_backup_bucket: =BUCKET=/backups

The “s3_region” setting is optional and defaults to “US East (N. Virginia)”. You should enter the location (eg. “EU (Frankfurt)”) that is nearest to your users for better performance. If you created the bucket manually, you’ll need to select the region you selected during the creation process.

Enjoy

That’s it. From now on, all your images will be uploaded to and served from S3.

Backups

Do you want store backups of your Discourse forum on S3 as well? Take a look at Configure automatic backups for Discourse.

Frequently Asked Questions

I reused the same bucket for uploads and backups and now backups aren’t working. What should I do?

Name of your bucket for backups

The easiest solution is to append a path to the s3_backup_bucket. Here’s an example of how your settings should look afterwards.

  • s3_upload_bucket: =BACKUPS=
  • s3_backup_bucket: =BACKUPS=/backups

You can use the S3 Console to move existing backups into the new folder.

Do I really need to use separate buckets for uploads and backups?

No, you don’t, but it’s usually the easiest way to set-up. Essentially you need to either use two different buckets or a prefix for the backup bucket. For example, the following combinations will work:

  1. Different buckets

    • s3_upload_bucket: =BUCKET=
    • s3_backup_bucket: =BACKUPS=
  2. Different prefixes

    • s3_upload_bucket: =BUCKET=/uploads
    • s3_backup_bucket: =BUCKET=/backups
  3. Prefix for backups (not recommended unless you previously reused the same bucket – see above question)

    • s3_upload_bucket: =BACKUPS=
    • s3_backup_bucket: =BACKUPS=/backups

I've enabled S3 uploads; what do I do with the existing local uploads?

To migrate your existing uploads to S3, you can do a rake task. To perform this, you need SSH access, root permissions, and have entered the discourse app (as per Administrative Bulk Operations). Oh, and you have to set some environmental variables in app.yml. Not for the faint-hearted.

Once you have done all that you are ready for the rake task:

rake uploads:migrate_to_s3

Once this is done (and the uploads are working well) you can then you no longer need to include uploads in your backups. And as a bonus, you will be able to Restore a backup from command line in the event of catastrophe (just keep a copy of app.yml somewhere).

One-way door

Unlike many configuration decisions in Discourse, note that using S3 is a “one-way door;” that is, a move that cannot easily be reversed. There is no safe or maintained way to move files out of S3 to be in the local uploads. In particular, the migrate_to_s3 moves more than just post images to S3; files for which there is no reverse path. (For more details, see Migrate_from_s3 problems)

80 Likes
Configure automatic backups for Discourse
Configure automatic backups for Discourse
S3 region vs. Discourse region
After setting up S3 - Access denied
IAM and bucket policy for S3 access
Optimize images before uploading?
Extend S3 configuration for other S3 API compatible services
Downloading remote images disabled due to disk space
Install Discourse on Amazon WS with Cloudflare
Setting up SSL with my domain name and Discourse instance
Configure automatic backups for Discourse
Images are disappearing off of s3!
S3 Backup ... suspect access issue
Backups have started failing due to server time being wrong
Awareness for path dependencies when setting up a discourse forum
Hosting the pictures on external storage
Setting up s3: what happens to previous uploads?
Unable to download backup (v2.2.0.beta5)
S3: Failed to optimize image: no images defined
Would it be worth resizing uploaded images (to save space)?
Backups failing, and admin page inaccessible
File Reference and Deletion, will it really be deleted?
Strange issues with s3
Uploading Images stalls and does not translate to img src tag
Upload img/content via image/content host
Data is missing after restore
Digital file repository in forum
Backups failing, and admin page inaccessible
Where to Upload a File
After setting up S3 - Access denied
Image uploads path in posts will not change after rebake and remap
Using Object Storage for Uploads (S3 & Clones)
Strange issues with s3
Configure automatic backups for Discourse
Configure automatic backups for Discourse
Best Practices for Backups
Incorrect S3 warning in admin menu
S3 Backup ... suspect access issue
Broken Images and Their S3 URLs
Broken Images and Their S3 URLs
Discourse Placeholder Forms theme component
Automatic backup not working
Automatic backup not working
Access Denied error message when trying to upload images
Please explain how hosting photos within the forum works. Does that get crazy expensive?
Strange issues with s3
Strange issues with s3

I just went through:

  • Setting up S3 using IAM user & the policy outlined above
  • Setting up Cloudfront distribution pointing to S3 bucket, along with custom CDN domain + SSL cert
  • Rebaked all posts using cd /var/www/discourse && ./launcher enter app && rake posts:rebake (yes, I’m too OCD to not put discourse under www :wink: ).

A few gotchas in case they’re useful to someone:

  • Got a permission denied when I tried to get Discourse to create the bucket - reason? I was using too generic a name (forum-media) and it was denying the create request. I switched it to something unique within the IAM user, Discourse config, and it worked as expected. Can the permission denied error be more granular so users know that that bucket name is not available?
  • AWS Cloudfront doesn’t do a good job telling you how to configure the CNAME - it was fairly obvious when I saw the origin, but, just in case

Beautiful new forum now blossoming at https://forum.makesoil.org

4 Likes

I set up a small forum a while back with S3 hosting of uploads and unfortunately used .'s in the bucket name. Oops. I just got around to switching on https and now, of course, things aren’t working due to the known issues with wild cards certificates and .'s in bucket names.

What’s the best way out of this pickle? If I create a new s3 bucket all the old images are in the old one and seem to have fixed references in the markup so all previously uploaded images are broken.
Some possibilities I thought of:

  1. It looks like there is a “rebake” script that can be run to apply updates to the markdown. If I copy all the files in old bucket to the new bucket, change the bucket location in settings, and do a “rebake” will that fix the images?

another other possibility:

  1. Use some of those scripts mentioned by @zogstrip i.e a) rake migrate_from_s3. b) switch bucket to new one. c) rake migrate_to_s3?

I couldn’t find much documentation about rebaking and these rake scripts other than what i’ve been able to glean from this thread so wanted to ask the community what they thought the best way to go was.

Thanks!

1 Like

The following should work:

  • create a database backup
  • move all objects to the new S3 bucket
  • execute discourse remap old_bucket_name new_bucket_name inside the docker container
6 Likes

We have created S3 store. How to clean the unused images?

1 Like

This happens automatically, the system takes care of it.

3 Likes

In this meta topic by team, its written that storing backups and uploads on the same bucket won’t work NOW.

Is that true?

Edit: I found out that its working. But I don’t myself want to edit the OP (lest there might be some else reason there).

I’m getting an error while executing rake uploads:migrate_to_s3:

Please provide the following environment variables
  - DISCOURSE_S3_BUCKET
  - DISCOURSE_S3_REGION
  and either
  - DISCOURSE_S3_ACCESS_KEY_ID
  - DISCOURSE_S3_SECRET_ACCESS_KEY

I tried adding them to .env to no avail. Any ideas?

Note, using DO Spaces!

I have followed each step mention in tutorial and has setup S3 bucket, when I enable upload images to S3 after image gets uploaded (progress shows 100%) I am getting alert of “Access Denied”

Does anyone know who can we debug this AWS S3 upload on discourse.

1 Like

One way to work out if this is a problem with the Access Keys and IAM stuff (as opposed to a Discourse issue) is to try to upload a simple file to AWS using the CLI GitHub - awslabs/aws-shell: An integrated shell for working with the AWS CLI. using those credentials. If they don’t work from the CLI then something’s wrong with the permissions applied to those keys.

I find that configuring an IAM user specifically for Discourse backups and uploads gives some nice compartmentalisation of concerns and additional security compared to using a user with global S3 bucket access policy, especially if you run multiple Discourses.

This snippet goes in the Access Control Policy which you will create, replacing <BUCKET_NAME> with your bucket name, and apply to your IAM user. It confers access ONLY to the named bucket you specify. Note that you need to be able to access the top level bucket name and it contents /*, and be able to ‘List All Buckets’.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<BUCKET_NAME>",
                "arn:aws:s3:::<BUCKET_NAME>/*"
            ]
        }
    ]
}
7 Likes

Thank you @pacharanero, my issue was global S3 bucket policy which was blocking everything.

3 Likes

Glad it helped @deepmehtait

4 Likes

Shouldnt we add this ? :

arn:aws:s3:::accesspoint/*"

1 Like

What this error mean ?

NameError: uninitialized constant FileStore::ToS3Migration::Aws

1 Like

The problem has been fixed:

8 Likes

Hi there guys,

I’m moving a Discourse installation to a new server. The domain stays the same, we have moved all old image uploads to a S3 digital ocean spaces with RCLONE. Did a clean discourse install, uploaded a backup without images. So far so good, all posts are there.

It is working with new uploads, but i am having a hard time rerouting old uploads in posts to this new location.

The old location was domain[dot]com/uploads/
The new location with all uploads in place is a space.ams3.digitaloceanspaces[dot]com/

I have tried remapping:

discourse remap [olddomain.com/uploads/](http://olddomain.com/uploads/) [space.ams3.digitaloceanspaces.com/](http://space.ams3.digitaloceanspaces.com/)

and i have tried:

rake posts:rebake
rake posts:rebake_match[“uploads”]

The paths that old images are linking to is still the old domain path. Seems like the remap and the rebake had no effect. Am i missing something obvious here?

  • edit i just learned that migrate_to_s3 first checks if a file is already there which is great. Downside is that i see that migrate_to_s3 does not like to work with Digital Ocean spaces.
1 Like

If you could tell, If we had more than 1 bucket, can we assign all those buckets in this section of the (single) policy (two lines for each/one bucket)?

1 Like

Updated guide to include placeholder forms, as it’s really easy to get confused with all the different bucket names!

8 Likes

What is the risk involved while keeping all public access always unblocked??

1 Like

Hmm, yeah, we seem to be missing instructions for setting up CloudFront which would allow for public access to be blocked.

3 Likes