# Set up file and image uploads to S3 **URL:** https://meta.discourse.org/t/set-up-file-and-image-uploads-to-s3/7229 **Category:** Self-Hosting **Tags:** configuring, how-to **Created:** [6월 6, 2013, 7:37오후 UTC](https://meta.discourse.org/t/set-up-file-and-image-uploads-to-s3/7229 "2013-06-06T19:37:43Z") **Posts on this page:** 1 **Showing post:** 1
### Author: ![Discourse](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/discourse/32/148734_2.png) [@Discourse](https://meta.discourse.org/u/Discourse) #### Post date: [6월 6, 2013, 7:37오후 UTC](https://meta.discourse.org/t/set-up-file-and-image-uploads-to-s3/7229/1 "2013-06-06T19:37:43Z")
So, you want to use S3 to handle image uploads? Here’s the _definitive_ guide, but also see [Configure an S3 compatible object storage provider for uploads](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916) to see how to configure your app.yml. ## S3 registration Head over to [https://aws.amazon.com/free/](https://aws.amazon.com/free/) and click on ![Create a Free Account](https://global.discourse-cdn.com/meta/original/3X/5/2/52347dfe5f0b7652db9c59085032728eb1edd27e.png) During the create account process, make sure you provide payment information, otherwise you won’t be able to use S3. There’s no registration fee, you will only be charged for [what you use](https://aws.amazon.com/s3/pricing/), if you exceed the [AWS Free Usage Tier](https://aws.amazon.com/free/). ## Bucket Go to [S3](https://s3.console.aws.amazon.com/s3/home?region=eu-west-1) and click on **Create bucket** , then fill out the **Bucket name**. Remember this name because we’ll need it for the next step. - The bucket name should _not_ contain periods as this will cause [huge HTTPS problems for you](https://shlomoswidler.com/2009/08/amazon-s3-gotcha-using-virtual-host.html). - The bucket name should be [all lowercase](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html) Name of your bucket Select a **Region**. You should enter the location (eg. “EU (Frankfurt)”) that is nearest to your users for better performance. Scroll down a little until you get to the **Permissions** panel. > ⚠ Since April 2023, new S3 buckets have ACLs disabled by default (“Bucket owner enforced”). You have two options: > > **Option A — ACLs disabled (simpler):** Keep the AWS defaults (ACLs disabled, all public access blocked). Later, set `s3_use_acls` to `false` in Discourse. Uploads will be served via presigned URLs or a CDN. You can also omit `s3:PutObjectAcl` and `s3:PutObjectVersionAcl` from the IAM policy below. > > **Option B — ACLs enabled (legacy, Discourse default):** Change **Object Ownership** to **“ACLs enabled”** (choose “Bucket owner preferred”). Then uncheck **Block all public access** and check the bottom two checkboxes. You’ll also have to acknowledge that your settings may make the bucket public. ![AWS-bucket-access](https://global.discourse-cdn.com/meta/original/3X/7/2/72401448608da2b7faf2b84c371c985b93ea5c0b.png) ## User creation ### Creating a policy Sign in to [AWS Management Console](https://console.aws.amazon.com/) and search for the “ **IAM** ” service to access the [AWS Identity and Access Management (IAM)](https://console.aws.amazon.com/iam/home#/policies) console which enables you to manage access to your AWS resources. First, click Policies in the sidebar. Then, click on **Create Policy** and choose the JSON tab: ![image](https://global.discourse-cdn.com/meta/original/3X/f/1/f14734efce0f60c836e144be7157cb8c946a7f5f.png) Use the following piece of code as a template for your policy document: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:List*", "s3:Get*", "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:PutObject", "s3:PutObjectAcl", "s3:PutObjectVersionAcl", "s3:PutLifecycleConfiguration", "s3:PutBucketCORS" ], "Resource": [ "arn:aws:s3:::=BUCKET=", "arn:aws:s3:::=BUCKET=/*" ] }, { "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets", "s3:ListBucket" ], "Resource": "*" } ] } ``` > ⚠ Make sure that these two lines contain the actual name of your bucket. 😊 > > > ![image](https://global.discourse-cdn.com/meta/original/3X/5/f/5fa20c06b21ec84c6d6d087a662d6b353fd7e136.png) > > 🌟 If you also intend to do S3 backups, you can include your backup bucket here too like this: > > > ![image](https://global.discourse-cdn.com/meta/original/3X/3/9/39feef9d647cd55f800c480a79edfe2a4890d04e.png) > ℹ If you chose **Option A** above (ACLs disabled), you can remove `s3:PutObjectAcl` and `s3:PutObjectVersionAcl` from the policy since they are only needed when `s3_use_acls` is enabled. Then click on **Review Policy** and fill out the **Name** field. For example, you can call it `s3-discourse-policy` Then click on **Create Policy** at the bottom. ### Creating a user account Now that you’ve created the right policy, you’re ready to create the user account. Click on the **Users** link on the left side of the [IAM console](https://console.aws.amazon.com/iam/home#/users) and then the ![Add user](https://global.discourse-cdn.com/meta/original/3X/c/f/cf00df33f3a169aa52ec49951ff6787464552c80.png) button. Type in a descriptive user name and make sure the “ **Programmatic access** ” checkbox is checked. ### Setting permissions The next step in the process is to configure the user’s permissions. Click on the button that says **Next: Permissions** and then click on «Attach existing policies directly»: ![image](https://global.discourse-cdn.com/meta/original/3X/7/c/7c22bf4cc492caea147d1da679825b9aa1bdd97b.png) Now search for the policy name you created in the previous step (in our case it’s `s3-discourse-policy`). When you find it, select the checkbox and click on **Next: Tags** , then **Next: Review** , then finally **Create user**. Here’s the critical step: Make sure you either download the credentials (Download .csv) or you copy and paste somewhere safe both **Access key ID** and **Secret access key** values. We will need them in the next step. ## Discourse configuration Now that you’ve properly set up S3, the final step is to configure your Discourse forum. These instructions should work, but **the preferred method is to use environment variables and a CDN** as described in [Configure an S3 compatible object storage provider for uploads](https://meta.discourse.org/t/configure-an-s3-compatible-object-storage-provider-for-uploads/148916). Make sure you’re logged in with an administrator account and go the **Settings** section in the admin panel. Type in “S3” in the textbox on the right to display only the relevant settings: ![image](https://global.discourse-cdn.com/meta/original/3X/a/d/ad46687ba7e291e3ba944dbfb3a2da253c1f0099.png) You will need to: - Check the “`enable s3 uploads`” checkbox to activate the feature - Paste in both “`Access Key Id`” and “`Secret Access Key`” in their respective text fields - Enter `=BUCKET=` in the “`s3 upload bucket`” field - If you chose **Option A** (ACLs disabled) for your bucket, set `s3_use_acls` to **false** You need to append a prefix to the bucket name if you want to use the same bucket for uploads and backups. > **Examples of valid bucket settings** > > 1. Different buckets > > 2. Different prefixes > > 3. Prefix for backups The “`s3_region`” setting is optional and defaults to “`US East (N. Virginia)`”. You should enter the location (eg. “EU (Frankfurt)”) that is nearest to your users for better performance. If you created the bucket manually, you’ll need to select the region you selected during the creation process. ## Enjoy That’s it. From now on, all your images will be uploaded to and served from S3. ## Backups Do you want store backups of your Discourse forum on S3 as well? Take a look at [Configure automatic backups for Discourse](https://meta.discourse.org/t/configure-automatic-backups-for-discourse/14855). ## Frequently Asked Questions ### I reused the same bucket for uploads and backups and now backups aren’t working. What should I do? Name of your bucket for backups The easiest solution is to append a path to the `s3_backup_bucket`. Here’s an example of how your settings should look afterwards. - **`s3_upload_bucket`:** `=BACKUPS=` - **`s3_backup_bucket`:** `=BACKUPS=/backups` You can use the [S3 Console](https://s3.console.aws.amazon.com/s3) to [move existing backups](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CopyingAnObject.html) into the new folder. ### Do I really need to use separate buckets for uploads and backups? No, you don’t, but it’s usually the easiest way to set-up. Essentially you need to either use two different buckets or a prefix for the backup bucket. For example, the following combinations will work: 1. Different buckets 2. Different prefixes 3. Prefix for backups (not recommended unless you previously reused the same bucket – see above question) ### I’ve enabled S3 uploads in my Discourse instance (which has been going for a while); what do I do with the existing local uploads? To migrate your existing uploads to S3, you can do a couple of **rake tasks**. To perform this, you need SSH access, root permissions, and have entered the discourse app (as per [Administrative Bulk Operations](https://meta.discourse.org/t/administrative-bulk-operations/118349)). Oh, and you [have to set some environmental variables in app.yml](https://meta.discourse.org/t/setting-up-file-and-image-uploads-to-s3/7229/175). Not for the faint-hearted. Once you have done all that you are ready for the rake tasks: ```plaintext rake uploads:migrate_to_s3 ``` Once that is done, posts that need to be rebaked will be marked accordingly and will be rebaked by a regular task. If you have resources and are in a hurry, you can issue this command (which is recommended by the above rake task): ```plaintext rake posts:rebake_uncooked_posts ``` Once the posts are all rebaked (and the uploads are working well) you no longer need to include uploads in your backups. And as a bonus, you will be able to [Restore a backup from the command line](https://meta.discourse.org/t/restore-a-backup-from-command-line/108034) in the event of catastrophe (just keep a copy of app.yml somewhere). ## One-way door Unlike many configuration decisions in Discourse, note that using S3 is a “one-way door;” that is, a move that _cannot easily be reversed._ There is no safe or maintained way to move files out of S3 to be in the local uploads. In particular, the `migrate_to_s3` moves more than just post images to S3; files for which there is no reverse path. (For more details, see [Migrate\_from\_s3 problems](https://meta.discourse.org/t/migrate-from-s3-problems/119064)). ## Backing up your S3 assets Using versioning, or syncing to a different region are all good strategies. > Last edited by @JammyDodger 2024-05-25T10:58:50Z > > > **Check document** > > > > Perform check on document: --- _[View the full topic](https://meta.discourse.org/t/set-up-file-and-image-uploads-to-s3/7229)._