# Schritt-für-Schritt-Anleitung zum Verschieben großer Site-Uploads nach S3? **URL:** https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321 **Category:** Support **Created:** [9. August 2018 um 16:01 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321 "2018-08-09T16:01:41Z") **Posts on this page:** 12 **Page:** 1
### Author: ![Gunnar](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gunnar/32/119647_2.png) [@Gunnar](https://meta.discourse.org/u/Gunnar) #### Post date: [9. August 2018 um 16:01 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/1 "2018-08-09T16:01:41Z")
Our site has close to 2 million posts. Our uploads directory is currently 18 GB in size. The site is active 24/7. What would be a recommended step-by-step procedure for moving our uploads to S3? Can we avoid downtime while doing this? Thanks, Gunnar ---
### Author: ![mmayo](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mmayo/32/119577_2.png) [@mmayo](https://meta.discourse.org/u/mmayo) #### Post date: [9. August 2018 um 20:06 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/2 "2018-08-09T20:06:17Z")
We recently went through this process, and there are a couple of ways to do it, the safe and slow way, using the `migrate_to_s3` rake task: > I think you could go ahead and turn on s3 uploads (there are a number of guides), and possible ssh into your container and run this task, and you wouldn’t experience downtime. We didn’t go this route because it takes ~15 seconds per upload, and this was going to take days. We were doing this as part of a host migration, and couldn’t have a downtime of that long. The quick and dirty route is as follows: 1. Enable S3 Uploads on your site 2. Back up your site with images, and download the archive. 3. Unzip the the archive, and navigate to `uploads` sub-folder, and upload the images to s3, using the aws-cli: ```plaintext uploads aws s3 cp . s3:// --recursive --acl public-read ``` 4. We then need to [remap](https://github.com/discourse/discourse/blob/master/script/discourse#L5) all of the references to the public uploads folder to the new location in s3. At the console in your docker container: ` root@dc53d70f611c:/var/www/discourse# discourse remap /uploads/default/ //.s3.amazonaws.com/ Rewriting all occurences of /uploads/default/ to //.s3.amazonaws.com/ THIS TASK WILL REWRITE DATA, ARE YOU SURE (type YES) YES Remapping ar_internal_metadata key 0 rows affected! Remapping ar_internal_metadata value ... many more rows ` We did the above during a scheduled maintenance window, and the site was in read only mode with a fresh backup on hand, so it was pretty low risk. I’m not sure I’d be comfortable doing it any other way, but it took less than an hour. ---
### Author: ![Gunnar](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gunnar/32/119647_2.png) [@Gunnar](https://meta.discourse.org/u/Gunnar) #### Post date: [9. August 2018 um 21:57 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/3 "2018-08-09T21:57:01Z")
> [@mmayo](#): > > The quick and dirty route is as follows: Awesome, I’m trying that right now. Thanks! Gunnar ---
### Author: ![Gunnar](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gunnar/32/119647_2.png) [@Gunnar](https://meta.discourse.org/u/Gunnar) #### Post date: [9. August 2018 um 22:34 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/4 "2018-08-09T22:34:21Z")
> [@mmayo](#): > > We then need to [remap](https://github.com/discourse/discourse/blob/master/script/discourse#L5) all of the references to the public uploads folder to the new location in s3. At the console in your docker container: > `root@dc53d70f611c:/var/www/discourse# discourse remap /uploads/default/ //.s3.amazonaws.com/ Rewriting all occurences of /uploads/default/ to //.s3.amazonaws.com/ THIS TASK WILL REWRITE DATA, ARE YOU SURE (type YES) YES Remapping ar_internal_metadata key 0 rows affected! Remapping ar_internal_metadata value ... many more rows` That did not work as expected. Images in posts now have URLs like this: `https://forums.jag-lovers.com//jl-discourse-uploads.s3.amazonaws.com/original/3X/b/3/b38c9407f8d17b269dc7ee8845d85cf5189ce8b5.jpeg` Instead of this: `https://jl-discourse-uploads.s3.amazonaws.com/original/3X/b/3/b38c9407f8d17b269dc7ee8845d85cf5189ce8b5.jpeg` While on the other hand the small category icons (we use a custom icon for each category) have the correct URLs. What did I miss? Thanks, Gunnar ---
### Author: ![mmayo](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mmayo/32/119577_2.png) [@mmayo](https://meta.discourse.org/u/mmayo) #### Post date: [9. August 2018 um 23:15 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/5 "2018-08-09T23:15:27Z")
@Gunnar Hmm, that’s odd. AFAIK the Uploads table should only reference the relative path, and not the full path to the image location. We had a couple of posts that ended up that way, but it was because the full path was referenced in the post, rather than using the image upload button or copy/pasting. Is this happening for _all_ posts, or perhaps a subset that use the full domain, rather than the relative path? Either way, something like this _should_ work ```plaintext remap https://forums.jag-lovers.com//jl-discourse-uploads.s3.amazonaws.com //jl-discourse-uploads.s3.amazonaws.com ``` You may need to tweak the input string, as I am not sure if you’ll need the `https://` or not, it’s best to take a look at either the Uploads table or the unbaked post to see what string your replacing. ---
### Author: ![Gunnar](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gunnar/32/119647_2.png) [@Gunnar](https://meta.discourse.org/u/Gunnar) #### Post date: [10. August 2018 um 01:50 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/6 "2018-08-10T01:50:53Z")
> [@mmayo](#): > > You may need to tweak the input string, as I am not sure if you’ll need the `https://` or not, it’s best to take a look at either the Uploads table or the unbaked post to see what string your replacing. It looks like this applies to all posts on the forum. I had a look at several raw/unbaked posts and the images are all referenced like this: `![IMG_20150512_182321|690x388](upload://d5tlLS5VtcU7PAaOgzFsROaB6vc.jpg)` When I look at the uploads table, the entries all have a URL of the form: `//jl-discourse-uploads.s3.amazonaws.com/original/3X/8/a/8a560451dcba13a7f9d4545368c078e82ef890ac.jpeg` Which should be correct, no? So, it almost looks as if something in Discourse is adding the [https://forums.jag-lovers.com](https://forums.jag-lovers.com) part on the fly? Do I have to rebake all the posts? Thanks, @mmayo! Thanks, Gunnar ---
### Author: ![Gunnar](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gunnar/32/119647_2.png) [@Gunnar](https://meta.discourse.org/u/Gunnar) #### Post date: [10. August 2018 um 03:01 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/7 "2018-08-10T03:01:02Z")
Hi @sam, perhaps you can help clear this up? Thanks, Gunnar ---
### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam) #### Post date: [10. August 2018 um 03:14 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/8 "2018-08-10T03:14:08Z")
@zogstrip can have a quick look next week ---
### Author: ![Gunnar](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gunnar/32/119647_2.png) [@Gunnar](https://meta.discourse.org/u/Gunnar) #### Post date: [10. August 2018 um 06:48 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/9 "2018-08-10T06:48:15Z")
> [@sam](#): > > a quick look next week Thanks, but I think I got it. I just need a rebake. Gunnar ---
### Author: ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip) #### Post date: [22. August 2018 um 13:55 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/10 "2018-08-22T13:55:38Z")
Is everything sorted now? ---
### Author: ![Gunnar](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/gunnar/32/119647_2.png) [@Gunnar](https://meta.discourse.org/u/Gunnar) #### Post date: [22. August 2018 um 15:14 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/11 "2018-08-22T15:14:29Z")
Thank you, yes. A rebake fixed it. We’re all set. In hindsight, if I had to do it again, I’d either just use the slow method, or follow @mmayo’s “quick and dirty route” minus the remap, but plus a rebake. Once the rebake was done, I’d delete the local uploads folder. That way, the site would be 100% functional the whole time. As it was, we were up and running, but images in posts didn’t display correctly until the posts were rebaked. Thanks, Gunnar ---
### Author: ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip) #### Post date: [22. August 2018 um 15:20 UTC](https://meta.discourse.org/t/step-by-step-for-moving-large-site-uploads-to-s3/94321/12 "2018-08-22T15:20:05Z")