# Stop using Amazon s3 for uploads
**URL:** https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286
**Category:** Support
**Created:** [January 23, 2019, 1:26pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286 "2019-01-23T13:26:23Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@discourseuser100](https://meta.discourse.org/u/discourseuser100)
#### Post date: [January 23, 2019, 1:26pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/1 "2019-01-23T13:26:23Z")
I want to stop using Amazon s3 for uploads.
I’ve not done a rake task like this so want to check things before I run on the live environment.
My plan is to use a clone of my live server to test the rake task uploads:migrate\_from\_s3.
Will this **move** the files from s3?
Or **copy** them to the local storage and leave the s3 files there until I manually delete them?
If it deletes them from s3 that will break the live forum and that would be bad!
thanks
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [January 24, 2019, 8:27am UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/2 "2019-01-24T08:27:31Z")
I believe @vinothkannans can point you in the right direction here.
---
### Author:  [@vinothkannans](https://meta.discourse.org/u/vinothkannans)
#### Post date: [January 24, 2019, 11:58am UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/3 "2019-01-24T11:58:51Z")
> [@discourseuser100](#):
>
> Will this **move** the files from s3?
Yes, it will remove the file from S3.
> [@discourseuser100](#):
>
> My plan is to use a clone of my live server to test the rake task uploads:migrate\_from\_s3.
You should clone your S3 bucket too. And change the bucket name to newly cloned one in the site settings before testing the migration process.
Also I will recommend you to take a backup of S3 bucket before running the migration in live environment.
---
### Author:  [@keven](https://meta.discourse.org/u/keven)
#### Post date: [April 4, 2021, 4:54pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/4 "2021-04-04T16:54:01Z")
The rake task uploads:migrate\_from\_s3 does not appear when I enter the app and enter rake -AT.
Is there a new method for migrating uploads from s3 back to local?
---
### Author:  [@Pravi](https://meta.discourse.org/u/Pravi)
#### Post date: [April 5, 2021, 12:06pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/5 "2021-04-05T12:06:13Z")
What output you get when `rake task uploads:migrate_from_s3` ??
---
### Author:  [@keven](https://meta.discourse.org/u/keven)
#### Post date: [April 5, 2021, 3:41pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/6 "2021-04-05T15:41:34Z")
```
rake aborted!
Don't know how to build task 'uploads:migrate_from_s3' (See the list of available tasks with `rake --tasks`)
Did you mean? uploads:migrate_to_s3
```
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [April 5, 2021, 4:46pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/7 "2021-04-05T16:46:50Z")
> [@keven](#):
>
> The rake task uploads:migrate\_from\_s3 does not appear when I enter the app and enter rake -AT.
It was removed in a community Pull Request. You can read the reasoning at
[https://github.com/discourse/discourse/pull/11703](https://github.com/discourse/discourse/pull/11703)
---
### Author:  [@keven](https://meta.discourse.org/u/keven)
#### Post date: [April 5, 2021, 5:01pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/8 "2021-04-05T17:01:55Z")
Thanks for that. I can’t seem to find instructions or guidance on how this should be accomplished manually.
---
### Author:  [@Pravi](https://meta.discourse.org/u/Pravi)
#### Post date: [April 5, 2021, 5:47pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/9 "2021-04-05T17:47:51Z")
You may use awscli to manually migrate all your files from S3 to your server,
```plaintext
cd /var/discourse
./launcher enter app
sudo apt install awscli
aws configure
```
Configure aws with your S3 ID and password. then
```plaintext
cd public/uploads/default/
aws s3 sync s3://mybucketname/
```
This command will download all the files from S3 to `public/uploads/default/`. Then,
```plaintext
rake posts:rebake
exit
./launcer rebuild app
```
---
### Author:  [@mcdanlj](https://meta.discourse.org/u/mcdanlj)
#### Post date: [April 5, 2021, 6:31pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/10 "2021-04-05T18:31:00Z")
I tried first to fix migration off S3, and was told after I had completed using my own migration as a test bed for the code i was writing that I had taken completely the wrong approach. This assertion was supported by data corruption in my (large) instance that I migrated off S3 and into local files. Relatively recently, I ended up fixing up broken data in the database by hand, and I don’t have an expectation that I’m done cleaning up after my attempt.
If I were going to start a migration off of S3 onto my own storage today, I would spin up a minio container next to discourse, use minio-client to copy from S3 to my minio container, reconfigure discourse for the new “S3” location of my local minio, and then `posts:remap` or rebake all posts to serve “S3” from my minio container.
Here’s an example of how `posts:remap` should work here. I have a sense that there’s a better example somewhere here, but this should be enough to get started.
> [@How do I move my s3 upload bucket from one provider to another?](https://meta.discourse.org/t/how-do-i-move-my-s3-upload-bucket-from-one-provider-to-another/178484):
>
> I have setup my upload bucket originally on AWS, but I want to now use Linode. Setup went well and I can see that images are being uploaded to the Linode already. However, when I look at the old posts, they are still pointing to AWS. How can I move those images to Linode so that I can remove the bucket on AWS? Thanks!
---
### Author:  [@keven](https://meta.discourse.org/u/keven)
#### Post date: [April 8, 2021, 8:27pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/11 "2021-04-08T20:27:13Z")
So I followed these instructions and was able to copy my uploads over, but there are still links to the s3 server throughout the instance even with S3 uploads turned off. I tried a discourse remap but that was not effective, tried a rake posts:remap but that was not effective, posts:rebake was not effective.
It sure was easy setting up uploads on S3 but not easy going back.
---
### Author:  [@Pravi](https://meta.discourse.org/u/Pravi)
#### Post date: [April 9, 2021, 6:21am UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/12 "2021-04-09T06:21:44Z")
Migrate from s3 is broken for a long time and it’s not fixed yet. Go through the above post by mcdanlj.
What’s the output you get for the below command?
```
./launcher enter app
rails c
Upload.all.sample(20).pluck(:url)
```
---
### Author:  [@keven](https://meta.discourse.org/u/keven)
#### Post date: [April 9, 2021, 5:22pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/13 "2021-04-09T17:22:55Z")
I’ve relinked uploads individually since the site is not entensive at this point.
`rake posts:missing_uploads` was useful in determining whether I had caught them all.
---
### Author:  [@Pravi](https://meta.discourse.org/u/Pravi)
#### Post date: [April 10, 2021, 3:32am UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/16 "2021-04-10T03:32:46Z")
Are you sure that all url are remapped properly? Not only for image, but you have to remap url for avatars, icons, backup..etc. Try the above command and see if S3 url in the output.
---
### Author:  [@keven](https://meta.discourse.org/u/keven)
#### Post date: [April 10, 2021, 6:31am UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/17 "2021-04-10T06:31:20Z")
The urls are all back to relative. Unfortunately for future people reading this post, I’m not sure which combination of rake tasks and app rebuilds was the secret to my success. Someone smarter than me about discourse should put together a fool-proof tutorial on how this should be accomplished.
Using awscli as decribed by Pravi above was effective for copying files from s3.
---
### Author:  [@Bathinda](https://meta.discourse.org/u/Bathinda)
#### Post date: [April 24, 2021, 8:32am UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/18 "2021-04-24T08:32:10Z")
> [@Pravi](#):
>
> sure that all url are remapped properly? Not only for image, but you have to remap url for avatars, icons, backup…etc. Try the above command and see if S3 url
I too am confused very much.
Earlier I had two S3 buckets at diff times. And `rake posts:missing_uploads ` showed some 500 posts were missing uploads/were affected.
But then I decided to copy both buckets assets to local ubuntu server (on GCP instance). And now the missing uploads are showing in the range of 1000.
If I command `Upload.all.sample(1000).pluck(:url)` I get different paths of uploads, some on local, some on bucket1 and very few on bucket2.
What should I do now? (currently I’ve configured my uploads on the local server).
Mine is a small [website](https://bathindahelper.com), with only some 1000 posts in 2 years.
---
### Author:  [@Pravi](https://meta.discourse.org/u/Pravi)
#### Post date: [April 24, 2021, 10:04am UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/19 "2021-04-24T10:04:31Z")
You have to remap the S3 URL. Can you post the output for `Upload.all.sample(1000).pluck(:url)`?
---
### Author:  [@Bathinda](https://meta.discourse.org/u/Bathinda)
#### Post date: [April 24, 2021, 10:08am UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/20 "2021-04-24T10:08:29Z")
Thanks ji.
> [@Pravi](#):
>
> You have to remap the S3 URL.
I can try doing that, if that has a good chance of success.
Edit: I did that, but that disappeared the images from my posts completely. Hovering over image placeholder showed no url link at all. Earlier it was not only showing a small image icon, but also the image url.
> [@Pravi](#):
>
> Can you post the output for `Upload.all.sample(1000).pluck(:url)` ?
I’m giving some 45 lines of output. I don’t know if you want me to paste all the 1000 lines of output here?
```plaintext
=> ["//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/7277b38e1d614b3d700f6266fa5b841a15e7c6ba.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/0c1f27b110ff476c636265b46d63f83c5bf575e9.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/a/a18dd8e63379c70e2a4014b0f6d85cf89ab12ed7.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/5/51798c75050206a784d9ae186cffb5c4558b1625.jpeg",
"/uploads/default/original/2X/1/159ca83317895ddaefec42ee75c8748201f34d1f.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/1/1593a5f91ac7642180c9651544806d89342a4e24.jpeg",
"/uploads/default/original/2X/f/fa7c701e58c198984707161364c81e9c59b27572.jpeg",
"/uploads/default/original/2X/b/b6b12c4df19b234f6d1551c26d92a14969f19f12.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/fcd4201295e6e63a7496e70e8b067225774834f3.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/f42150c03cce15a1bdc34c6af746fe51b63db36c.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/e66cfb3a0bdc558a02fd999d1bf5a185521456cf.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/1885bc0fe4cf6ee25ab8d6e250807c9836168cdc.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/c/c1f240c94d45e0be0cfcd7a3cc714e389bb49f23.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/2/293518b6119a17d64222cf5d691aee271fa572cf.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/617f0bb525f05934ae455582deacb96e7fd097e9.jpeg",
"//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/3/331a610d2237294bdb126eecf1f087173c5087df.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/d/d0b3efa53ceab849e17e781e63f2b969f94dbde0.jpeg",
"/images/default-favicon.ico",
"/uploads/default/original/2X/b/b399c60eb0b83232531ef2ead5a060fbd9c2a64e.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/9/987178dc86175225b999151e112deb3bf26b13af.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/c/c99c8a9dd3e59cecc6fb5010f362b1dc3f49ca66.jpeg",
"//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/5/5c145297f26cd91f6f157d507324724ba15cb618.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/e/ef5c223cd604e3574c300671da61693dc5399e62.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/9c99b85e976c42bc3c692f75b8828abbbc2bc4ea.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/6/6f88f27ee155e69bb5b701b86eba9dc3e369bc09.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/3/3b454434e58b05db14f51c3fcdfbd36d0d96d4eb.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/b7e92457674b8c28f6ecfbc9340816fc3583d20e.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/7/731c0ee200660fb1888cd058c79282bf7f6dab95.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/a42f92c28be8bd5ac86bb55f36fc9ca0bec37c72.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/ac7dc4c531d428b152de3eebceb44bfe779c349d.jpeg",
"//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/f361a9ad5194ef13c2c6b24e2db98455e7cc48d9.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/43ec21eafc00f8cd05976e0fbec41ef686e0e3b3.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/8b41f21b02df80a9e5c4ea14a5dffa5776811f3d.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/6/6eee5c5a5a2b274e7d8ccc911cd4f594cc4930de.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/c1736a8c2923f0bd341f7112db74cd62392f7e14.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/c/c3e2fbb84c55dbf2f1fdccf835964eb4319e2371.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/0/0b41e87ab7728dbead217e237c4ad8940bdfe776.png",
"//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/a/a3ac121566d741d5b396ba49625f3206ff0c498f.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/b/bbbb6973cb0900389675691c5c129c907cf4611f.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/2/28f5d5bf46b30a33bde8e67c184cb099cc71511a.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/ff34c3fc207781d5881d4583cceb26aa54619c0e.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/f85db75b4721b8d24abf78ee3d25f2ffbef26da9.jpeg",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/35c6402fbf4a8fca05f30924800fa0c90cf891e7.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/9/9d8f29892278f164e8ce27a6b58cc8af0760802c.png",
"//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/1/1a533eb9f4ec5186427a541a7ebbf06817627f97.png",
```
---
### Author:  [@Pravi](https://meta.discourse.org/u/Pravi)
#### Post date: [April 25, 2021, 8:25am UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/21 "2021-04-25T08:25:41Z")
I see several broken avatars on your site. I assume it’s because you have either deleted the S3 bucket or you have not rebaked once you have copied all files back from S3. Why there are two bucket in the output?
```
bhdisco.s3.dualstack.ap-south-1.amazonaws.com
bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com
```
Have you configured you site with one bucket and later changed to another?
---
### Author:  [@Bathinda](https://meta.discourse.org/u/Bathinda)
#### Post date: [April 26, 2021, 12:22pm UTC](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286/22 "2021-04-26T12:22:18Z")
Sorry, I somehow missed the notif that you’ve replied. Thanks indeed.
> [@Pravi](#):
>
> I assume it’s because you have either deleted the S3 bucket or you have not rebaked once you have copied all files back from S3. Why there are two bucket in the output?
> [@Pravi](#):
>
> Have you configured you site with one bucket and later changed to another?
I’ve not deleted the S3 bucket, but indeed I had switched the buckets earlier (at that time I wasn’t aware of its complications). So all my assets were in 2 S3 buckets, but now I’ve manually copied them onto local server also (/var/discourse/shared/web\_only/uploads/default)
How should I proceed to clear this mess up?
Note: I’m familiar with basic commands of the container and rails console and Aws S3.
[Next page](https://meta.discourse.org/t/stop-using-amazon-s3-for-uploads/107286.md?page=2)