Stop using Amazon s3 for uploads

support
1

I want to stop using Amazon s3 for uploads.
I’ve not done a rake task like this so want to check things before I run on the live environment.
My plan is to use a clone of my live server to test the rake task uploads:migrate_from_s3.

Will this move the files from s3?
Or copy them to the local storage and leave the s3 files there until I manually delete them?

If it deletes them from s3 that will break the live forum and that would be bad!
thanks

2 Likes
2

I believe @vinothkannans can point you in the right direction here.

1 Like
3

Yes, it will remove the file from S3.

You should clone your S3 bucket too. And change the bucket name to newly cloned one in the site settings before testing the migration process.

Also I will recommend you to take a backup of S3 bucket before running the migration in live environment.

4 Likes
4

The rake task uploads:migrate_from_s3 does not appear when I enter the app and enter rake -AT.

Is there a new method for migrating uploads from s3 back to local?

2 Likes
5

What output you get when rake task uploads:migrate_from_s3 ??

6 
rake aborted!

Don't know how to build task 'uploads:migrate_from_s3' (See the list of available tasks with `rake --tasks`)

Did you mean? uploads:migrate_to_s3
7

It was removed in a community Pull Request. You can read the reasoning at

https://github.com/discourse/discourse/pull/11703

4 Likes
8

Thanks for that. I can’t seem to find instructions or guidance on how this should be accomplished manually.

1 Like
9

You may use awscli to manually migrate all your files from S3 to your server,

cd /var/discourse
./launcher enter app
sudo apt install awscli
aws configure

Configure aws with your S3 ID and password. then

cd public/uploads/default/
aws s3 sync s3://mybucketname/

This command will download all the files from S3 to public/uploads/default/. Then,

rake posts:rebake
exit
./launcer rebuild app
4 Likes
10

I tried first to fix migration off S3, and was told after I had completed using my own migration as a test bed for the code i was writing that I had taken completely the wrong approach. This assertion was supported by data corruption in my (large) instance that I migrated off S3 and into local files. Relatively recently, I ended up fixing up broken data in the database by hand, and I don’t have an expectation that I’m done cleaning up after my attempt.

If I were going to start a migration off of S3 onto my own storage today, I would spin up a minio container next to discourse, use minio-client to copy from S3 to my minio container, reconfigure discourse for the new “S3” location of my local minio, and then posts:remap or rebake all posts to serve “S3” from my minio container.

Here’s an example of how posts:remap should work here. I have a sense that there’s a better example somewhere here, but this should be enough to get started.

6 Likes
11

So I followed these instructions and was able to copy my uploads over, but there are still links to the s3 server throughout the instance even with S3 uploads turned off. I tried a discourse remap but that was not effective, tried a rake posts:remap but that was not effective, posts:rebake was not effective.

It sure was easy setting up uploads on S3 but not easy going back.

4 Likes
12

Migrate from s3 is broken for a long time and it’s not fixed yet. Go through the above post by mcdanlj.

What’s the output you get for the below command?

./launcher enter app
rails c
Upload.all.sample(20).pluck(:url)
3 Likes
13

I’ve relinked uploads individually since the site is not entensive at this point.
rake posts:missing_uploads was useful in determining whether I had caught them all.

2 Likes
16

Are you sure that all url are remapped properly? Not only for image, but you have to remap url for avatars, icons, backup…etc. Try the above command and see if S3 url in the output.

3 Likes
17

The urls are all back to relative. Unfortunately for future people reading this post, I’m not sure which combination of rake tasks and app rebuilds was the secret to my success. Someone smarter than me about discourse should put together a fool-proof tutorial on how this should be accomplished.

Using awscli as decribed by Pravi above was effective for copying files from s3.

6 Likes
18

I too am confused very much.
Earlier I had two S3 buckets at diff times. And rake posts:missing_uploads showed some 500 posts were missing uploads/were affected.

But then I decided to copy both buckets assets to local ubuntu server (on GCP instance). And now the missing uploads are showing in the range of 1000.

If I command Upload.all.sample(1000).pluck(:url) I get different paths of uploads, some on local, some on bucket1 and very few on bucket2.

What should I do now? (currently I’ve configured my uploads on the local server).

Mine is a small website, with only some 1000 posts in 2 years.

1 Like
19

You have to remap the S3 URL. Can you post the output for Upload.all.sample(1000).pluck(:url)?

1 Like
20

Thanks ji.

I can try doing that, if that has a good chance of success.

Edit: I did that, but that disappeared the images from my posts completely. Hovering over image placeholder showed no url link at all. Earlier it was not only showing a small image icon, but also the image url.

I’m giving some 45 lines of output. I don’t know if you want me to paste all the 1000 lines of output here?

=> ["//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/7277b38e1d614b3d700f6266fa5b841a15e7c6ba.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/0c1f27b110ff476c636265b46d63f83c5bf575e9.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/a/a18dd8e63379c70e2a4014b0f6d85cf89ab12ed7.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/5/51798c75050206a784d9ae186cffb5c4558b1625.jpeg",
 "/uploads/default/original/2X/1/159ca83317895ddaefec42ee75c8748201f34d1f.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/1/1593a5f91ac7642180c9651544806d89342a4e24.jpeg",
 "/uploads/default/original/2X/f/fa7c701e58c198984707161364c81e9c59b27572.jpeg",
 "/uploads/default/original/2X/b/b6b12c4df19b234f6d1551c26d92a14969f19f12.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/fcd4201295e6e63a7496e70e8b067225774834f3.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/f42150c03cce15a1bdc34c6af746fe51b63db36c.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/e66cfb3a0bdc558a02fd999d1bf5a185521456cf.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/1885bc0fe4cf6ee25ab8d6e250807c9836168cdc.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/c/c1f240c94d45e0be0cfcd7a3cc714e389bb49f23.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/2/293518b6119a17d64222cf5d691aee271fa572cf.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/617f0bb525f05934ae455582deacb96e7fd097e9.jpeg",
 "//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/3/331a610d2237294bdb126eecf1f087173c5087df.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/d/d0b3efa53ceab849e17e781e63f2b969f94dbde0.jpeg",
 "/images/default-favicon.ico",
 "/uploads/default/original/2X/b/b399c60eb0b83232531ef2ead5a060fbd9c2a64e.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/9/987178dc86175225b999151e112deb3bf26b13af.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/c/c99c8a9dd3e59cecc6fb5010f362b1dc3f49ca66.jpeg",
 "//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/5/5c145297f26cd91f6f157d507324724ba15cb618.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/e/ef5c223cd604e3574c300671da61693dc5399e62.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/9c99b85e976c42bc3c692f75b8828abbbc2bc4ea.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/6/6f88f27ee155e69bb5b701b86eba9dc3e369bc09.png",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/3/3b454434e58b05db14f51c3fcdfbd36d0d96d4eb.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/b7e92457674b8c28f6ecfbc9340816fc3583d20e.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/7/731c0ee200660fb1888cd058c79282bf7f6dab95.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/a42f92c28be8bd5ac86bb55f36fc9ca0bec37c72.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/ac7dc4c531d428b152de3eebceb44bfe779c349d.jpeg",
 "//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/f361a9ad5194ef13c2c6b24e2db98455e7cc48d9.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/43ec21eafc00f8cd05976e0fbec41ef686e0e3b3.png",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/8b41f21b02df80a9e5c4ea14a5dffa5776811f3d.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/6/6eee5c5a5a2b274e7d8ccc911cd4f594cc4930de.png",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/c1736a8c2923f0bd341f7112db74cd62392f7e14.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/c/c3e2fbb84c55dbf2f1fdccf835964eb4319e2371.png",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/0/0b41e87ab7728dbead217e237c4ad8940bdfe776.png",
 "//bhdisco.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/a/a3ac121566d741d5b396ba49625f3206ff0c498f.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/b/bbbb6973cb0900389675691c5c129c907cf4611f.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/2/28f5d5bf46b30a33bde8e67c184cb099cc71511a.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/ff34c3fc207781d5881d4583cceb26aa54619c0e.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/f/f85db75b4721b8d24abf78ee3d25f2ffbef26da9.jpeg",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/1X/35c6402fbf4a8fca05f30924800fa0c90cf891e7.png",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/9/9d8f29892278f164e8ce27a6b58cc8af0760802c.png",
 "//bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com/uploads/original/2X/1/1a533eb9f4ec5186427a541a7ebbf06817627f97.png",
1 Like
21

I see several broken avatars on your site. I assume it’s because you have either deleted the S3 bucket or you have not rebaked once you have copied all files back from S3. Why there are two bucket in the output?

bhdisco.s3.dualstack.ap-south-1.amazonaws.com
bxyzbucket1.s3.dualstack.ap-south-1.amazonaws.com

Have you configured you site with one bucket and later changed to another?

3 Likes
22

Sorry, I somehow missed the notif that you’ve replied. Thanks indeed.

I’ve not deleted the S3 bucket, but indeed I had switched the buckets earlier (at that time I wasn’t aware of its complications). So all my assets were in 2 S3 buckets, but now I’ve manually copied them onto local server also (/var/discourse/shared/web_only/uploads/default)

How should I proceed to clear this mess up?

Note: I’m familiar with basic commands of the container and rails console and Aws S3.

1 Like