Hi there! We are migrating all our uploads/images between two different s3 compatible services (both are digital ocean spaces if it matters) and I have decided that we are stuck in a quite bad state. I’ll start with explaining how the migration was done: We cloned/synced the initial bucket to th…

After running the toombstone recovery rake task and then the fix_missing_uploads rake task, I finally got it to start “fix” the images. It seems like it’s downloading them and uploading them again, and it takes a lot of time and uses a lot of resources, but at the least the users will get their ima…

Might be worth mentioning that I have tried to use the CDN uri provided for the spaces bucket as well (with a rebake).

Output from missing uploads: rake posts:missing_uploads Looking for missing uploads on: default Fixing missing uploads: 🚫 17075 post uploads are missing. 16906 uploads are missing. 1 of 16906 are old scheme uploads. 14646 of 139801 posts are affected. post_uploads have 3448 entries optimized_ima…

You can see Moving from one S3 bucket to another I think that i have a draft of howto that I will try to post tomorrow.

Hey @Jite ! See if this works for you. If it does, I’ll go about creating a proper howto Old buckets This assumes that you can install and configure a tool to move your data from your old bucket to a local machine and then again do the same from local to the new bucket. See aws cli sync (which…

Thank you for the response. This is basically what I did the last time, but I tried this again. The issue is that posts.count returns 0. All posts have the transperent.png file in the cooked post and contains a hash in the uncooked. Is there any way to make it resolve the image correctly during ba…

Hmm. Right. That cooked change works only to avoid the remake. If a remake does not work then something else is wrong. Maybe the assets are not where discourse thinks?

Well, it’s possible, but the “move” was pretty much a 1:1 move of all files in the bucket, hehe…

So you can replace the old bucket url with the new and it works? Do the values in Uploads look right?

I’m a bit scared that changing back to old bucket might start of a job to move everything to tombstone due to them being “old” now, but yes, the database seems to point to the correct (new location) images, it’s basically just the old posts that don’t resolve to the correct image (I’m guessing…).

Changing s3 bucket for uploads

Support

Jite (Johannes Tegnér) April 30, 2021, 7:10am 5

That would be very helpful, tyvm!

Migrate S3 bucket to new S3 bucket (compatible service)

Topic		Replies	Views
Moving from one S3 bucket to another Support	6	1998	August 8, 2021
Migrate S3 bucket to new S3 bucket (compatible service) Support	1	746	March 31, 2022
Stop using Amazon s3 for uploads Support	52	5000	August 22, 2021
Migrating old uploads to S3 Self-hosting s3	8	1618	May 11, 2021
Moving uploads to a different S3 location Support	3	444	October 31, 2020

Changing s3 bucket for uploads

Related topics