Hi there! We are migrating all our uploads/images between two different s3 compatible services (both are digital ocean spaces if it matters) and I have decided that we are stuck in a quite bad state. I’ll start with explaining how the migration was done: We cloned/synced the initial bucket to th…

After running the toombstone recovery rake task and then the fix_missing_uploads rake task, I finally got it to start “fix” the images. It seems like it’s downloading them and uploading them again, and it takes a lot of time and uses a lot of resources, but at the least the users will get their ima…

Might be worth mentioning that I have tried to use the CDN uri provided for the spaces bucket as well (with a rebake).

Output from missing uploads: rake posts:missing_uploads Looking for missing uploads on: default Fixing missing uploads: 🚫 17075 post uploads are missing. 16906 uploads are missing. 1 of 16906 are old scheme uploads. 14646 of 139801 posts are affected. post_uploads have 3448 entries optimized_ima…

You can see Moving from one S3 bucket to another I think that i have a draft of howto that I will try to post tomorrow.

That would be very helpful, tyvm!

Hey @Jite ! See if this works for you. If it does, I’ll go about creating a proper howto Old buckets This assumes that you can install and configure a tool to move your data from your old bucket to a local machine and then again do the same from local to the new bucket. See aws cli sync (which…

Thank you for the response. This is basically what I did the last time, but I tried this again. The issue is that posts.count returns 0. All posts have the transperent.png file in the cooked post and contains a hash in the uncooked. Is there any way to make it resolve the image correctly during ba…

Hmm. Right. That cooked change works only to avoid the remake. If a remake does not work then something else is wrong. Maybe the assets are not where discourse thinks?

Well, it’s possible, but the “move” was pretty much a 1:1 move of all files in the bucket, hehe…

So you can replace the old bucket url with the new and it works? Do the values in Uploads look right?

I’m a bit scared that changing back to old bucket might start of a job to move everything to tombstone due to them being “old” now, but yes, the database seems to point to the correct (new location) images, it’s basically just the old posts that don’t resolve to the correct image (I’m guessing…).

업로드용 S3 버킷 변경

제품 지원

Jite (Johannes Tegnér) 4월 30, 2021, 7:10오전 5

정말 큰 도움이 될 것 같아요, 정말 감사합니다!

Migrate S3 bucket to new S3 bucket (compatible service)

글		댓글	조회수
Moving from one S3 bucket to another Product Support	6	2151	8월 8, 2021
Migrate S3 bucket to new S3 bucket (compatible service) Product Support	1	809	3월 31, 2022
Stop using Amazon s3 for uploads Product Support	51	5682	8월 22, 2021
Migrating old uploads to S3 Self-hosting s3	8	1750	5월 11, 2021
Moving uploads to a different S3 location Product Support	2	493	10월 1, 2020

업로드용 S3 버킷 변경

Related topics