Missing uploads after S3 → S3 migration

Support

AI-generated summary

The user, domstubbs, has migrated their Discourse uploads from one S3 bucket to another and has encountered issues with some uploads not working. They have tried running uploads:recover_from_tombstone, posts:rebake, and uploads:recover, which have helped recover most uploads, but some still remain broken. The uploads:recover task outputs warnings about incorrect upload hashes, and the user has tried running uploads:fix_relative_upload_links to fix these issues, but it does not seem to work. They have also tried converting broken upload:// URLs and comparing them against records in the uploads table, but this has not shed any light on the issue. Despite having valid data in the database and files existing on S3, the user is unable to access the broken uploads.

1

I’ve migrated uploads from one S3 bucket to another following the tips suggested here and, with the added assistance of uploads:recover_from_tombstone and posts:rebake, I have pretty much everything working again – but not everything.

I’m not a Ruby dev so my ability to debug is limited. When I run uploads:recover now it gives me lots of output like this:

Warning /t/foo/489/5 had an incorrect 4ff3ad05b76e1888161f41fd86da66efc16c290c should be 0b029ef3549e72dd5e3498abc7f979736d279a5e storing in custom field ‘rake uploads:fix_relative_upload_links’ can fix this
Warning /t/bar/498/4 had an incorrect 0649edb23196a8c84a9d71c04147bd8f0c96d0f7 should be f5b4e4bb3eedcd0fb83a06eecc4920f20c5e2bc4 storing in custom field ‘rake uploads:fix_relative_upload_links’ can fix this
Warning /t/bar/498/4 had an incorrect 0649edb23196a8c84a9d71c04147bd8f0c96d0f7 should be f5b4e4bb3eedcd0fb83a06eecc4920f20c5e2bc4 storing in custom field ‘rake uploads:fix_relative_upload_links’ can fix this

These warnings do seem to correspond to posts with broken uploads, so it sounds fairly promising. However, when I run fix_relative_upload_links, it doesn’t seem to do anything.

rake uploads:fix_relative_upload_links --trace
** Invoke uploads:fix_relative_upload_links (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute uploads:fix_relative_upload_links
skipping missing upload in https://foo.com/t/bar/7/2 <img src="/uploads/default/35/f450f88541e87c52.png" width="198" height="70">
skipping missing upload in https://foo.com/t/bar/7/2 <img src="/uploads/default/36/0b73d33d9df37a66.png" width="120" height="43">

(I think the two skipped files are ancient pre-S3 uploads. There’s no output at all for the S3 uploads that generated the prior warnings.)

There’s a report of a similar issue from a year ago but I can’t see any similar reports which include a fix.

I appreciate I’m not doing something that’s officially supported, but any pointers would be appreciated.

I only spotted this issue a few days after the migration. Had I spotted it immediately I’d have tried re-migrating using a fresh DB/S3 restore, but since I don’t want to lose any recent posts I’ll have to work with the live site from here on out.

Thanks.

Edit: I’ve also now tried converting the broken upload:// URLs and comparing them against records in the uploads table and I’m not really any the wiser.

When I inspect a working upload:// URL it maps to an uploads table record with a valid URL, but when I inspect a broken one it… also maps to a record with a valid URL. It seems as though there is valid data in the DB and the files do exist on S3 at the URL assigned in the uploads table. I can access the S3 URLs in my browser without issue, so I’ve no idea why they’re failing to pull through.

1 Like