Migrate_to_S3 Fails on Rebake

Edit: It looks like your error was different to mine, sounds like some of your images didn’t get moved over to s3. Sorry, I misread. Maybe some of the info below is still useful. Did you set up your s3 uploads from this guide? Using Object Storage for Uploads (S3 & Clones)

@evenif - Sorry to hear you’re having issues with this. I was planning to write a guide for this for people getting the same issues, as a fair bit of info needed to be pieced together from different threads. But waiting to hopefully fix avatars, as you can see above I’m still having issues with that.

If you run this in rails do you get true or false?

SiteSetting.migrate_to_new_scheme

If it’s false, then you can try setting it to true like:

SiteSetting.migrate_to_new_scheme = true

Then either wait for a bit and come back later (I believe this runs every 15 minutes IIRC), but if you want to run it immediately:

Jobs::MigrateUploadScheme.new.execute(nil)

Then check again afterwards/later if SiteSetting.migrate_to_new_scheme is now false (means it should have completed).

Then run:
Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '/uploads/default/original/_X/%'").to_a

This should find uploads that still have issues and couldn’t be migrated. In my case, all of these uploads existed as records in the db, but the actual images were not existing.

Depending on how big the list is, you can copy page then find + replace to make a list of commands to remove these problem records via the list of upload ids.

Upload.find(1).destroy
Upload.find(2).destroy
Upload.find(3).destroy

With the upload ids replacing 1, 2, 3 etc. Copy and paste the entire list into rails and press enter. That should delete these problem records.

Then, exit rails (type exit) and all you should need to do is run is:

rake posts:rebake
or
rake posts:rebake_uncooked_posts

Uncooked allows you to resume the rebake if it fails. I’d recommend just using normal rebake unless you have tons of uploads.

Then everything should work fine, hopefully. But there is a good chance that your optimized avatars will be broken like mine, but the originals should exist on s3.

You can check if avatars moved over successfully (or at least for some users) by running this in rails for a user who is displaying a default avatar:
User.find_by_username('username').uploaded_avatar

Also can check if optimized versions exist by using this:
OptimizedImage.where(upload_id: upload_id).where(version: 2)

3 Likes