I’m trying to migrate to S3, but my migrate_to_s3 fails.
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
9756 posts were flagged for a rebake
1 of 16545 uploads are not migrated to S3. S3 migration failed for db 'default'.
Here’s output from rake posts:missing_uploads
Looking for missing uploads on: default
10 post uploads are missing.
10 uploads are missing.
9 of 108619 posts are affected.
Next I did enter rails console, checked all the posts with the missing uploads (all were .mp4 btw) and deleted them.
And then after trying - once again - to do the migration:
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
5178 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:138:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:389:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:127:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.1.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)
So after all of this, turned off and on again and unfortunately:
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
5493 posts were flagged for a rebake
1 of 16551 uploads are not migrated to S3. S3 migration failed for db 'default'.
Just edit your local copy of migrator as suggested in the topic. The value needs to be changed from 0 to 1 (since you have just one failing) this would fool the migrator into completion except for the one problem upload.
root@vps-9d50af6e:/var/discourse# ./launcher enter app
x86_64 arch detected.
root@vps-9d50af6e-app:/var/www/discourse# sed -i '85s/0/100/' lib/file_store/to_s3_migration.rb
root@vps-9d50af6e-app:/var/www/discourse# sed -i '77s/0/100/' lib/file_store/to_s3_migration.rb
root@vps-9d50af6e-app:/var/www/discourse# SKIP_FAILED=1 rake uploads:migrate_to_s3
Please note that migrating to S3 is currently not reversible!
[CTRL+c] to cancel, [ENTER] to continue
Migrating uploads to S3 for 'default'...
...........................................
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
5953 posts were flagged for a rebake
1 of 16552 uploads are not migrated to S3. S3 migration failed for db 'default'.
root@vps-9d50af6e-app:/var/www/discourse#
Even though - as you see before - I’ve “sed -i” these two lines in to_s3_migration.rb
I’m gonna add - what I think is an answer - but it doesn’t work great. It is something I’m gonna try in the future though:
Doing fresh installation with S3 + CDN, out of the box,
Moving all of the uploads from current server to S3,
Restoring from backup from old server,
Changing app.yml to newer one (that works with S3 + CDN).
So basically I’m gonna try and trick discourse to thing that everything was in the cloud all the way. But it’s not elegant solution to have to reinstall Discourse just because migrate_to_s3 fails miserably.
Also it should let me make a decision - do I need that X of Y uploads in s3? Because in that case (and when I was trying to find an answer to my question - in other cases I found) I’d happily get rid of that upload and move on.
Additional comment to this issue: every time when migrate_to_s3 fails, it doesn’t fail with uploading REST of the images. This could (and did) use lots and lots of data because with each failed migration it has put 99% of my data in S3.
And then because of migrate_to_s3 being irreversible, I have files on my own HDD and S3, at the same time. Because I can’t get rid of data on HDD. It’s still linked to posts. And I can’t go back to old structure, because it’s irreversible. And can’t put anything between like CDN, because - not everything is on S3.