Migrate_to_S3 Fails on Rebake

[Zup]

I too am getting this error after importing a smf2 database, enabling S3, and running rake --trace uploads:migrate_to_s3

Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
.. => 2244 files
 - Listing S3 files
... => 2383 files
 - Syncing files to S3
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
454 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 7165 of 9554 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:373:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:66:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:123:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:102:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-3.1.0/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-3.1.0/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:100:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:96:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/cli.rb:477:in `exec'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/cli.rb:31:in `dispatch'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/cli.rb:25:in `start'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/exe/bundle:49:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/lib/bundler/friendly_errors.rb:128:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.26/exe/bundle:37:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_to_s3

@markersocial

Then run:
Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '/uploads/default/original/_X/%'").to_a

This should find uploads that still have issues and couldn’t be migrated. In my case, all of these uploads existed as records in the db, but the actual images were not existing.

Depending on how big the list is, you can copy page then find + replace to make a list of commands to remove these problem records via the list of upload ids.

markersocial, when I run Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '/uploads/default/original/_X/%'").to_a the output is []

I’m presuming that means my issue is different than yours?

@ipoopfool

I ran into a similar problem, possibly due to migrating from AWS S3 → local → DO Spaces. I ended up up writing some ruby code to manually replace old broken upload urls with the working updated url format and it seems to have fixed my problem.

ipoopfool, I think I might be experiencing the same issue as you. Would you mind sharing your ruby code that fixed this for you?