Today I tried once again - did fresh installation, then restored from backup. Same errors everywhere. There’s no way to see what 1 media is missing, this 1 of 16545 uploads are not migrated to S3. S3 migration failed for db 'default'. is going to haunt me.
Bumping this thread, maybe someone could help out…
Some more information. This happens on Discourse (on Docker):
root@forum-app:/var/www/discourse# rake uploads:missing
--------------------------------------------------------------------------------
WARNING! WARNING! WARNING!
--------------------------------------------------------------------------------
default has uploads on S3!
validating without inventory is likely to take an enormous amount of time.
We recommend you run SKIP_EXTERNAL=1 rake uploads:missing to skip validating if on a multisite.
rake aborted!
PG::SyntaxError: ERROR: syntax error at end of input
LINE 1: INSERT INTO verified_ids VALUES
^
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/lib/file_store/s3_store.rb:423:in `list_missing'
/var/www/discourse/lib/file_store/s3_store.rb:303:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:277:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:270:in `block (2 levels) in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:255:in `block in <main>'
/var/www/discourse/lib/tasks/uploads.rake:281:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:missing_files
(See full trace by running task with --trace)
and
root@forum-app:/var/www/discourse# rake uploads:missing_files
--------------------------------------------------------------------------------
WARNING! WARNING! WARNING!
--------------------------------------------------------------------------------
default has uploads on S3!
validating without inventory is likely to take an enormous amount of time.
We recommend you run SKIP_EXTERNAL=1 rake uploads:missing to skip validating if on a multisite.
rake aborted!
PG::SyntaxError: ERROR: syntax error at end of input
LINE 1: INSERT INTO verified_ids VALUES
^
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.0/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/lib/file_store/s3_store.rb:423:in `list_missing'
/var/www/discourse/lib/file_store/s3_store.rb:303:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:277:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:270:in `block (2 levels) in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:255:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:missing_files
(See full trace by running task with --trace)
then:
root@forum-app:/var/www/discourse# rake posts:missing_uploads
Looking for missing uploads on: default
0 post uploads are missing.
root@forum-app:/var/www/discourse#
And - the main star - rake uploads:migrate_to_s3 --trace
root@forum-app:/var/www/discourse# rake uploads:migrate_to_s3 --trace
** Invoke uploads:migrate_to_s3 (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute uploads:migrate_to_s3
Please note that migrating to S3 is currently not reversible!
[CTRL+c] to cancel, [ENTER] to continue
Migrating uploads to S3 for 'default'...
Uploading files to S3...
- Listing local files
=> 3 files
- Listing S3 files
................... => 18784 files
- Syncing files to S3
...
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
10116 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 of 17210 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:138:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:79:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:389:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:127:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:491:in `exec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/exe/bundle:45:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/exe/bundle:33:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
root@forum-app:/var/www/discourse#