Can't migrate uploads to S3

Support
1

Hi, we’ve set up upload hosting (using the S3 settings) on Backblaze.
New uploads are working (after the x-amz-checksum-crc32 header fix), however trying to migrate existing uploads with uploads:migrate_to_s3 fails immediately.

This is on the latest Discourse version (3.5.0.beta5-dev).
Here is the full output:

root@goactuary-app:/var/www/discourse# rake uploads:migrate_to_s3 --trace                                                                                                                  
** Invoke uploads:migrate_to_s3 (first_time)                                                                                                                                               
** Invoke environment (first_time)                                                                                                                                                         
** Execute environment                                                                                                                                                                     
** Execute uploads:migrate_to_s3                                                                                                                                                           
Please note that migrating to S3 is currently not reversible!                                                                                                                              
[CTRL+c] to cancel, [ENTER] to continue                                                                                                                                                    
                                                                                                                                                                                           
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Running the migration, this may take a while...
rake aborted!                                                                                                                                                                              
FileStore::ToS3MigrationError: Some uploads could not be migrated to the new scheme. You need to fix this manually. (FileStore::ToS3MigrationError)
/var/www/discourse/lib/file_store/to_s3_migration.rb:156:in `migrate_to_s3'       
/var/www/discourse/lib/file_store/to_s3_migration.rb:59:in `migrate'                                                                                                                       
/var/www/discourse/lib/tasks/uploads.rake:126:in `migrate_to_s3'                                                                                                                           
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'                                                                                                        
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'                                                                                                                 
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'                                                                                                                         
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:188:in `invoke' 
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:188:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:147:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:132:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:214:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:80:in `run'
bin/rake:13:in `<top (required)>'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `load'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `kernel_load' 
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:452:in `exec'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:35:in `dispatch'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:29:in `start'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:28:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:20:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
3

You need to put them in your app.yml as described at Configure an S3 compatible object storage provider for uploads

Do you mean that you added code to your app.yml to revert the aws library to a version that works with backblaze?

2 Likes
4

Yes, they are in app.yml and I’ve rebuilt the app.
I’ve confirmed that newly uploaded files are going there.

Not quite. I’m using the following environmental variables:

  AWS_REQUEST_CHECKSUM_CALCULATION: WHEN_REQUIRED
  AWS_RESPONSE_CHECKSUM_VALIDATION: WHEN_REQUIRED

Those fixed not being able to upload new files after the settings change.

5

You’ll likely need to track those down yourself.

In rails you can

 Upload.pluck(:url)

And then look for which uploads are the problem.

I think in my experience that wasn’t enough, but you say it’s working for you.

1 Like
6

Manually downgrading the aws gems in the container did get the uploads:migrate_to_s3 job to actually upload images so apparently the env variables wasn’t enough, as you pointed out.

However the process failed later at a different point:

FileStore::ToS3MigrationError: 2 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'. (FileStore::ToS3MigrationError)

That will be a separate troubleshooting adventure.

1 Like
7

That may be because you have one boxes referring to local posts. I made a topic about it but can’t quite remember the details.

If it’s just 2,you might just ignore it, but in rails you can find all the uploads that don’t have your new bucket in them. I think it’ll be empty?

I think the issue is that you have two posts that have some unexpected path in the cooked post.

I found it!
Can’t restore a backup if it includes link to Discourse onebox

Edit: so I think you’ve got two posts with local one boxes. I think you just need to find and rebake those posts.