Hi, we’ve set up upload hosting (using the S3 settings) on Backblaze. New uploads are working (after the x-amz-checksum-crc32 header fix), however trying to migrate existing uploads with uploads:migrate_to_s3 fails immediately. This is on the latest Discourse version (3.5.0.beta5-dev). Here is the full output: root@goactuary-app:/var/www/discourse# rake uploads:migrate_to_s3 --trace ** Invoke uploads:migrate_to_s3 (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute uploads:migrate_to_s3 Please note that migrating to S3 is currently not reversible! [CTRL+c] to cancel, [ENTER] to continue Migrating uploads to S3 for 'default'... Some uploads were not migrated to the new scheme. Running the migration, this may take a while... rake aborted! FileStore::ToS3MigrationError: Some uploads could not be migrated to the new scheme. You need to fix this manually. (FileStore::ToS3MigrationError) /var/www/discourse/lib/file_store/to_s3_migration.rb:156:in `migrate_to_s3' /var/www/discourse/lib/file_store/to_s3_migration.rb:59:in `migrate' /var/www/discourse/lib/tasks/uploads.rake:126:in `migrate_to_s3' /var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `each_connection' /var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites' /var/www/discourse/lib/tasks/uploads.rake:100:in `block in ' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `each' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:188:in `invoke' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:188:in `invoke_task' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block (2 levels) in top_level' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `each' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block in top_level' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:147:in `run_with_threads' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:132:in `top_level' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:83:in `block in run' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:214:in `standard_exception_handling' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:80:in `run' bin/rake:13:in ` ' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `load' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `kernel_load' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:23:in `run' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:452:in `exec' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:35:in `dispatch' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:29:in `start' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:28:in `block in ' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors' /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:20:in ` ' /usr/local/bin/bundle:25:in `load' /usr/local/bin/bundle:25:in ` ' Tasks: TOP => uploads:migrate_to_s3

[תמונה] stanski: using the S3 settings You need to put them in your app.yml as described at Configure an S3 compatible object storage provider for uploads [תמונה] stanski: after the x-amz-checksum-crc32 header fix Do you mean that you added code to your app.yml to revert the aws library to a version that works with backblaze?

[תמונה] pfaffman: You need to put them in your app.yml as described at Configure an S3 compatible object storage provider for uploads Yes, they are in app.yml and I’ve rebuilt the app. I’ve confirmed that newly uploaded files are going there. [תמונה] pfaffman: Do you mean that you added code to your app.yml to revert the aws library to a version that works with backblaze? Not quite. I’m using the following environmental variables: AWS_REQUEST_CHECKSUM_CALCULATION: WHEN_REQUIRED AWS_RESPONSE_CHECKSUM_VALIDATION: WHEN_REQUIRED Those fixed not being able to upload new files after the settings change.

[תמונה] stanski: Some uploads were not migrated to the new scheme. Running the migration, this may take a while... rake aborted! You’ll likely need to track those down yourself. In rails you can Upload.pluck(:url) And then look for which uploads are the problem. [תמונה] stanski: I’m using the following environmental variables: I think in my experience that wasn’t enough, but you say it’s working for you.

Manually downgrading the aws gems in the container did get the uploads:migrate_to_s3 job to actually upload images so apparently the env variables wasn’t enough, as you pointed out. However the process failed later at a different point: FileStore::ToS3MigrationError: 2 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'. (FileStore::ToS3MigrationError) That will be a separate troubleshooting adventure.

That may be because you have one boxes referring to local posts. I made a topic about it but can’t quite remember the details. If it’s just 2,you might just ignore it, but in rails you can find all the uploads that don’t have your new bucket in them. I think it’ll be empty? I think the issue is that you have two posts that have some unexpected path in the cooked post. I found it! Can’t restore a backup if it includes link to Discourse onebox Edit: so I think you’ve got two posts with local one boxes. I think you just need to find and rebake those posts.

[תמונה] pfaffman: Can’t restore a backup if it includes link to Discourse onebox Edit: so I think you’ve got two posts with local one boxes. I think you just need to find and rebake those posts. @stanski did this solve your problem?

Sorry, got caught up and forgot to update this. After rebaking all posts, I got a different error message when trying the migration again. I can’t find the backtrace anymore unfortunately. I realized that despite of the error, it appeared that uploads were migrated to Backblaze and the posts updated. I archived the local uploads and moved them elsewhere and have not heard any complaints since. So in conclusion, yes, my issue was fixed. Thanks to @pfaffman for the continued suggestions. Now I have a complaint that various firewalls are blocking backblaze-hosted uploads, but that’s another matter.

[תמונה] stanski: Now I have a complaint that various firewalls are blocking backblaze-hosted uploads, but that’s another matter You really want to use a cdn, I think.

לא מצליחים להעביר את ההעלאות ל-S3

תמיכה

pfaffman (Jay Pfaffman) 12 ביוני,‏ 2025,‏ 11:28pm 10

You really want to use a cdn, I think.

נושא		תגובות	צפיות
Migrating old uploads to S3 Self-hosting s3	8	1777	11 במאי,‏ 2021
Unable to migrate to S3, therefore unable to restore from backup Self-hosting	2	1377	9 במרץ,‏ 2022
Uploads migration to S3 fails Product Support	26	3681	7 במאי,‏ 2021
Restore process cancelled at migrating uploads to S3 step Product Support	48	4033	24 בינואר,‏ 2026
Migrating Site to S3 Self-hosting s3	2	821	25 במאי,‏ 2020

לא מצליחים להעביר את ההעלאות ל-S3

נושאים קשורים