Hello! I’m having a lot of trouble getting my local file uploads to be migrated to S3. I’ve done the following: Restored to a new environment Disabled “enable S3 uploads” Ranrails c SiteSetting.migrate_to_new_scheme = true Jobs::MigrateUploadScheme.new.execute(nil) #repeatedly until it stops taking a long time exit rake posts:rebake rake uploads:recover DISCOURSE_S3_REGION=us-east-1 DISCOURSE_S3_USE_IAM_PROFILE=true DISCOURSE_S3_BUCKET= rake uploads:migrate_to_s3 Basically, I’ve referenced these: How can I migrate files from the old upload scheme (default/XXX) to the new one (default/original/{1,2}X)? - #5 by kansaichris [image] Migrating uploaded files from DO to S3 Support I was thinking to migrate all the files already uploaded to my discourse site to S3 and then continue serving them from S3 instead of Digital Ocean local disk. (I already managed to set-up the backups to be stored on another S3 bucket.) What is the correct procedure to set-up S3 for file uploads? If I just add the s3 upload bucket name and s3 cdn url names, enable Place uploads on Amazon S3 storage, and discourse will take care of the migration, or is there something else I need to do? Do I ne… I feel like I’m really close to getting this to work, but I’m stuck at these errors: ** Invoke uploads:migrate_to_s3 (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute uploads:migrate_to_s3 Migrating uploads to S3 for 'default'... Some uploads were not migrated to the new scheme. Running the migration, this may take a while... rake aborted! FileStore::ToS3MigrationError: Some uploads could not be migrated to the new scheme. You need to fix this manually. /var/www/discourse/lib/file_store/to_s3_migration.rb:155:in `migrate_to_s3' /var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate' /var/www/discourse/lib/tasks/uploads.rake:245:in `migrate_to_s3' /var/www/discourse/lib/tasks/uploads.rake:224:in `block in migrate_to_s3_all_sites' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:76:in `with_connection' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:86:in `each_connection' /var/www/discourse/lib/tasks/uploads.rake:222:in `migrate_to_s3_all_sites' /var/www/discourse/lib/tasks/uploads.rake:218:in `block in ' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `each' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain' /usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `each' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:80:in `run' bin/rake:13:in ` ' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:46:in `block in ' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors' /usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:34:in ` ' /usr/local/bin/bundle:23:in `load' /usr/local/bin/bundle:23:in ` ' Tasks: TOP => uploads:migrate_to_s3 Do any of you have any advice on where I should turn next? Edit: More info here, I seem to still have uploads in the original scheme: [1] pry(main)> Upload.where("url NOT LIKE '//%' AND url NOT LIKE '/uploads/default/original/_X/%'").exists? => true

Migrating uploads to S3 fails

ajschmidt (AJ Schmidt) October 25, 2024, 10:19pm 2

did you ever get this resolved? I’m also facing this issue

Topic		Replies	Views
Migrating Site to S3 Self-hosting s3	2	819	May 25, 2020
Migrating old uploads to S3 Self-hosting s3	8	1774	May 11, 2021
Rake migrate_to_s3 errors Self-hosting s3	3	846	February 4, 2020
How to migrate old uploads to S3? Product Support	3	1332	July 22, 2021
Migrating uploaded files from DO to S3 Product Support	2	2045	June 5, 2020