How can I migrate files from the old upload scheme (default/XXX) to the new one (default/original/{1,2}X)?

[kansaichris]

@zogstrip

My bad, it’s a manual process.

Oh, okay. Does this mean that I will also need to disable the setting (i.e. run SiteSetting.migrate_to_new_scheme = false) after the migration process is complete?

@zogstrip

So, the UploadRecovery class that @tgxworld built is an improvement over the uploads:recover_from_tombstone . They both do the same thing, only the UploadRecovery works for both the local and S3 storages, whereas the rake task only works for the local storage.

Oh, neat! Thanks for the clarification.

@zogstrip

What @sam suggested won’t restore uploads that were put in the tombstone but is needed to be able to restore them.

I think I understand what you mean, but just to be clear: are you saying that changing the value of SiteSetting.migrate_to_new_scheme to true will tell Discourse to migrate all uploads to the new storage scheme, but it won’t touch any uploads that have already been placed in the tombstone directory (i.e. you need to move uploads out of the tombstone directory before they can be migrated)?

@zogstrip

So, we’re what you should do

./launcher enter app
rails c
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

Hmm…do I need to run Jobs::MigrateUploadScheme.new.execute(nil) manually, or will it automatically be scheduled to run eventually (hence @sam’s advice to “wait a day”)?

In any event, I changed SiteSetting.migrate_to_new_scheme to true from the Rails console in my Discourse container yesterday. Interestingly enough, all of the numbered subdirectories under /var/discourse/shared/standalone/uploads/default/ (e.g. 100, 101, 102, 103, 104, and 105) still appear to exist, and I found seven messages like the following in my error logs:

Job exception: undefined method `unlink' for #<File:0x00007f368703ab00>

Here’s a backtrace, in case you’re interested:

/var/www/discourse/app/models/optimized_image.rb:410:in `block in migrate_to_new_scheme'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.2/lib/active_record/relation/delegation.rb:71:in `each'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.2/lib/active_record/relation/delegation.rb:71:in `each'
/var/www/discourse/app/models/optimized_image.rb:367:in `migrate_to_new_scheme'
/var/www/discourse/app/jobs/scheduled/migrate_upload_scheme.rb:28:in `execute'
/var/www/discourse/app/jobs/base.rb:196:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:185:in `block in perform'
/var/www/discourse/app/jobs/base.rb:181:in `each'
/var/www/discourse/app/jobs/base.rb:181:in `perform'
/var/www/discourse/app/jobs/base.rb:243:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:82:in `process_queue'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:30:in `block in initialize'

At this point, would it still be safe for me to run the following commands or should I hold off until I have investigated the errors above?

cd /var/discourse/
sudo ./launcher enter app
rails c
> Jobs::MigrateUploadScheme.new.execute(nil)
[...]
> Jobs::MigrateUploadScheme.new.execute(nil)
> exit
rake posts:rebake
rake uploads:recover

Thanks again for all your help!