Hi,
I ran an migration of system images to an S3 bucket using the rake task here:
# ./var/discourse/launcher enter app
# rake uploads:migrate_to_s3
Two issues:
- Some images’ link didn’t get rewritten to the S3 image URL
- for those that did get rewritten, the rewritten URL, instead of the correct URL:
https://hungryonionstatic.s3-us-west-1.amazonaws.com/original/2X/5/54e7f74106b4b007b90ace5bbe9d432b87c5e80a.png
It gets rewritten as
https://www.hungryonion.org//hungryonionstatic.s3-us-west-1.amazonaws.com/original/2X/5/54e7f74106b4b007b90ace5bbe9d432b87c5e80a.png
Is there a way for me to rewrite all URLs in the DB that has the wrong URL back to the original URL? One thought is permalink but I’d rather correct the URL than redirect.
Alternatively I tried restoring the backup just before the rake task. But it failed.
Any pointers of what went wrong and how to fix it would be much appreciated! I’d rather not go through 2.4k topics to change the URLs manually if possible! Thank you in advance!
Also, any idea why some URLs got left alone after the rake task i.e. still pointing to the original location vs the S3 location while some was modified?
Here’s the restore log:
[2016-03-12 08:43:09] 'hungryonion' has started the restore!
[2016-03-12 08:43:09] Marking restore as running...
[2016-03-12 08:43:09] Making sure /var/www/discourse/tmp/restores/default/2016-03-12-084309 exists...
[2016-03-12 08:43:09] Copying archive to tmp directory...
[2016-03-12 08:43:16] Unzipping archive...
[2016-03-12 08:43:16] Extracting metadata file...
[2016-03-12 08:43:17] Validating metadata...
[2016-03-12 08:43:17] Current version: 20160308193142
[2016-03-12 08:43:17] EXCEPTION: undefined method `[]' for nil:NilClass
[2016-03-12 08:43:17] /var/www/discourse/lib/backup_restore/restorer.rb:177:in `validate_metadata'
/var/www/discourse/lib/backup_restore/restorer.rb:38:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:17:in `restore!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:80:in `restore'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/base.rb:198:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.2/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/base.rb:137:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.2.5.2/lib/action_view/rendering.rb:30:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.9.2/lib/mini_profiler/profiling_methods.rb:76:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:237:in `block in action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:43:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/mapper.rb:49:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:815:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:129:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.2/lib/active_record/query_cache.rb:36:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.1.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:22:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:10:in `call_with_quiet_assets'
/var/www/discourse/config/initializers/100-silence_logger.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.9.2/lib/mini_profiler/profiler.rb:167:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-2.0.0.beta.4/lib/message_bus/rack/middleware.rb:60:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:73:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/engine.rb:518:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/application.rb:165:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:562:in `process_client'
/var/www/discourse/lib/scheduler/defer.rb:85:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:658:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:132:in `start'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:22:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:22:in `<main>'
[2016-03-12 08:43:17] Trying to rollback...
[2016-03-12 08:43:17] There was no need to rollback
[2016-03-12 08:43:17] Notifying 'hungryonion' of the end of the restore...
[2016-03-12 08:43:17] Cleaning stuff up...
[2016-03-12 08:43:17] Removing tmp '/var/www/discourse/tmp/restores/default/2016-03-12-084309' directory...
[2016-03-12 08:43:17] Unpausing sidekiq...
[2016-03-12 08:43:17] Marking restore as finished...
[2016-03-12 08:43:17] Finished!