# S3 migration issues- some images got pointed to S3, but with the wrong URL format

**URL:** https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927
**Category:** Support
**Created:** [12.Март.2016 08:49:15 UTC](https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927 "2016-03-12T08:49:15Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![ckshen](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@ckshen](https://meta.discourse.org/u/ckshen)
#### Post date: [12.Март.2016 08:49:15 UTC](https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927/1 "2016-03-12T08:49:15Z")

</div>

Hi,  
I ran an migration of system images to an S3 bucket using the rake task here:  
# ./var/discourse/launcher enter app  
# rake uploads:migrate\_to\_s3

Two issues:  
- Some images’ link didn’t get rewritten to the S3 image URL  
- for those that did get rewritten, the rewritten URL, instead of the correct URL:  
`https://hungryonionstatic.s3-us-west-1.amazonaws.com/original/2X/5/54e7f74106b4b007b90ace5bbe9d432b87c5e80a.png`

It gets rewritten as  
`https://www.hungryonion.org//hungryonionstatic.s3-us-west-1.amazonaws.com/original/2X/5/54e7f74106b4b007b90ace5bbe9d432b87c5e80a.png`

Is there a way for me to rewrite all URLs in the DB that has the wrong URL back to the original URL? One thought is permalink but I’d rather correct the URL than redirect.

Alternatively I tried restoring the backup just before the rake task. But it failed.

Any pointers of what went wrong and how to fix it would be much appreciated! I’d rather not go through 2.4k topics to change the URLs manually if possible! Thank you in advance!

Also, any idea why some URLs got left alone after the rake task i.e. still pointing to the original location vs the S3 location while some was modified?

Here’s the restore log:

```
[2016-03-12 08:43:09] 'hungryonion' has started the restore!
[2016-03-12 08:43:09] Marking restore as running...
[2016-03-12 08:43:09] Making sure /var/www/discourse/tmp/restores/default/2016-03-12-084309 exists...
[2016-03-12 08:43:09] Copying archive to tmp directory...
[2016-03-12 08:43:16] Unzipping archive...
[2016-03-12 08:43:16] Extracting metadata file...
[2016-03-12 08:43:17] Validating metadata...
[2016-03-12 08:43:17] Current version: 20160308193142
[2016-03-12 08:43:17] EXCEPTION: undefined method `[]' for nil:NilClass
[2016-03-12 08:43:17] /var/www/discourse/lib/backup_restore/restorer.rb:177:in `validate_metadata'
/var/www/discourse/lib/backup_restore/restorer.rb:38:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:17:in `restore!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:80:in `restore'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/base.rb:198:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:92:in ` __run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.2/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/base.rb:137:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.2.5.2/lib/action_view/rendering.rb:30:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.9.2/lib/mini_profiler/profiling_methods.rb:76:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:237:in `block in action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:43:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/mapper.rb:49:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:815:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:129:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.2/lib/active_record/query_cache.rb:36:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:88:in ` __run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.1.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:22:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:10:in `call_with_quiet_assets'
/var/www/discourse/config/initializers/100-silence_logger.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.9.2/lib/mini_profiler/profiler.rb:167:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-2.0.0.beta.4/lib/message_bus/rack/middleware.rb:60:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:73:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/engine.rb:518:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/application.rb:165:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:562:in `process_client'
/var/www/discourse/lib/scheduler/defer.rb:85:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:658:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:132:in `start'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:22:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:22:in `<main>'
[2016-03-12 08:43:17] Trying to rollback...
[2016-03-12 08:43:17] There was no need to rollback
[2016-03-12 08:43:17] Notifying 'hungryonion' of the end of the restore...
[2016-03-12 08:43:17] Cleaning stuff up...
[2016-03-12 08:43:17] Removing tmp '/var/www/discourse/tmp/restores/default/2016-03-12-084309' directory...
[2016-03-12 08:43:17] Unpausing sidekiq...
[2016-03-12 08:43:17] Marking restore as finished...
[2016-03-12 08:43:17] Finished!

```

---

<div class="post-metadata">

### Author: ![michaeld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/michaeld/32/1594_2.png) [@michaeld](https://meta.discourse.org/u/michaeld)
#### Post date: [12.Март.2016 09:17:13 UTC](https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927/2 "2016-03-12T09:17:13Z")

</div>

> [@ckshen](#):
>
> Is there a way for me to rewrite all URLs in the DB that has the wrong URL back to the original URL?

Yes,  
`script/discourse remap https://www.hungryonion.org//hungryonionstatic.s3-us-west-1.amazonaws.com //hungryonionstatic.s3-us-west-1.amazonaws.com`

Seems like your backup is corrupted. Does it contain a file called `meta.json` ?

---

<div class="post-metadata">

### Author: ![ckshen](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@ckshen](https://meta.discourse.org/u/ckshen)
#### Post date: [12.Март.2016 09:26:28 UTC](https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927/3 "2016-03-12T09:26:28Z")

</div>

Thank you very much. Let me look into the remap solution.

It does- it says in the file (47 bytes):

{“source”:“discourse”,“version”:20160308193142}

---

<div class="post-metadata">

### Author: ![ckshen](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@ckshen](https://meta.discourse.org/u/ckshen)
#### Post date: [12.Март.2016 16:58:31 UTC](https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927/4 "2016-03-12T16:58:31Z")

</div>

The URL is actually written correctly in the posts, i.e. if i edit the post, the text is [//hungryonionstatic.s3-us-west-1.amazonaws.com/original/2X/5/54e7f74106b4b007b90ace5bbe9d432b87c5e80a.png](https://hungryonionstatic.s3-us-west-1.amazonaws.com/original/2X/5/54e7f74106b4b007b90ace5bbe9d432b87c5e80a.png)

which is the same format as if i upload a new image now. the HTML of each post just needs rebuilding, I think. I am going to try to rebake everything. For some reasons i thought rebaking wasn’t necessary. i will also look into permissions.

Ok. I think the original got pointed to S3. The optimized images are not as already discussed in the forums. The avatars as well. So I understand that part.

---

<div class="post-metadata">

### Author: ![ckshen](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@ckshen](https://meta.discourse.org/u/ckshen)
#### Post date: [15.Март.2016 19:12:10 UTC](https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927/5 "2016-03-15T19:12:10Z")

</div>

Rebaking fixes all the original image links.

---

<div class="post-metadata">

### Author: ![system](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/system/32/443519_2.png) [@system](https://meta.discourse.org/u/system)
#### Post date: [28.Декабрь.2018 19:58:33 UTC](https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927/6 "2018-12-28T19:58:33Z")

</div>



---

<div class="post-metadata">

### Author: ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [29.Декабрь.2018 07:58:24 UTC](https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927/8 "2018-12-29T07:58:24Z")

</div>

I [completely changed](https://github.com/discourse/discourse/commit/5381096bfd4e208657f928601d3792a5f0d1d0c8) the way the `migrate_to_s3` rake task work so this should not be an issue anymore 😉

---

<div class="post-metadata">

### Author: ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [29.Декабрь.2018 07:58:31 UTC](https://meta.discourse.org/t/s3-migration-issues-some-images-got-pointed-to-s3-but-with-the-wrong-url-format/40927/9 "2018-12-29T07:58:31Z")

</div>


