Onebox 404 on Amazon Link - redirection forbidden

Hi - I’ve got a new install of Discourse (v1.5.3-Dirty) via Bitnami, hosted on an Azure A1 VM.

When i use amazon with a onebox link, I receive a 404.

I logged into the rails console via ssh, and was able to reproduce it, and received this error (using the sample from the Onebox github)

irb(main):002:0> url = "http://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2"
=> "http://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2"
irb(main):003:0> preview = Onebox.preview(url)
=> #<Onebox::Preview:0x007f4d96897ed0 @url="http://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2", @options=#<OpenStruct cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, connect_timeout=5, timeout=10, load_paths=["/opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/templates"], twitter_client=TwitterApi>, @cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, @engine_class=Onebox::Engine::AmazonOnebox>
irb(main):004:0> preview.to_s
RuntimeError: redirection forbidden: http://www.amazon.com/gp/aw/d/B005T3GRNW -> https://www.amazon.com/gp/aw/d/B005T3GRNW
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:223:in `open_loop'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
        from /opt/bitnami/apps/discourse/htdocs/lib/freedom_patches/open_uri_redirections.rb:70:in `open_uri'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:704:in `open'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:34:in `open'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine/html.rb:12:in `raw'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine/amazon_onebox.rb:50:in `data'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine.rb:71:in `block in record'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/moneta-0.8.0/lib/moneta/mixins.rb:165:in `fetch'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine.rb:71:in `record'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/layout_support.rb:9:in `layout'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/layout_support.rb:13:in `to_html'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/preview.rb:37:in `engine_html'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/preview.rb:14:in `to_s'
        from (irb):4
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `block in exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `loop'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/cli.rb:5:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `load'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `<main>'
irb(main):005:0> 

Any ideas?

Use the redirection target link, rather than the link that’s being redirected?

This doesn’t seem correct either (but it doesn’t error at least) …

irb(main):008:0> url = "https://www.amazon.com/gp/aw/d/B005T3GRNW"
=> "https://www.amazon.com/gp/aw/d/B005T3GRNW"
irb(main):009:0> preview = Onebox.preview(url)
=> #<Onebox::Preview:0x007f4d98cd28b8 @url="https://www.amazon.com/gp/aw/d/B005T3GRNW", @options=#<OpenStruct cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, connect_timeout=5, timeout=10, load_paths=["/opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/templates"], twitter_client=TwitterApi>, @cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, @engine_class=Onebox::Engine::AmazonOnebox>
irb(main):010:0> preview.to_s
=> ""

Just trying the original link here to see what happens

HTTP Link

HTTPS Link

https://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2

I get redirected to a different URL, and that one oneboxes up quite nicely:

[1] pry(main)> url = "https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW"
=> "https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW"                  
[2] pry(main)> preview = Onebox.preview(url)
=> #<Onebox::Preview:0x00562a9555b5c0       
 @cache=
  #<Moneta::Expires:0x00562a95c4f598
   @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x00562a95c4f638 @adapter=#<Moneta::Adapters::Memory:0x00562a95c15c80 @backend={}>, @features=[:increment, :create], @prefix="">,
   @default_expires=nil>,
 @engine_class=Onebox::Engine::AmazonOnebox,
 @options=
  #<OpenStruct cache=#<Moneta::Expires:0x00562a95c4f598 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x00562a95c4f638 @adapter=#<Moneta::Adapters::Memory:0x00562a95c15c80 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, connect_timeout=5, timeout=10, load_paths=["/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/onebox-1.5.43/templates"], twitter_client=TwitterApi>,
 @url="https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW">
[3] pry(main)> preview.to_s
=> "<aside class=\"onebox amazon\">\n  <header class=\"source\">\n    <a href=\"https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW\">\n      \n      www.amazon.com\n    </a>\n  </header>\n  <article class=\"onebox-body\">\n    <img src=\"https://images-na.ssl-images-amazon.com/images/I/51jzvyPoV6L._SY400_.jpg\" class=\"thumbnail\"/>\n\n<h3><a href='https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW'>Seagate 1TB Desktop HDD SATA 6Gb/s 64MB Cache 3.5-Inch Internal Bare Drive (ST1000DM003)</a></h3>\n\n<p>\n\nBy now the world knows that Seagate Barracuda 7200 drives, from the world&#39;s premier storage provider, deliver years of reliable service and high performance. The ideal choice for workstations, desktop ...\n\n</p>\n<p><strong>\n$49.99\n<strong></p>\n\n  </article>\n  <div class=\"onebox-metadata\">\n    \n    \n  </div>\n  <div style=\"clear: both\"></div>\n</aside>\n"

What’s strange to me is that even if I pass in an https url, it does a redirect from http.

I just tried using OpenURI directly:

 open ("http://www.amazon.com/gp/aw/d/B005T3GRNW")
RuntimeError: redirection forbidden: http://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW -> https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:223:in `open_loop'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
        from /opt/bitnami/apps/discourse/htdocs/lib/freedom_patches/open_uri_redirections.rb:70:in `open_uri'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:704:in `open'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:34:in `open'
        from (irb):16
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `block in exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `loop'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/cli.rb:5:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `load'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `<main>'

but https works:

irb(main):017:0> open ("https://www.amazon.com/gp/aw/d/B005T3GRNW")
=> #<Tempfile:/tmp/open-uri20160802-44470-a00ltr>

Looking at the Onebox code, it appears that it is forcing it to be http which is causing OpenURI to blow up

https://github.com/discourse/onebox/blob/master/lib/onebox/engine/amazon_onebox.rb#L13-L20

2 Likes

Aha if that is the case we should use protocol agnostic URL there. Can you add a quick fix to onebox @techAPJ?

Hm, I’m not yet sure this is the problem. If it were, this would happen to everyone, which it obviously doesn’t…

Just make it always https.

It isn’t forcing it to be HTTP but rather maintaining the same scheme as the URL that was provided :slight_smile:

Anyway we’ll now force HTTPS on all URLs that go through Amazon onebox.

https://github.com/discourse/onebox/commit/0fd5ededa05961833ce45a243e7e9bea7b63f5f7

4 Likes