Onebox 404 on Amazon Link - redirection forbidden


(tarr11) #1

Hi - I’ve got a new install of Discourse (v1.5.3-Dirty) via Bitnami, hosted on an Azure A1 VM.

When i use amazon with a onebox link, I receive a 404.

I logged into the rails console via ssh, and was able to reproduce it, and received this error (using the sample from the Onebox github)

irb(main):002:0> url = "http://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2"
=> "http://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2"
irb(main):003:0> preview = Onebox.preview(url)
=> #<Onebox::Preview:0x007f4d96897ed0 @url="http://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2", @options=#<OpenStruct cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, connect_timeout=5, timeout=10, load_paths=["/opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/templates"], twitter_client=TwitterApi>, @cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, @engine_class=Onebox::Engine::AmazonOnebox>
irb(main):004:0> preview.to_s
RuntimeError: redirection forbidden: http://www.amazon.com/gp/aw/d/B005T3GRNW -> https://www.amazon.com/gp/aw/d/B005T3GRNW
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:223:in `open_loop'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
        from /opt/bitnami/apps/discourse/htdocs/lib/freedom_patches/open_uri_redirections.rb:70:in `open_uri'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:704:in `open'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:34:in `open'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine/html.rb:12:in `raw'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine/amazon_onebox.rb:50:in `data'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine.rb:71:in `block in record'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/moneta-0.8.0/lib/moneta/mixins.rb:165:in `fetch'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine.rb:71:in `record'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/layout_support.rb:9:in `layout'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/layout_support.rb:13:in `to_html'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/preview.rb:37:in `engine_html'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/preview.rb:14:in `to_s'
        from (irb):4
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `block in exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `loop'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/cli.rb:5:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `load'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `<main>'
irb(main):005:0> 

Any ideas?


(Matt Palmer) #2

Use the redirection target link, rather than the link that’s being redirected?


(tarr11) #3

This doesn’t seem correct either (but it doesn’t error at least) …

irb(main):008:0> url = "https://www.amazon.com/gp/aw/d/B005T3GRNW"
=> "https://www.amazon.com/gp/aw/d/B005T3GRNW"
irb(main):009:0> preview = Onebox.preview(url)
=> #<Onebox::Preview:0x007f4d98cd28b8 @url="https://www.amazon.com/gp/aw/d/B005T3GRNW", @options=#<OpenStruct cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, connect_timeout=5, timeout=10, load_paths=["/opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/templates"], twitter_client=TwitterApi>, @cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, @engine_class=Onebox::Engine::AmazonOnebox>
irb(main):010:0> preview.to_s
=> ""

(tarr11) #4

Just trying the original link here to see what happens

HTTP Link

HTTPS Link


(Matt Palmer) #5

I get redirected to a different URL, and that one oneboxes up quite nicely:

[1] pry(main)> url = "https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW"
=> "https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW"                  
[2] pry(main)> preview = Onebox.preview(url)
=> #<Onebox::Preview:0x00562a9555b5c0       
 @cache=
  #<Moneta::Expires:0x00562a95c4f598
   @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x00562a95c4f638 @adapter=#<Moneta::Adapters::Memory:0x00562a95c15c80 @backend={}>, @features=[:increment, :create], @prefix="">,
   @default_expires=nil>,
 @engine_class=Onebox::Engine::AmazonOnebox,
 @options=
  #<OpenStruct cache=#<Moneta::Expires:0x00562a95c4f598 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x00562a95c4f638 @adapter=#<Moneta::Adapters::Memory:0x00562a95c15c80 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, connect_timeout=5, timeout=10, load_paths=["/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/onebox-1.5.43/templates"], twitter_client=TwitterApi>,
 @url="https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW">
[3] pry(main)> preview.to_s
=> "<aside class=\"onebox amazon\">\n  <header class=\"source\">\n    <a href=\"https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW\">\n      \n      www.amazon.com\n    </a>\n  </header>\n  <article class=\"onebox-body\">\n    <img src=\"https://images-na.ssl-images-amazon.com/images/I/51jzvyPoV6L._SY400_.jpg\" class=\"thumbnail\"/>\n\n<h3><a href='https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW'>Seagate 1TB Desktop HDD SATA 6Gb/s 64MB Cache 3.5-Inch Internal Bare Drive (ST1000DM003)</a></h3>\n\n<p>\n\nBy now the world knows that Seagate Barracuda 7200 drives, from the world&#39;s premier storage provider, deliver years of reliable service and high performance. The ideal choice for workstations, desktop ...\n\n</p>\n<p><strong>\n$49.99\n<strong></p>\n\n  </article>\n  <div class=\"onebox-metadata\">\n    \n    \n  </div>\n  <div style=\"clear: both\"></div>\n</aside>\n"

(tarr11) #6

What’s strange to me is that even if I pass in an https url, it does a redirect from http.

I just tried using OpenURI directly:

 open ("http://www.amazon.com/gp/aw/d/B005T3GRNW")
RuntimeError: redirection forbidden: http://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW -> https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:223:in `open_loop'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
        from /opt/bitnami/apps/discourse/htdocs/lib/freedom_patches/open_uri_redirections.rb:70:in `open_uri'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:704:in `open'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:34:in `open'
        from (irb):16
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `block in exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `loop'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/cli.rb:5:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `load'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `<main>'

but https works:

irb(main):017:0> open ("https://www.amazon.com/gp/aw/d/B005T3GRNW")
=> #<Tempfile:/tmp/open-uri20160802-44470-a00ltr>

(tarr11) #7

Looking at the Onebox code, it appears that it is forcing it to be http which is causing OpenURI to blow up


(Jeff Atwood) #8

Aha if that is the case we should use protocol agnostic URL there. Can you add a quick fix to onebox @techAPJ?


(tarr11) #9

Hm, I’m not yet sure this is the problem. If it were, this would happen to everyone, which it obviously doesn’t…


(Kane York) #10

Just make it always https.


(Alan Tan) #11

It isn’t forcing it to be HTTP but rather maintaining the same scheme as the URL that was provided :slight_smile:

Anyway we’ll now force HTTPS on all URLs that go through Amazon onebox.


(Jeff Atwood) #12