Onebox 404 on Amazon Link - redirection forbidden

tarr11 · August 2, 2016, 12:49am

Hi - I’ve got a new install of Discourse (v1.5.3-Dirty) via Bitnami, hosted on an Azure A1 VM.

When i use amazon with a onebox link, I receive a 404.

I logged into the rails console via ssh, and was able to reproduce it, and received this error (using the sample from the Onebox github)

irb(main):002:0> url = "http://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2"
=> "http://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2"
irb(main):003:0> preview = Onebox.preview(url)
=> #<Onebox::Preview:0x007f4d96897ed0 @url="http://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2", @options=#<OpenStruct cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, connect_timeout=5, timeout=10, load_paths=["/opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/templates"], twitter_client=TwitterApi>, @cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, @engine_class=Onebox::Engine::AmazonOnebox>
irb(main):004:0> preview.to_s
RuntimeError: redirection forbidden: http://www.amazon.com/gp/aw/d/B005T3GRNW -> https://www.amazon.com/gp/aw/d/B005T3GRNW
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:223:in `open_loop'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
        from /opt/bitnami/apps/discourse/htdocs/lib/freedom_patches/open_uri_redirections.rb:70:in `open_uri'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:704:in `open'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:34:in `open'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine/html.rb:12:in `raw'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine/amazon_onebox.rb:50:in `data'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine.rb:71:in `block in record'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/moneta-0.8.0/lib/moneta/mixins.rb:165:in `fetch'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/engine.rb:71:in `record'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/layout_support.rb:9:in `layout'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/layout_support.rb:13:in `to_html'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/preview.rb:37:in `engine_html'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/lib/onebox/preview.rb:14:in `to_s'
        from (irb):4
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `block in exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `loop'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/cli.rb:5:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `load'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `<main>'
irb(main):005:0>

Any ideas?

mpalmer · August 2, 2016, 12:52am

Use the redirection target link, rather than the link that’s being redirected?

tarr11 · August 2, 2016, 12:54am

This doesn’t seem correct either (but it doesn’t error at least) …

irb(main):008:0> url = "https://www.amazon.com/gp/aw/d/B005T3GRNW"
=> "https://www.amazon.com/gp/aw/d/B005T3GRNW"
irb(main):009:0> preview = Onebox.preview(url)
=> #<Onebox::Preview:0x007f4d98cd28b8 @url="https://www.amazon.com/gp/aw/d/B005T3GRNW", @options=#<OpenStruct cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, connect_timeout=5, timeout=10, load_paths=["/opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/onebox-1.5.38/templates"], twitter_client=TwitterApi>, @cache=#<Moneta::Expires:0x007f4d9870b420 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x007f4d9870b4c0 @adapter=#<Moneta::Adapters::Memory:0x007f4d988a74a0 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, @engine_class=Onebox::Engine::AmazonOnebox>
irb(main):010:0> preview.to_s
=> ""

tarr11 · August 2, 2016, 12:56am

Just trying the original link here to see what happens

HTTP Link

HTTPS Link

https://www.amazon.com/gp/product/B005T3GRNW/ref=s9_simh_gw_p147_d0_i2

mpalmer · August 2, 2016, 12:57am

I get redirected to a different URL, and that one oneboxes up quite nicely:

[1] pry(main)> url = "https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW"
=> "https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW"                  
[2] pry(main)> preview = Onebox.preview(url)
=> #<Onebox::Preview:0x00562a9555b5c0       
 @cache=
  #<Moneta::Expires:0x00562a95c4f598
   @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x00562a95c4f638 @adapter=#<Moneta::Adapters::Memory:0x00562a95c15c80 @backend={}>, @features=[:increment, :create], @prefix="">,
   @default_expires=nil>,
 @engine_class=Onebox::Engine::AmazonOnebox,
 @options=
  #<OpenStruct cache=#<Moneta::Expires:0x00562a95c4f598 @adapter=#<Moneta::Transformer::JsonPrefixKeyJsonValue:0x00562a95c4f638 @adapter=#<Moneta::Adapters::Memory:0x00562a95c15c80 @backend={}>, @prefix="", @features=[:increment, :create]>, @default_expires=nil>, connect_timeout=5, timeout=10, load_paths=["/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/onebox-1.5.43/templates"], twitter_client=TwitterApi>,
 @url="https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW">
[3] pry(main)> preview.to_s
=> "<aside class=\"onebox amazon\">\n  <header class=\"source\">\n    <a href=\"https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW\">\n      \n      www.amazon.com\n    </a>\n  </header>\n  <article class=\"onebox-body\">\n    <img src=\"https://images-na.ssl-images-amazon.com/images/I/51jzvyPoV6L._SY400_.jpg\" class=\"thumbnail\"/>\n\n<h3><a href='https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW'>Seagate 1TB Desktop HDD SATA 6Gb/s 64MB Cache 3.5-Inch Internal Bare Drive (ST1000DM003)</a></h3>\n\n<p>\n\nBy now the world knows that Seagate Barracuda 7200 drives, from the world&#39;s premier storage provider, deliver years of reliable service and high performance. The ideal choice for workstations, desktop ...\n\n</p>\n<p><strong>\n$49.99\n<strong></p>\n\n  </article>\n  <div class=\"onebox-metadata\">\n    \n    \n  </div>\n  <div style=\"clear: both\"></div>\n</aside>\n"

tarr11 · August 2, 2016, 1:05am

What’s strange to me is that even if I pass in an https url, it does a redirect from http.

I just tried using OpenURI directly:

 open ("http://www.amazon.com/gp/aw/d/B005T3GRNW")
RuntimeError: redirection forbidden: http://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW -> https://www.amazon.com/Seagate-Desktop-3-5-Inch-Internal-ST1000DM003/dp/B005T3GRNW
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:223:in `open_loop'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
        from /opt/bitnami/apps/discourse/htdocs/lib/freedom_patches/open_uri_redirections.rb:70:in `open_uri'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:704:in `open'
        from /opt/bitnami/ruby/lib/ruby/2.1.0/open-uri.rb:34:in `open'
        from (irb):16
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:45:in `block in exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `loop'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/app_rails_loader.rb:34:in `exec_app_rails'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/cli.rb:5:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `require'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/bin/rails:9:in `<top (required)>'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `load'
        from /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/2.1.0/bin/rails:23:in `<main>'

but https works:

irb(main):017:0> open ("https://www.amazon.com/gp/aw/d/B005T3GRNW")
=> #<Tempfile:/tmp/open-uri20160802-44470-a00ltr>

tarr11 · August 2, 2016, 1:20am

Looking at the Onebox code, it appears that it is forcing it to be http which is causing OpenURI to blow up

https://github.com/discourse/onebox/blob/master/lib/onebox/engine/amazon_onebox.rb#L13-L20

codinghorror · August 2, 2016, 1:26am

Aha if that is the case we should use protocol agnostic URL there. Can you add a quick fix to onebox @techAPJ?

tarr11 · August 2, 2016, 1:39am

Hm, I’m not yet sure this is the problem. If it were, this would happen to everyone, which it obviously doesn’t…

riking · August 2, 2016, 3:23am

Just make it always https.

tgxworld · August 2, 2016, 7:16am

It isn’t forcing it to be HTTP but rather maintaining the same scheme as the URL that was provided

Anyway we’ll now force HTTPS on all URLs that go through Amazon onebox.

https://github.com/discourse/onebox/commit/0fd5ededa05961833ce45a243e7e9bea7b63f5f7

Topic		Replies	Views
Onebox not working after Discourse upgrade Bug	17	2883	November 1, 2017
Error upgrading 2.3.5 to 2.4.2 Installation unsupported-install	16	950	May 4, 2020
Cannot rebake after setting up CDN Installation unsupported-install	18	1256	April 18, 2021
"Error updating information, contact site admin" WordPress	26	4904	February 18, 2016
Amazon onebox fails on my site, but works here Bug	19	3755	March 31, 2016

Onebox 404 on Amazon Link - redirection forbidden

HTTP Link

HTTPS Link

Related topics