Onebox not working after Discourse upgrade


(John Lynch) #1

I upgraded my Discourse install this morning to v1.9.0.beta14 +35. After doing so, I noticed that Onebox wasn’t working anymore. It wouldn’t Onebox links I pasted into the body nor the title of a post. It seemed like it was attempting to do so, but the process was failing. I tried links from multiple sites that had worked previously. I also entered the Docker instance and checked that I could fetch the page using curl. That also worked just fine.

I tried hunting around in Discourse settings to see if something had changed, but didn’t see anything jump out at me. I decided perhaps I’d better rebuild the app. I didn’t have sufficient disk space to do this, so I ran apt-get autoclean to free space up. Afterwards, I tried rebuilding the app. This failed, and failed in such a way that Docker would not respond to any command from the launcher (the rebuild had hung for a considerable period of time attempting to stop the docker instance for a rebuild, and I had to Ctrl+C the rebuild attempt). At this point, I rebooted the entire machine and was able to rebuild the app afterwards.

Unfortunately, this did not fix the Onebox problem. I decided perhaps I had better git pull the latest image. I did this, went through a merge (though I’m not quite sure why), and then attempted to rebuild the app again. Unfortunately, now I’m seeing this when I run sudo ./launcher rebuild app:

ERROR: Docker version 1.13.1 not supported, please upgrade to at least 17.03.1, or recommended 17.06.2

I had successfully run sudo ./launcher rebuild app immediately prior to pulling the latest Discourse image using git. I have no idea why after pulling the latest it would start failing. (EDIT: I’m a newb still learning about server administration and had to apt-get upgrade; carry on.)

I’m now totally lost. I’m hoping someone can help me:

1. Fix the launcher/Docker issue so that I can at least do normal maintenance on my server.
2. Fix the Onebox issue so I can have good looking links again on my posts. Perhaps this is a known issue that I’m not aware of?

I did my best to search the site here to see if anyone else was having these issues, so I apologize if this is a duplicate question. Any help would be greatly appreciated.


(John Lynch) #2

OK, issue #1 is just my inexperience with Docker and server administration catching up with me. A bit more poking around indicates that I probably hadn’t done an apt-get upgrade in a while. That got me past the Docker version error and the rebuild is now underway. I can report back on the Onebox issue if/when that rebuild succeeds. Apologies if I wasted anyone’s time.


(John Lynch) #3

Issue #2 remains unresolved. My links will not create Oneboxes when I paste links into titles or bodies of posts. I don’t know what to do about that at this point. Again, using curl to access the page from within the Docker instance works just fine.


(John Lynch) #4

I should also mention that when I check sidekiq, it does not appear to be busy. It is not working on anything and no jobs are enqueued.


(Joshua Rosenfeld) #5

Any errors in your logs? See Where are all the Discourse logs. The logs web UI may have relevant information. So may your browser console.


(John Lynch) #6

I didn’t see anything suspicious in the main logs (at mydomain.com/logs). I can dig deeper.


(Joshua Rosenfeld) #7

Is this a completely standard install via discourse/INSTALL-cloud.md at master · discourse/discourse · GitHub? Any NGINX, HAProxy, CloudFlare, or other proxy in front?


(John Lynch) #8

Yeah, those are the instructions I followed. The only real thing I did was to setup SSH support. I don’t believe I’m using any form of proxy; I did not set up a CDN. I did set up an Amazon S3 instance to handle uploads; no problems there.


(John Lynch) #9

OK. I found a log item in production.log that probably explains it, but I’m not sure what can be done about it:

Started GET "/onebox?url=http%3A%2F%2Fwww.espn.com%2Fmens-college-basketball%2Fstory%2F_%2Fid%2F21251356%2Fduke-blue-devils-ranked-no-1-ap-preseason-men-basketball-poll-2nd-straight-year&refresh=false&user_id=3" for 192.241.57.20 at 2017-11-01 16:54:57 +0000
Processing by OneboxController#show as HTML
  Parameters: {"url"=>"http://www.espn.com/mens-college-basketball/story/_/id/21251356/duke-blue-devils-ranked-no-1-ap-preseason-men-basketball-poll-2nd-straight-year", "refresh"=>"false", "user_id"=>"3"}
Failed to onebox http://www.espn.com/mens-college-basketball/story/_/id/21251356/duke-blue-devils-ranked-no-1-ap-preseason-men-basketball-poll-2nd-straight-year undefined method `strip' for nil:NilClass ["/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/engine/whitelisted_generic_onebox.rb:220:in `data'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/engine/whitelisted_generic_onebox.rb:247:in `is_article?'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/engine/whitelisted_generic_onebox.rb:239:in `generic_html'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/engine/whitelisted_generic_onebox.rb:186:in `to_html'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/preview.rb:35:in `engine_html'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/preview.rb:16:in `to_s'", "/var/www/discourse/lib/oneboxer.rb:166:in `block in onebox_raw'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:297:in `block in fetch'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:586:in `block in save_block_result_to_cache'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:555:in `block in instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:168:in `instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:555:in `instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:585:in `save_block_result_to_cache'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:297:in `fetch'", "/var/www/discourse/lib/oneboxer.rb:153:in `onebox_raw'", "/var/www/discourse/lib/oneboxer.rb:31:in `preview'", "/var/www/discourse/app/controllers/onebox_controller.rb:20:in `show'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:186:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/rendering.rb:30:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:20:in `block in process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:131:in `run_callbacks'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:19:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/rescue.rb:20:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `block in instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:30:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/railties/controller_runtime.rb:22:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:124:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionview-5.1.4/lib/action_view/rendering.rb:30:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-mini-profiler-0.10.5/lib/mini_profiler/profiling_methods.rb:76:in `block in profile_method'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal.rb:189:in `dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal.rb:253:in `dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:31:in `serve'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:50:in `block in serve'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `each'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `serve'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:834:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-protection-2.0.0/lib/rack/protection/frame_options.rb:31:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/builder.rb:63:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/conditional_get.rb:25:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'", "/var/www/discourse/lib/middleware/anonymous_cache.rb:149:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/cookies.rb:613:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:97:in `run_callbacks'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:24:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/logster-1.2.8/lib/logster/middleware/reporter.rb:31:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/rack/logger.rb:36:in `call_app'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/rack/logger.rb:26:in `call'", "/var/www/discourse/config/initializers/100-quiet_logger.rb:16:in `call'", "/var/www/discourse/config/initializers/100-silence_logger.rb:29:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/request_id.rb:25:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/executor.rb:12:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-mini-profiler-0.10.5/lib/mini_profiler/profiler.rb:171:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.0.8/lib/message_bus/rack/middleware.rb:63:in `call'", "/var/www/discourse/lib/middleware/request_tracker.rb:110:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/engine.rb:522:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/urlmap.rb:68:in `block in call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `each'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_server.rb:606:in `process_client'", "/var/www/discourse/lib/scheduler/defer.rb:87:in `process_client'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_server.rb:702:in `worker_loop'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_server.rb:142:in `start'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/bin/unicorn:126:in `<top (required)>'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/bin/unicorn:23:in `load'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/bin/unicorn:23:in `<main>'"]
  Rendering text template
  Rendered text template (0.0ms)
Completed 404 Not Found in 366ms (Views: 0.9ms | ActiveRecord: 1.7ms)

Any thoughts? I can try spelunking around in that code myself, but my knowledge of Discourse’s source is still very limited.


(John Lynch) #10

Pretty sure I found the culprit:

This commit added lines to parse some Twitter stuff that call split on some values returned from a dictionary whether those keys exist or not. When those keys don’t exist in the dictionary, split is invoked on nil. This commit is also currently failing automated tests. Fantastic.

Do I have something misconfigured such that I’m pulling down changes that haven’t passed automated testing? Because if there’s a way for me to set my install up to not get changes that haven’t passed automated testing, that would be great.


(Arpit Jalan) #12

The onebox gem tests are not integrated in core Discourse source code, so the tests-passed branch does not take that into consideration. It was an error on my part to not run specs before pushing the commit and I apologize for that.

Pushing a fix and cutting a new onebox version now.


(John Lynch) #13

And, as we would expect from the commit above, Twitter links do Onebox correctly, since they have the necessary metadata to avoid the undefined method exception. I have not found any other links that Onebox correctly, so I’m guessing that Twitter and only Twitter links will Onebox at the moment.


(Arpit Jalan) #14

No, any site that supports twitter label meta tags will onebox, also all the sites that has dedicated onebox engine (for example: wikipedia, github, amazon, etc) will onebox correctly.


(John Lynch) #15

Ironically, Oneboxing Discourse pages also works, so there’s a non-Twitter example of the Onebox code working.


(John Lynch) #16

Gotcha. Thanks for clarifying.


(Arpit Jalan) #17

Pushed a fix. Updating to latest Discourse version will fix this issue.

Thanks for bringing this issue in our notice @john-lynch.


(John Lynch) #18

@techAPJ: Thanks so much. I have successfully upgraded to latest and the problem is solved.


(Arpit Jalan) #19