Onebox not working after Discourse upgrade

Bug
1

I upgraded my Discourse install this morning to v1.9.0.beta14 +35. After doing so, I noticed that Onebox wasn’t working anymore. It wouldn’t Onebox links I pasted into the body nor the title of a post. It seemed like it was attempting to do so, but the process was failing. I tried links from multiple sites that had worked previously. I also entered the Docker instance and checked that I could fetch the page using curl. That also worked just fine.

I tried hunting around in Discourse settings to see if something had changed, but didn’t see anything jump out at me. I decided perhaps I’d better rebuild the app. I didn’t have sufficient disk space to do this, so I ran apt-get autoclean to free space up. Afterwards, I tried rebuilding the app. This failed, and failed in such a way that Docker would not respond to any command from the launcher (the rebuild had hung for a considerable period of time attempting to stop the docker instance for a rebuild, and I had to Ctrl+C the rebuild attempt). At this point, I rebooted the entire machine and was able to rebuild the app afterwards.

Unfortunately, this did not fix the Onebox problem. I decided perhaps I had better git pull the latest image. I did this, went through a merge (though I’m not quite sure why), and then attempted to rebuild the app again. Unfortunately, now I’m seeing this when I run sudo ./launcher rebuild app:

ERROR: Docker version 1.13.1 not supported, please upgrade to at least 17.03.1, or recommended 17.06.2

I had successfully run sudo ./launcher rebuild app immediately prior to pulling the latest Discourse image using git. I have no idea why after pulling the latest it would start failing. (EDIT: I’m a newb still learning about server administration and had to apt-get upgrade; carry on.)

I’m now totally lost. I’m hoping someone can help me:

1. Fix the launcher/Docker issue so that I can at least do normal maintenance on my server.
2. Fix the Onebox issue so I can have good looking links again on my posts. Perhaps this is a known issue that I’m not aware of?

I did my best to search the site here to see if anyone else was having these issues, so I apologize if this is a duplicate question. Any help would be greatly appreciated.

2

OK, issue #1 is just my inexperience with Docker and server administration catching up with me. A bit more poking around indicates that I probably hadn’t done an apt-get upgrade in a while. That got me past the Docker version error and the rebuild is now underway. I can report back on the Onebox issue if/when that rebuild succeeds. Apologies if I wasted anyone’s time.

3

Issue #2 remains unresolved. My links will not create Oneboxes when I paste links into titles or bodies of posts. I don’t know what to do about that at this point. Again, using curl to access the page from within the Docker instance works just fine.

4

I should also mention that when I check sidekiq, it does not appear to be busy. It is not working on anything and no jobs are enqueued.

5

Any errors in your logs? See Where are all the Discourse logs. The logs web UI may have relevant information. So may your browser console.

1 Like
6

I didn’t see anything suspicious in the main logs (at mydomain.com/logs). I can dig deeper.

1 Like
7

Is this a completely standard install via https://github.com/discourse/discourse/blob/master/docs/INSTALL-cloud.md? Any NGINX, HAProxy, CloudFlare, or other proxy in front?

8

Yeah, those are the instructions I followed. The only real thing I did was to setup SSH support. I don’t believe I’m using any form of proxy; I did not set up a CDN. I did set up an Amazon S3 instance to handle uploads; no problems there.

9

OK. I found a log item in production.log that probably explains it, but I’m not sure what can be done about it:

Started GET "/onebox?url=http%3A%2F%2Fwww.espn.com%2Fmens-college-basketball%2Fstory%2F_%2Fid%2F21251356%2Fduke-blue-devils-ranked-no-1-ap-preseason-men-basketball-poll-2nd-straight-year&refresh=false&user_id=3" for 192.241.57.20 at 2017-11-01 16:54:57 +0000
Processing by OneboxController#show as HTML
  Parameters: {"url"=>"http://www.espn.com/mens-college-basketball/story/_/id/21251356/duke-blue-devils-ranked-no-1-ap-preseason-men-basketball-poll-2nd-straight-year", "refresh"=>"false", "user_id"=>"3"}
Failed to onebox http://www.espn.com/mens-college-basketball/story/_/id/21251356/duke-blue-devils-ranked-no-1-ap-preseason-men-basketball-poll-2nd-straight-year undefined method `strip' for nil:NilClass ["/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/engine/whitelisted_generic_onebox.rb:220:in `data'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/engine/whitelisted_generic_onebox.rb:247:in `is_article?'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/engine/whitelisted_generic_onebox.rb:239:in `generic_html'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/engine/whitelisted_generic_onebox.rb:186:in `to_html'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/preview.rb:35:in `engine_html'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/onebox-1.8.20/lib/onebox/preview.rb:16:in `to_s'", "/var/www/discourse/lib/oneboxer.rb:166:in `block in onebox_raw'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:297:in `block in fetch'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:586:in `block in save_block_result_to_cache'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:555:in `block in instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:168:in `instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:555:in `instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:585:in `save_block_result_to_cache'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/cache.rb:297:in `fetch'", "/var/www/discourse/lib/oneboxer.rb:153:in `onebox_raw'", "/var/www/discourse/lib/oneboxer.rb:31:in `preview'", "/var/www/discourse/app/controllers/onebox_controller.rb:20:in `show'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:186:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/rendering.rb:30:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:20:in `block in process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:131:in `run_callbacks'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:19:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/rescue.rb:20:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `block in instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `instrument'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:30:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/railties/controller_runtime.rb:22:in `process_action'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:124:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionview-5.1.4/lib/action_view/rendering.rb:30:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-mini-profiler-0.10.5/lib/mini_profiler/profiling_methods.rb:76:in `block in profile_method'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal.rb:189:in `dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_controller/metal.rb:253:in `dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:31:in `serve'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:50:in `block in serve'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `each'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `serve'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:834:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-protection-2.0.0/lib/rack/protection/frame_options.rb:31:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/builder.rb:63:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/conditional_get.rb:25:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'", "/var/www/discourse/lib/middleware/anonymous_cache.rb:149:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/cookies.rb:613:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:97:in `run_callbacks'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:24:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/logster-1.2.8/lib/logster/middleware/reporter.rb:31:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/rack/logger.rb:36:in `call_app'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/rack/logger.rb:26:in `call'", "/var/www/discourse/config/initializers/100-quiet_logger.rb:16:in `call'", "/var/www/discourse/config/initializers/100-silence_logger.rb:29:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/request_id.rb:25:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.4/lib/action_dispatch/middleware/executor.rb:12:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-mini-profiler-0.10.5/lib/mini_profiler/profiler.rb:171:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.0.8/lib/message_bus/rack/middleware.rb:63:in `call'", "/var/www/discourse/lib/middleware/request_tracker.rb:110:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/engine.rb:522:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/urlmap.rb:68:in `block in call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `each'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_server.rb:606:in `process_client'", "/var/www/discourse/lib/scheduler/defer.rb:87:in `process_client'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_server.rb:702:in `worker_loop'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/lib/unicorn/http_server.rb:142:in `start'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.0/bin/unicorn:126:in `<top (required)>'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/bin/unicorn:23:in `load'", "/var/www/discourse/vendor/bundle/ruby/2.4.0/bin/unicorn:23:in `<main>'"]
  Rendering text template
  Rendered text template (0.0ms)
Completed 404 Not Found in 366ms (Views: 0.9ms | ActiveRecord: 1.7ms)

Any thoughts? I can try spelunking around in that code myself, but my knowledge of Discourse’s source is still very limited.

10

Pretty sure I found the culprit:

https://github.com/discourse/onebox/commit/0e27263c95189750ab6926fc721fd7210db60175#diff-22ea1d10c5ef83557e8139ca487016b8

This commit added lines to parse some Twitter stuff that call split on some values returned from a dictionary whether those keys exist or not. When those keys don’t exist in the dictionary, split is invoked on nil. This commit is also currently failing automated tests. Fantastic.

Do I have something misconfigured such that I’m pulling down changes that haven’t passed automated testing? Because if there’s a way for me to set my install up to not get changes that haven’t passed automated testing, that would be great.

1 Like
12

The onebox gem tests are not integrated in core Discourse source code, so the tests-passed branch does not take that into consideration. It was an error on my part to not run specs before pushing the commit and I apologize for that.

Pushing a fix and cutting a new onebox version now.

4 Likes
13

And, as we would expect from the commit above, Twitter links do Onebox correctly, since they have the necessary metadata to avoid the undefined method exception. I have not found any other links that Onebox correctly, so I’m guessing that Twitter and only Twitter links will Onebox at the moment.

14

No, any site that supports twitter label meta tags will onebox, also all the sites that has dedicated onebox engine (for example: wikipedia, github, amazon, etc) will onebox correctly.

1 Like
15

Ironically, Oneboxing Discourse pages also works, so there’s a non-Twitter example of the Onebox code working.

16

Gotcha. Thanks for clarifying.

17

Pushed a fix. Updating to latest Discourse version will fix this issue.

Thanks for bringing this issue in our notice @john-lynch.

2 Likes
18

@techAPJ: Thanks so much. I have successfully upgraded to latest and the problem is solved.

4 Likes