The discussion revolves around a bug where inline onebox link previews in Discourse fail due to user agent mismatches with Anubis, a bot protection system. osmith explains that Discourse uses two different user agents: Discourse Forum Onebox v3.4.1 for full oneboxes and a browser-like Mozilla agent for inline previews, which Anubis blocks. Despite attempts to fix this via Force custom user agent hosts and the onebox_user_agent setting, the issue persists because the inline onebox user agent isn’t configurable. lilydjwg confirms similar behavior, noting that HEAD requests with browser agents trigger challenges. sam suggests this is a feature request, not a bug, and proposes a new site setting. mattdm notes the issue affects other communities. kris.kotlarek announces that a hidden inline_onebox_user_agent setting has been merged in a PR, which should resolve the issue by allowing customization of the inline preview user agent. osmith confirms this should solve the problem, though testing is pending.
we have recently deployed Anubis on our server to protect against resource-intense AI crawlers. It sends a static page with title “Making sure you’re not a bot!” to requests sent with browser-like user agents, does a javascript proof-of-work and then redirects to the proper page. User agents that don’t have Mozilla in the name can directly access the page.
Now the discourse onebox link previews are broken, but only for inline links.
Looking at server logs, discourse sends two different types of user agents:
Discourse Forum Onebox v3.4.1 for big onebox link previews (see lib/onebox/helpers.rb#L217-L226 in 03f3e84ce520d375aa1c66bb0e95cc5ed9832776)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15 for inline link previews (seelib/final_destination.rb#L40 in 03f3e84ce520d375aa1c66bb0e95cc5ed9832776)
I’ve tried “Force custom user agent hosts” in the admin settings, which claims “Hosts for which to use the custom onebox user agent on all requests.”, but it didn’t make a difference.
I’ve also tested changing the onebox_user_agent hidden setting, but it only changes the regular onebox link preview user agent, not the inline one.
This looks like a bug to me, wouldn’t it make more sense to use the same configurable Discourse Forum Onebox user agent for inline link previews too?
I get two different onebox too, one HEAD request using a browser user-agent, and one GET request using an identifiable user-agent. If the HEAD request is redirected (to a challenge page), three requests follow, all to the redirected page.
I have to whitelist all HEAD requests in our challenge system for onebox to work. Not only inline-onebox, but block onebox as well.
This feels like a feature request vs bug. inline onebox and onebox are different features which use the network in different ways. Maybe we need another site setting here.
I am ok with adding another hidden setting for inline oneboxes at least it will unblock you.
I cannot test it right now. But if the hidden setting overrides the user-agent as described, then it solves the problem with inline links and Anubis. Thanks for implementing this!