The discussion revolves around a bug where inline onebox link previews in Discourse fail due to user agent mismatches with Anubis, a bot protection system. osmith explains that Discourse uses two different user agents: Discourse Forum Onebox v3.4.1 for full oneboxes and a browser-like Mozilla agent for inline previews, which Anubis blocks. Despite attempts to fix this via Force custom user agent hosts and the onebox_user_agent setting, the issue persists because the inline onebox user agent isn’t configurable. lilydjwg confirms similar behavior, noting that HEAD requests with browser agents trigger challenges. sam suggests this is a feature request, not a bug, and proposes a new site setting. mattdm notes the issue affects other communities. kris.kotlarek announces that a hidden inline_onebox_user_agent setting has been merged in a PR, which should resolve the issue by allowing customization of the inline preview user agent. osmith confirms this should solve the problem, though testing is pending.
we have recently deployed Anubis on our server to protect against resource-intense AI crawlers. It sends a static page with title “Making sure you’re not a bot!” to requests sent with browser-like user agents, does a javascript proof-of-work and then redirects to the proper page. User agents that don’t have Mozilla in the name can directly access the page.
Now the discourse onebox link previews are broken, but only for inline links.
Looking at server logs, discourse sends two different types of user agents:
Discourse Forum Onebox v3.4.1 for big onebox link previews (see lib/onebox/helpers.rb#L217-L226 in 03f3e84ce520d375aa1c66bb0e95cc5ed9832776)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15 for inline link previews (seelib/final_destination.rb#L40 in 03f3e84ce520d375aa1c66bb0e95cc5ed9832776)
I’ve tried “Force custom user agent hosts” in the admin settings, which claims “Hosts for which to use the custom onebox user agent on all requests.”, but it didn’t make a difference.
I’ve also tested changing the onebox_user_agent hidden setting, but it only changes the regular onebox link preview user agent, not the inline one.
This looks like a bug to me, wouldn’t it make more sense to use the same configurable Discourse Forum Onebox user agent for inline link previews too?
אני מקבל שני onebox שונים גם כן, בקשת HEAD אחת המשתמשת בסוכן משתמש של דפדפן, ובקשת GET אחת המשתמשת בסוכן משתמש שניתן לזיהוי. אם בקשת ה-HEAD מופנית מחדש (לדף אתגר), שלוש בקשות עוקבות, כולן לדף המופנה מחדש.
אני צריך להכניס לרשימת ההיתרים את כל בקשות ה-HEAD במערכת האתגרים שלנו כדי ש-onebox יעבוד. לא רק inline-onebox, אלא גם block onebox.
I cannot test it right now. But if the hidden setting overrides the user-agent as described, then it solves the problem with inline links and Anubis. Thanks for implementing this!