Oneboxing geblokkeerd door robotcontrole

AI-generated summary

Users are reporting that Discourse’s oneboxing feature is being blocked by Bloomberg’s anti-scraping measures, which trigger a “Are you a robot?” verification page. breetai first noticed the issue, noting that oneboxing previously worked but now fails, sharing an example link that redirects to a robot-check page. pfaffman suggests this is likely rate-limiting by Bloomberg, with little recourse except staying under their limits. codinghorror questions the URL’s strangeness, prompting breetai to clarify it’s a Bloomberg opinion article. merefield recommends using a known working method from a Meta Discourse thread and notes the shared URL is a redirect, not the original article. breetai provides the correct article link. codinghorror confirms the issue stems from Bloomberg’s aggressive anti-scraping, which blocks even simple metadata requests, and suggests oneboxing should be disabled for such links since they lack images or descriptions — a change he plans to backport to stable. JimPas verifies the issue occurs even outside Discourse, confirming Bloomberg’s robot check is triggered by direct browser access to the tosv2.html endpoint, displaying a message asking users to verify they’re not robots, along with a reference ID for support.

I am seeing this on a sites and it just started. When Discourse tries to pull the information from the site it is blocked. THis use to work in prior versions.
I have included a link as an example

Bloomberg - Are you a robot?

It looks like rate limiting by Bloomberg. There likely isn’t much you can do other than infer what the limits are and contrive to stay under them.

What exactly are you trying to onebox here? The URL is quite strange.

Bloomberg news article. If you click on the link. Thahs the article.

Try "Onebox Assistant", crawl for those previews reliably!

Works with Bloomberg links iirc.

What’s the original link? The one you’ve pasted above is not an article, it’s a destination you’ve been redirected to I suspect.

https://www.bloomberg.com/opinion/articles/2020-01-29/peer-review-is-science-s-wheel-of-misfortune

This is the link.

I see, here’s the link

http://www.bloomberg.com/opinion/articles/2020-01-29/peer-review-is-science-s-wheel-of-misfortune

http://www.bloomberg.com/opinion/articles/2020-01-29/peer-review-is-science-s-wheel-of-misfortune

Looks like they have pretty aggressive anti-scraping in place, since all we’re doing is checking for metadata headers..

Also yet another example of where we shouldn’t be oneboxing at all because we have neither image nor description cc @techAPJ @sam .. we really gotta backport that change to stable once it goes in next week.

I just tried the link ending up to the html extension (minus all the trailing characters) just using Firefox, not Discourse Onebox. The error extended error message is below the line. The 1st link (which has the error msg below) is enclosed in <> here. The second link is without being enclosed in <> and gives the title of the URL as shown.
https://www.bloomberg.com/tosv2.html
Bloomberg - Are you a robot?


We’ve detected unusual activity from your computer network

To continue, please click the box below to let us know you’re not a robot.

Why did this happen?

Please make sure your browser supports JavaScript and cookies and that you are not blocking them from loading. For more information you can review our Terms of Service and Cookie Policy.

Need Help?

For inquiries related to this message please contact our support team and provide the reference ID below.

Block reference ID: 13215fd0-4285-11eb-8faf-b7e9262e99b2