OneBox fails for Bookshop.org - crawl blocked?

Product Support

AI-generated summary

In a Support discussion titled OneBox fails for Bookshop.org - crawl blocked?, ToddZ reports that OneBox previews for Bookshop.org book pages fail despite valid OpenGraph tags being present. Multiple validators either can’t reach the URL or return incomplete data, while tools like iFramely and Facebook’s debugger succeed. ToddZ suspects Bookshop.org is blocking certain crawls and asks what to request from them — perhaps allowing the Discourse user agent. Falco confirms the issue is Cloudflare blocking OneBox crawls, suggesting ToddZ ask Bookshop.org to relax bot protection rules to allow Discourse OneBox. ToddZ thanks Falco and says they’ll reach out to Bookshop.org. The issue appears to stem from server-side blocking rather than missing metadata, and no plugin workaround is preferred if a policy change can resolve it.

1

What should I ask of a site that disallows OneBox crawls?

Attempts to OneBox for Bookshop.org book listings result in: “Sorry, we were unable to generate a preview for this web page, because the following oEmbed / OpenGraph tags could not be found: description, image” – despite the required OG tags being present on the destination page.

(At time of posting, the result here on meta is the same as my site):

https://bookshop.org/p/books/parable-of-the-sower-octavia-e-butler/19767724?ean=9781538732182

OG details

The og tags do exist on the page:

image
image619×190 5.4 KB

I’ve tried several OpenGraph validators:

  • most report the URL unreachable.
  • a couple of them retrieve a few OG fields from Bookshop.org, but nothing specific to the requested page.
  • the iFramely debugger and the Facebook Sharing Debugger manage a full preview.

I’m gathering that Bookshop.org is rejecting some crawls. I’d rather not introduce a plugin workaround if there’s something appropriate I might ask Bookshop.org to do, e.g. “Please allow the Discourse user agent to crawl?”

2

Cloudflare is blocking onebox from reaching the site.

Please relax you bot protection rule to allow Discourse Onebox ?

3

Thanks for confirming! I’ll ask them.