Block crawlers from embedded topics only?

[angus (Angus McLeod)]

Hey @haydenjames,

There’s nothing you can do in the WP Discourse plugin to add posts it creates in Discourse to a robots.txt file. This is actually just a pure Discourse question, namely “Can I automatically noindex embedded topics?” (or something along those lines). A topic embedded from Wordpress is functionally the same as any other embedded topic. The avenue of investigation you want to pursue is there, for example the origin of the embed set canonical url site setting and related discussions.

Add option to set canonical_url to embed_url

From time to time we get requests to set the canonical URL of embedded topics to the URL of the blog post. I’ve created a pull request that does exactly that. It unconditionally uses the URL of the original blog post (embed_url) as canonical for the topic. There have been various previous discussions like Google indexed link not pointing to the correct post and Duplicate Content in the past. After reading those posts I’m not so sure about my solution anymore, so I’d like to get some feedback f…

I don’t think (but happy to be corrected) that what you want to do is a current Discourse feature. Discourse currently adds a X-Robots-Tag: noindex header to GET requests for hidden topics. You could do the same for embedded topics via a plugin.