Other than manually, is it possible to set up robots.txt to block all WP-Discouse-connected forum threads? (so that the forum post isn’t indexed) A simple toggle option that can block all connected forum threads from crawling? Or even a per-post toggle setting when publishing or editing a WP post that is linked to wp-discourse forum thread?

[image] Hayden James: Other than manually, is it possible to set up robots.txt to block all WP-Discouse-connected forum threads? (so that the forum post isn’t indexed) Why do you want to do that? I’m fairly certain that the forum post includes a pointer to the canonical version on your WP site.

A drop in ranking/reputation of WP’s articles (originals) since addition. I was not looking to start a debate or complain, just asking how to achieve this. I’m sure there are many other reasons others may want to have this level of control for what’s indexed.

[image] Hayden James: I was not looking to start a debate or complain, just asking how to achieve this Just suggesting - perhaps you should re-define your goals because you are heading to wrong direction. But with reverse proxy that is quite trivial task.

Not all web crawlers honor the robots.txt file.

Hey @haydenjames, There’s nothing you can do in the WP Discourse plugin to add posts it creates in Discourse to a robots.txt file. This is actually just a pure Discourse question, namely “Can I automatically noindex embedded topics?” (or something along those lines). A topic embedded from Wordpress is functionally the same as any other embedded topic. The avenue of investigation you want to pursue is there, for example the origin of the embed set canonical url site setting and related discussions. [image] Add option to set canonical_url to embed_url Feature From time to time we get requests to set the canonical URL of embedded topics to the URL of the blog post. I’ve created a pull request that does exactly that. It unconditionally uses the URL of the original blog post (embed_url) as canonical for the topic. There have been various previous discussions like Google indexed link not pointing to the correct post and Duplicate Content in the past. After reading those posts I’m not so sure about my solution anymore, so I’d like to get some feedback f… I don’t think (but happy to be corrected) that what you want to do is a current Discourse feature. Discourse currently adds a X-Robots-Tag: noindex header to GET requests for hidden topics. You could do the same for embedded topics via a plugin.

Heading in the wrong direction to block indexing of a forum thread with the duplicate article that I prefer Google search users to find via the WP blog? I’m ok with that. The Wp-discourse main benefit for me has been allowing discussions of blog posts without having to use solutions like Disqus or the very limited default WP comments. I don’t need any SEO benefit from the forums unless it’s other unique threads that are not connected to already existing content.

[image] Hayden James: with the duplicate article There isn’t any duplicates unless you have changed something. Because of: [image] Jay Pfaffman: to the canonical version on your WP site And: [image] Angus McLeod: A topic embedded from Wordpress is functionally the same as any other embedded topic. The avenue of investigation you want to pursue is there, for example the origin of the embed set canonical url site setting and related discussions.

[image] Angus McLeod: I don’t think (but happy to be corrected) that what you want to do is a current Discourse feature. Discourse currently adds a X-Robots-Tag: noindex header to GET requests for hidden topics. You could do the same for embedded topics via a plugin. Thanks @angus To clarify, if I make the category that stores the WP-discourse connected post hidden (is hidden different to private?) then it will hide the post from the forums/public/crawlers but the inserted comments at the end of each Wordpress blog post with comments will still be visible? Sorry about the noob questions, I’m not experienced with Discourse and want to make sure I’m not misinterpreting your response.

[image] Jakke Flemming: There isn’t any duplicates unless you have changed something. …depends on your definition of duplicate. Canonical is in place, but for me personally, since both the Blog post and the forum thread contain the exact paste/text (duplicate). I would like just to block those threads altogether. That’s just my preference. Maybe in the future, the reasoning behind this topic will make more sense. But for now, I am honestly not trying to provoke a debate or anything like that. I think think that blocking is a more absolute solution for me. It’s like going to your mechanic and asking him to “change your oil twice”. I understand the initial “why” by @angus - but in the end, it’s just about whether it can be done somehow, or not possible. Edit: Now thinking about it, I could then just add the blog post forum category to robots.txt, correct? Or will that be overwritten? (I will search the forums for how Discourse robots.txt works/can be edited. So something like: forum.domain.com/c/blog-articles/xx/*

A “hidden” topic is one that isn’t listed on topic lists, i.e. it isn’t “discoverable” in the normal fashion. You can tell a hidden topic by the eye with a line through it symbol. [Screenshot 2023-06-19 at 20.37.37] [image] Hayden James: To clarify, if I make the category that stores the WP-discourse connected post hidden (is hidden different to private?) then it will hide the post from the forums/public/crawlers but the inserted comments at the end of each Wordpress blog post with comments will still be visible? Actually there is a way to automatically make posts from the WP Discourse plugin “hidden” :slight_smile: You can use the “Publish as Unlisted Topics” setting. [Screenshot 2023-06-19 at 20.35.41] Keep in mind both what I said up top, and what it says next to that setting. This will mean that topics published from Discourse to Wordpress do not appear on the topic lists of your forum. Comments will work in the normal fashion. If you have the sync comment data webhook enabled the topic will no longer be hidden after the first comment. That feature wasn’t exactly designed for this purpose. See further [image] Hide topic until first reply WordPress I’m wondering if it’s possible to default the new topics from Wordpress to “hidden” or something, until somebody bothers to comment on them. We put up a lot of posts, and some of them will generate significant discussion, but others not so much. I’m worried about having Discourse cluttered with a lot of posts that have no activity. I see a similar question here which says it isn’t possible: However, I swear I saw something like this option somewhere in the past. Perhaps now I’m just missing t… If you want to just add a X-Robots-Tag: noindex header to an embedded topic (without bothering about this hidden business), you’ll need to either request that as a new feature of Discourse itself or add it via a plugin.

Block crawlers from embedded topics only?

Product Support

haydenjames (Hayden James) June 20, 2023, 10:09am 18

Even though the rel=canonical tag can help you avoid a duplicate content penalty when you republish posts, you can still get penalized if you misuse the tag. I’ll find a solution. Will bump this thread at a later date.

Topic		Replies	Views
Old topics published via WordPress embed are missing x-robots: noindex and canonical tags WordPress wordpress	10	722	August 15, 2026
Need a way to add "meta robots noindex" in topics from a category Feature seo	6	1426	February 12, 2025
Posts in WP Discourse interferes with the SEO of the Wordpress site WordPress	2	696	November 24, 2021
Google changed how they process robots.txt in Discourse? Product Support	20	2031	December 22, 2020
Discourse SEO overview (sitemap / robots.txt) Site Management seo , explanation	0	2670	October 18, 2023

Block crawlers from embedded topics only?

Related topics