仅阻止来自嵌入式主题的爬虫吗？

[angus (Angus McLeod)]

嘿 @haydenjames，

在 WP Discourse 插件中，您无法在 Discourse 中创建的帖子添加到 robots.txt 文件中。这实际上是一个纯粹的 Discourse 问题，即“我可以自动不对嵌入式主题使用 noindex 吗？”（或类似的话）。从 WordPress 嵌入的主题在功能上与其他嵌入式主题相同。您想探索的途径是存在的，例如 embed set canonical url 站点设置的来源以及相关的讨论。

Add option to set canonical_url to embed_url

From time to time we get requests to set the canonical URL of embedded topics to the URL of the blog post. I’ve created a pull request that does exactly that. It unconditionally uses the URL of the original blog post (embed_url) as canonical for the topic. There have been various previous discussions like Google indexed link not pointing to the correct post and Duplicate Content in the past. After reading those posts I’m not so sure about my solution anymore, so I’d like to get some feedback f…

我认为（但欢迎纠正）您想做的事情目前不是 Discourse 的功能。Discourse 目前会向隐藏主题的 GET 请求添加 X-Robots-Tag: noindex 标头。您可以通过插件为嵌入式主题执行相同的操作。