it is a fact that search engines, bots can index topics in a category on discourse. How to block Google, bing, yandex, … from indexing topics of a category?
We look forward to Discourse experts’ support.
AI-generated summary
Users are seeking a way to prevent search engines like Google, Bing, and Yandex from indexing topics within a specific Discourse category. hoangviet raises the issue, and Don suggests setting the category’s permission level to TL0, making it visible only to logged-in users — effectively blocking bots. This can be configured via the category’s Security tab, restricting anonymous access. Jagster agrees this is the most reliable method but notes it’s unsuitable if the forum must remain open to public visitors. Alternatives like robots.txt or meta tags are unreliable, as many crawlers — including SEO bots and rogue scripts — ignore them. Even Google may index pages if links are shared via Gmail. The best solution depends on the desired level of privacy: full access restriction (TL0) for maximum protection, or imperfect technical workarounds for public forums.
You can change the category permission to TL0 so only logged in users will see it and also will block bots.
If that Don suggested works use it. It is the only 100% sure way. But if that forum must be open for random visitors you can’t use TL0. Sure you can use User-agent: * Disallow: / or similar meta, but sometimes those are respected and sometimes not. If somebody shares a link using gmail Google will index no matter you have in robots.txt or meta.
And those tricks are effective only if a bot obeys those. Most of SEO crawlers don’t follow. And then there is a zillion rogue bots and script kiddies trying to do something.
So, what you have to do depends on what kind level of privacy you are looking for.