I’m a little hesitant to ask this question, because previous topics I’ve read about modifying robots.txt seem to cause annoyance, but here we go… :crossed_fingers: We use a unified search in our knowledge base. Search results include official support documentation as well as Community content so th…

I think it is reasonable for us to allow people to add an arbitrary piece of text to robots.txt at the moment we are going to need a plugin, on the upside it is a very simple plugin I think a reasonable middle ground could be a hidden site setting for “appending something” to robots, that way this …

One easy work around is limiting those categories to trust_level_0, which accomplishes that with few side effects.

:thinking: Clever! But I want specific categories to be indexed by Google and other common search engines. I want to exclude them from only the search engine our knowledge base uses.

We plan to add a hidden site setting here so we can append stuff on demand to robots.txt for rare cases like this. It will require console access, but we can take care of that for you. We will let you know once it is deployed ping our team email with the specific text you need added :blush:

[image] sam: ping our team email with the specific text you need added Huh.:thinking: Topics aren’t grouped in directories by category, so this won’t be as easy as disallowing a subdirectory.

Ouch, yeah this is definitely getting into custom plugin area and a much more complex feature discussion, to keep all topics in “folders” internal routing needs to change We can not rush here at all but if you see this as something critical, let us know via a team email request and we can talk thro…

Robots.txt: Is it possible to exclude indexing of a specific category by a specific search engine?

Product Support

southpaw (Southpaw) October 12, 2018, 1:42am 10

Huh. Topics aren’t grouped in directories by category, so this won’t be as easy as disallowing a subdirectory.

Topic		Replies	Views
Is it possible to specify noindex for all topics in a specified category? Product Support	6	332	February 4, 2025
Need to find a solution to block indexing topics from a category Product Support	2	169	November 27, 2024
No-index for categories Product Support	0	390	April 7, 2021
Hiding 'Categories' seo/meta link? Product Support	3	808	September 28, 2019
How to stop search engines from indexing a particular Community topic? Product Support	5	670	December 8, 2023

Robots.txt: Is it possible to exclude indexing of a specific category by a specific search engine?

Related topics