Robots.txt: Is it possible to exclude indexing of a specific category by a specific search engine?

Support
1

I’m a little hesitant to ask this question, because previous topics I’ve read about modifying robots.txt seem to cause annoyance, but here we go… :crossed_fingers:

We use a unified search in our knowledge base. Search results include official support documentation as well as Community content so that users can find answers in either place.

I’d like to limit the search results from Community to specific categories that we carefully curate, not for PR reasons, but to be sure quality information is available in the knowledge base search results. As it is now, the search results are too cluttered to be useful.

Is it possible for me to modify the robots.txt file to limit our unified search’s search engine to indexing only the curated categories? The only settings I’ve been able to find in site_settings appear to allow me to blacklist specific search engines, but not to refine that blacklist so only specific categories are blacklisted for a specific search engine.

3 Likes
2

I think it is reasonable for us to allow people to add an arbitrary piece of text to robots.txt at the moment we are going to need a plugin, on the upside it is a very simple plugin

I think a reasonable middle ground could be a hidden site setting for “appending something” to robots, that way this can be done without a plugin

5 Likes
5

One easy work around is limiting those categories to trust_level_0, which accomplishes that with few side effects.

3 Likes
6

:thinking: Clever! But I want specific categories to be indexed by Google and other common search engines. I want to exclude them from only the search engine our knowledge base uses.

7

We plan to add a hidden site setting here so we can append stuff on demand to robots.txt for rare cases like this. It will require console access, but we can take care of that for you.

We will let you know once it is deployed ping our team email with the specific text you need added :blush:

5 Likes
10

Huh.:thinking: Topics aren’t grouped in directories by category, so this won’t be as easy as disallowing a subdirectory.

2 Likes
12

Ouch, yeah this is definitely getting into custom plugin area and a much more complex feature discussion, to keep all topics in “folders” internal routing needs to change

We can not rush here at all but if you see this as something critical, let us know via a team email request and we can talk through options

4 Likes