# Robots.txt: Is it possible to exclude indexing of a specific category by a specific search engine?
**URL:** https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326
**Category:** Support
**Created:** [11 oktober 2018 om 21:47 UTC](https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326 "2018-10-11T21:47:58Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@southpaw](https://meta.discourse.org/u/southpaw)
#### Post date: [11 oktober 2018 om 21:47 UTC](https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326/1 "2018-10-11T21:47:58Z")
I’m a _little_ hesitant to ask this question, because previous topics I’ve read about modifying robots.txt seem to cause annoyance, but here we go… 🤞
We use a unified search in our knowledge base. Search results include official support documentation as well as Community content so that users can find answers in either place.
I’d like to limit the search results from Community to specific categories that we carefully curate, not for PR reasons, but to be sure quality information is available in the knowledge base search results. As it is now, the search results are too cluttered to be useful.
Is it possible for me to modify the robots.txt file to limit our unified search’s search engine to indexing only the curated categories? The only settings I’ve been able to find in site\_settings appear to allow me to blacklist specific search engines, but not to refine that blacklist so only specific categories are blacklisted for a specific search engine.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [11 oktober 2018 om 21:58 UTC](https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326/2 "2018-10-11T21:58:00Z")
I think it is reasonable for us to allow people to add an arbitrary piece of text to robots.txt at the moment we are going to need a plugin, on the upside it is a very simple plugin
I think a reasonable middle ground could be a hidden site setting for “appending something” to robots, that way this can be done without a plugin
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [11 oktober 2018 om 23:29 UTC](https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326/5 "2018-10-11T23:29:07Z")
One easy work around is limiting those categories to trust\_level\_0, which accomplishes that with few side effects.
---
### Author:  [@southpaw](https://meta.discourse.org/u/southpaw)
#### Post date: [12 oktober 2018 om 00:06 UTC](https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326/6 "2018-10-12T00:06:52Z")
🤔 Clever! But I want specific categories to be indexed by Google and other common search engines. I want to exclude them from only the search engine our knowledge base uses.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [12 oktober 2018 om 00:09 UTC](https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326/7 "2018-10-12T00:09:24Z")
We plan to add a hidden site setting here so we can append stuff on demand to `robots.txt` for rare cases like this. It will require console access, but we can take care of that for you.
We will let you know once it is deployed ping our team email with the specific text you need added 😊
---
### Author:  [@southpaw](https://meta.discourse.org/u/southpaw)
#### Post date: [12 oktober 2018 om 01:42 UTC](https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326/10 "2018-10-12T01:42:41Z")
> [@sam](#):
>
> ping our team email with the specific text you need added
Huh.🤔 Topics aren’t grouped in directories by category, so this won’t be as easy as disallowing a subdirectory.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [12 oktober 2018 om 03:23 UTC](https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326/12 "2018-10-12T03:23:40Z")
Ouch, yeah this is definitely getting into custom plugin area and a much more complex feature discussion, to keep all topics in “folders” internal routing needs to change
We can not rush here at all but if you see this as something critical, let us know via a team email request and we can talk through options
---
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [2 juli 2021 om 12:57 UTC](https://meta.discourse.org/t/robots-txt-is-it-possible-to-exclude-indexing-of-a-specific-category-by-a-specific-search-engine/99326/13 "2021-07-02T12:57:31Z")