I’m a little hesitant to ask this question, because previous topics I’ve read about modifying robots.txt seem to cause annoyance, but here we go…
We use a unified search in our knowledge base. Search results include official support documentation as well as Community content so that users can find answers in either place.
I’d like to limit the search results from Community to specific categories that we carefully curate, not for PR reasons, but to be sure quality information is available in the knowledge base search results. As it is now, the search results are too cluttered to be useful.
Is it possible for me to modify the robots.txt file to limit our unified search’s search engine to indexing only the curated categories? The only settings I’ve been able to find in site_settings appear to allow me to blacklist specific search engines, but not to refine that blacklist so only specific categories are blacklisted for a specific search engine.