Discourse Meta
How to disable indexing by crawlers
Support
codinghorror
(Jeff Atwood)
February 9, 2016, 11:14am
3
There is a boolean site setting for this, search for “index” or “robots”.
7 Likes
MegaIndex bot did about 4,000 pageviews on one day
show post in topic
Related topics
Topic
Replies
Views
Activity
Why there are lots of Disallow rule in robots.txt?
Support
34
4423
December 22, 2020
Google complaining – Indexed, though blocked by robots.txt
Support
24
2407
September 28, 2023
What can crawlers see?
Data & reporting
16
629
February 5, 2024
Pages listed in the robots.txt are crawled and indexed by Google
Support
19
3201
July 30, 2019
Robots.txt to completely block all indexing of private site
Support
5
827
April 5, 2020