Discourse Meta
How to disable indexing by crawlers
Support
codinghorror
(Jeff Atwood)
February 9, 2016, 11:14am
3
There is a boolean site setting for this, search for “index” or “robots”.
7 Likes
MegaIndex bot did about 4,000 pageviews on one day
show post in topic
Related topics
Topic
Replies
Views
Activity
Robots.txt to completely block all indexing of private site
Support
5
832
April 5, 2020
Why there are lots of Disallow rule in robots.txt?
Support
34
4483
December 22, 2020
Temporarily block Google crawler while setting up live site?
Support
4
459
February 5, 2020
Pages listed in the robots.txt are crawled and indexed by Google
Support
19
3230
July 30, 2019
Option to index topics
Support
8
521
December 31, 2022