Discourse Meta
How to disable indexing by crawlers
Product Support
codinghorror
(Jeff Atwood)
9 בפברואר, 2016, 11:14am
3
There is a boolean site setting for this, search for “index” or “robots”.
MegaIndex bot did about 4,000 pageviews on one day
הצגת פוסט בנושא
נושאים קשורים
נושא
תגובות
צפיות
פעילות
Robots.txt to completely block all indexing of private site
Product Support
4
1026
6 במרץ, 2020
Why there are lots of Disallow rule in robots.txt?
Product Support
34
5035
22 בדצמבר, 2020
Temporarily block Google crawler while setting up live site?
Product Support
3
563
6 בינואר, 2020
Pages listed in the robots.txt are crawled and indexed by Google
Product Support
18
3649
30 ביולי, 2019
Option to index topics
Product Support
7
753
31 בדצמבר, 2022