I wonder what’s the best place to disable indexing (set noindex) on the entire site, and I mean across the board. Is it enough to simply insert NOINDEX in Discourse header (I wonder if that would apply to all pages/URLs generated by the site)? P.S. And for misbehaving crawlers such as some search …

How to disable indexing by crawlers

codinghorror (Jeff Atwood) 9 בפברואר,‏ 2016,‏ 11:14am 3

There is a boolean site setting for this, search for “index” or “robots”.

7 לייקים

MegaIndex bot did about 4,000 pageviews on one day

נושא		תגובות	צפיות
Robots.txt to completely block all indexing of private site Support	5	852	5 באפריל,‏ 2020
Why there are lots of Disallow rule in robots.txt? Support	34	4558	22 בדצמבר,‏ 2020
Temporarily block Google crawler while setting up live site? Support	4	466	5 בפברואר,‏ 2020
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3277	30 ביולי,‏ 2019
Option to index topics Support	8	553	31 בדצמבר,‏ 2022