I wonder what’s the best place to disable indexing (set noindex) on the entire site, and I mean across the board. Is it enough to simply insert NOINDEX in Discourse header (I wonder if that would apply to all pages/URLs generated by the site)? P.S. And for misbehaving crawlers such as some search …

How to disable indexing by crawlers

codinghorror (Jeff Atwood) February 9, 2016, 11:14am 3

There is a boolean site setting for this, search for “index” or “robots”.

7 Likes

MegaIndex bot did about 4,000 pageviews on one day

Topic		Replies	Views
Why there are lots of Disallow rule in robots.txt? Support	34	4430	December 22, 2020
Google complaining – Indexed, though blocked by robots.txt Support	24	2410	September 28, 2023
What can crawlers see? Data & reporting	16	634	February 5, 2024
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3205	July 30, 2019
Robots.txt to completely block all indexing of private site Support	5	827	April 5, 2020