I wonder what’s the best place to disable indexing (set noindex) on the entire site, and I mean across the board. Is it enough to simply insert NOINDEX in Discourse header (I wonder if that would apply to all pages/URLs generated by the site)? P.S. And for misbehaving crawlers such as some search …

How to disable indexing by crawlers

codinghorror (Jeff Atwood) February 9, 2016, 11:14am 3

There is a boolean site setting for this, search for “index” or “robots”.

7 Likes

MegaIndex bot did about 4,000 pageviews on one day

Topic		Replies	Views
Why there are lots of Disallow rule in robots.txt? Support	34	4456	December 22, 2020
Google complaining – Indexed, though blocked by robots.txt Support	24	2429	September 28, 2023
What can crawlers see? Data & reporting	16	659	February 5, 2024
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3225	July 30, 2019
Robots.txt to completely block all indexing of private site Support	5	830	April 5, 2020