Pages meant to be hidden from Google are in the robots.txt However, Google attempts to crawl them anyway. Since they are accessible through links on web pages, they get indexed. The crawler then hits the robots.txt and crawl “blank”/“blocked”. This results in a lot of error in our analytics conso…

Completed per: https://github.com/discourse/discourse/commit/d84256a876a9fa4fc7bcb4b8ac8c5865f8c10701

Pages listed in the robots.txt are crawled and indexed by Google

sam (Sam Saffron) July 30, 2019, 1:34am 19

Reverted per:

2 Likes

Topic		Replies	Views
Google changed how they process robots.txt in Discourse? support	20	1464	December 22, 2020
Generic rules in "robots.txt" not picked up by Googlebot support	6	833	April 2, 2022
Why there are lots of Disallow rule in robots.txt? support	34	4229	December 22, 2020
Needing to edit robots.txt file - where is it? support	42	6766	April 29, 2023
Google notification to remove "noindex" statements from robots.txt support	8	2259	July 30, 2019