Google changed how they process robots.txt in Discourse?

codinghorror (Jeff Atwood) December 22, 2020, 6:02am 21

We should probably backport this.

2 Likes

Topic		Replies	Views
Pages listed in the robots.txt are crawled and indexed by Google support	19	2874	July 30, 2019
Google complaining – Indexed, though blocked by robots.txt support	24	2260	September 28, 2023
Generic rules in "robots.txt" not picked up by Googlebot support	6	854	April 2, 2022
Google notification to remove "noindex" statements from robots.txt support	8	2288	July 30, 2019
Why there are lots of Disallow rule in robots.txt? support	34	4270	December 22, 2020