Google changed how they process robots.txt in Discourse?

codinghorror (Jeff Atwood) December 22, 2020, 6:02am 21

We should probably backport this.

2 Likes

Topic		Replies	Views
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3201	July 30, 2019
Google complaining – Indexed, though blocked by robots.txt Support	24	2407	September 28, 2023
Generic rules in "robots.txt" not picked up by Googlebot Support	6	965	April 2, 2022
Google notification to remove "noindex" statements from robots.txt Support	8	2400	July 30, 2019
Why there are lots of Disallow rule in robots.txt? Support	34	4423	December 22, 2020