Google changed how they process robots.txt in Discourse?

codinghorror (Jeff Atwood) December 22, 2020, 6:02am 21

We should probably backport this.

2 Likes

Topic		Replies	Views
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3223	July 30, 2019
Google complaining – Indexed, though blocked by robots.txt Support	24	2421	September 28, 2023
Generic rules in "robots.txt" not picked up by Googlebot Support	6	973	April 2, 2022
Google notification to remove "noindex" statements from robots.txt Support	8	2414	July 30, 2019
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Support	18	96	December 18, 2024