Google changed how they process robots.txt in Discourse?

codinghorror (Jeff Atwood) 22 בדצמבר,‏ 2020,‏ 6:02am 21

We should probably backport this.

2 לייקים

נושא		תגובות	צפיות
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3289	30 ביולי,‏ 2019
Google notification to remove "noindex" statements from robots.txt Support	8	2442	30 ביולי,‏ 2019
Google complaining – Indexed, though blocked by robots.txt Support	24	2501	28 בספטמבר,‏ 2023
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Support	18	195	18 בדצמבר,‏ 2024
Generic rules in "robots.txt" not picked up by Googlebot Support	6	1001	2 באפריל,‏ 2022