Google changed how they process robots.txt in Discourse?

codinghorror (Jeff Atwood) Dezembro 22, 2020, 6:02am 21

We should probably backport this.

2 curtidas

Tópico		Respostas	Visualizações
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3286	30 de Julho de 2019
Google notification to remove "noindex" statements from robots.txt Support	8	2442	30 de Julho de 2019
Google complaining – Indexed, though blocked by robots.txt Support	24	2500	28 de Setembro de 2023
Issues Google Search Console is throwing at me for wrong discourse structure (or some for wrong administration of my site) Support	18	192	18 de Dezembro de 2024
Generic rules in "robots.txt" not picked up by Googlebot Support	6	1000	2 de Abril de 2022