Excluding user profiles in robots.txt (or allow edit of file)

codinghorror (Jeff Atwood) Maio 24, 2014, 4:19am 5

User profiles are now disallowed from indexing by default.

2 curtidas

Generic rules in "robots.txt" not picked up by Googlebot

Tópico		Respostas	Visualizações
How to edit robot.txt for indexing user profiles? Support	9	901	8 de Junho de 2023
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3270	30 de Julho de 2019
Indexing User Profiles Support	2	56	4 de Setembro de 2025
Why there are lots of Disallow rule in robots.txt? Support	34	4546	22 de Dezembro de 2020
Generic rules in "robots.txt" not picked up by Googlebot Support	6	997	2 de Abril de 2022