Excluding user profiles in robots.txt (or allow edit of file)

codinghorror (Jeff Atwood) 24 mei 2014 om 04:19 5

User profiles are now disallowed from indexing by default.

2 likes

Generic rules in "robots.txt" not picked up by Googlebot

Topic		Antwoorden	Weergaven
How to edit robot.txt for indexing user profiles? Support	9	886	8 juni 2023
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3248	30 juli 2019
Why there are lots of Disallow rule in robots.txt? Support	34	4526	22 december 2020
Generic rules in "robots.txt" not picked up by Googlebot Support	6	990	2 april 2022
Google indexing issue (robots.txt) Support	9	678	23 mei 2024