Excluding user profiles in robots.txt (or allow edit of file)

codinghorror (Jeff Atwood) May 24, 2014, 4:19am 5

User profiles are now disallowed from indexing by default.

2 Likes

Generic rules in "robots.txt" not picked up by Googlebot

Topic		Replies	Views
How to edit robot.txt for indexing user profiles? Support	9	876	June 8, 2023
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3239	July 30, 2019
Why there are lots of Disallow rule in robots.txt? Support	34	4513	December 22, 2020
Generic rules in "robots.txt" not picked up by Googlebot Support	6	985	April 2, 2022
Google indexing issue (robots.txt) Support	9	668	May 23, 2024