Excluding user profiles in robots.txt (or allow edit of file)

codinghorror (Jeff Atwood) May 24, 2014, 4:19am 5

User profiles are now disallowed from indexing by default.

2 Likes

Generic rules in "robots.txt" not picked up by Googlebot

Topic		Replies	Views
How to edit robot.txt for indexing user profiles? Support	9	883	June 8, 2023
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3245	July 30, 2019
Why there are lots of Disallow rule in robots.txt? Support	34	4521	December 22, 2020
Generic rules in "robots.txt" not picked up by Googlebot Support	6	989	April 2, 2022
Google indexing issue (robots.txt) Support	9	672	May 23, 2024