Excluding user profiles in robots.txt (or allow edit of file)

codinghorror (Jeff Atwood) May 24, 2014, 4:19am 5

User profiles are now disallowed from indexing by default.

2 Likes

Generic rules in "robots.txt" not picked up by Googlebot

Topic		Replies	Views
Robots.txt to completely block all indexing of private site support	5	752	April 5, 2020
How to edit robot.txt for indexing user profiles? support	9	748	June 8, 2023
Pages listed in the robots.txt are crawled and indexed by Google support	19	2778	July 30, 2019
Google changed how they process robots.txt in Discourse? support	20	1461	December 22, 2020
How to noindex all pages to keep them out of Google indexes support	8	1841	February 22, 2024