Excluding user profiles in robots.txt (or allow edit of file)

codinghorror (Jeff Atwood) May 24, 2014, 4:19am 5

User profiles are now disallowed from indexing by default.

2 Likes

Generic rules in "robots.txt" not picked up by Googlebot

Topic		Replies	Views
Robots.txt to completely block all indexing of private site Support	5	827	April 5, 2020
How to edit robot.txt for indexing user profiles? Support	9	841	June 8, 2023
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3201	July 30, 2019
Google changed how they process robots.txt in Discourse? Support	20	1577	December 22, 2020
How to noindex all pages to keep them out of Google indexes Support	8	2461	February 22, 2024