Excluding user profiles in robots.txt (or allow edit of file)

codinghorror (Jeff Atwood) May 24, 2014, 4:19am 5

User profiles are now disallowed from indexing by default.

2 Likes

Generic rules in "robots.txt" not picked up by Googlebot

Topic		Replies	Views
Robots.txt to completely block all indexing of private site Support	5	825	April 5, 2020
How to edit robot.txt for indexing user profiles? Support	9	838	June 8, 2023
Pages listed in the robots.txt are crawled and indexed by Google Support	19	3193	July 30, 2019
Google changed how they process robots.txt in Discourse? Support	20	1576	December 22, 2020
How to noindex all pages to keep them out of Google indexes Support	8	2437	February 22, 2024