Discourse Meta
Excluding user profiles in robots.txt (or allow edit of file)
Feature
codinghorror
(Jeff Atwood)
May 24, 2014, 4:19am
5
User profiles are now disallowed from indexing by default.
2 Likes
Generic rules in "robots.txt" not picked up by Googlebot
show post in topic
Related topics
Topic
Replies
Views
Activity
How to edit robot.txt for indexing user profiles?
Support
9
876
June 8, 2023
Pages listed in the robots.txt are crawled and indexed by Google
Support
19
3239
July 30, 2019
Why there are lots of Disallow rule in robots.txt?
Support
34
4513
December 22, 2020
Generic rules in "robots.txt" not picked up by Googlebot
Support
6
985
April 2, 2022
Google indexing issue (robots.txt)
Support
9
668
May 23, 2024