Discourse Meta
How to edit robot.txt for indexing user profiles?
Ondersteuning
codinghorror
(Jeff Atwood)
6 september 2019 om 16:10
7
We do not advise messing with this at all.
bericht weergeven in topic
Gerelateerde topics
Topic
Antwoorden
Weergaven
Activiteit
Excluding user profiles in robots.txt (or allow edit of file)
Feature
4
2637
24 mei 2014
Indexing User Profiles
Product Support
2
202
4 september 2025
Pages listed in the robots.txt are crawled and indexed by Google
Product Support
18
3646
30 juli 2019
Generic rules in "robots.txt" not picked up by Googlebot
Product Support
5
1131
3 maart 2022
Why there are lots of Disallow rule in robots.txt?
Product Support
34
5035
22 december 2020