How to edit robot.txt for indexing user profiles?

User profiles are disallowed as the version which is seen by the crawler contains no useful information.

It’s not a question of whether the server can handle it or “take the hit”, it’s the tens of thousands of effectively empty pages which will waste the crawl time allocated to your site by the search engines.

2 Likes