I want the user profiles on my website to get indexed in Google. I know it might attract a lot of spam but we can handle that, although getting user profiles indexed in Google is more important for us. So is there a way to do this?
I checked the old topics on meta and they suggested to edit robot.txt. Is the answer still the same? And if the yes, then how to edit the robot.txt?
User profiles are disallowed as the version which is seen by the crawler contains no useful information.
It’s not a question of whether the server can handle it or “take the hit”, it’s the tens of thousands of effectively empty pages which will waste the crawl time allocated to your site by the search engines.
2 个赞
You are absolutely right, for most websites your answer makes absolute sense. But, my use case is slightly different and that’s why I wanted to know how to index user profiles. Even if tons of useless pages (user profiles) gets indexed, then also that’s what I need for my use case.
If there’s a way to achieve it then, please let me know. 
I am also want to index the Categories and Tags.
Is there a way to edit the robots.txt?
Search meta, the reasons this is disabled is well documented.
1 个赞
@Stephen, now that we can “play” with robot.txt here -> /admin/customize/robots do you guys advise to allow those two -> Disallow: /tags - Disallow: /tags/ ?
2 个赞
We do not advise messing with this at all.
4 个赞
旧的，但有一个快速技巧：发送一封包含用户类别、标签等链接的 Gmail，然后让用户——比如管理员和版主——打开这些链接。这样 Googlebot 就不再遵循 robots.txt 了，并且会索引它们。
其余的，一如既往，取决于 Google 是否会深入跟踪或忘记。
这个话题，以及许多类似的话题，在这里给人一种 robots.txt 是严格规则的印象。它不是。它只是一条指南，仅此而已。
垃圾邮件/盗窃的担忧是合理的，但垃圾机器人不在乎 robots.txt 中的内容。如果某些东西是可访问的，它们就会去那里。
1 个赞