I want the user profiles on my website to get indexed in Google. I know it might attract a lot of spam but we can handle that, although getting user profiles indexed in Google is more important for us. So is there a way to do this?
I checked the old topics on meta and they suggested to edit robot.txt. Is the answer still the same? And if the yes, then how to edit the robot.txt?
User profiles are disallowed as the version which is seen by the crawler contains no useful information.
It’s not a question of whether the server can handle it or “take the hit”, it’s the tens of thousands of effectively empty pages which will waste the crawl time allocated to your site by the search engines.
You are absolutely right, for most websites your answer makes absolute sense. But, my use case is slightly different and that’s why I wanted to know how to index user profiles. Even if tons of useless pages (user profiles) gets indexed, then also that’s what I need for my use case.
If there’s a way to achieve it then, please let me know. 
I am also want to index the Categories and Tags.
Is there a way to edit the robots.txt?
Search meta, the reasons this is disabled is well documented.
@Stephen, now that we can “play” with robot.txt here → /admin/customize/robots do you guys advise to allow those two → Disallow: /tags - Disallow: /tags/ ?
We do not advise messing with this at all.
오래된 글이지만, 빠른 트릭이 하나 있습니다: 사용자 카테고리, 태그 등 링크가 포함된 Gmail 메일을 보내고, 관리자나 모더레이터와 같은 사용자들이 해당 링크를 열게 하세요. 그러면 구글봇이 더 이상 robots.txt를 따르지 않고 해당 페이지를 인덱싱합니다.
나머지는, 평소처럼, 구글이 더 깊이 파고들지 잊어버릴지의 문제입니다.
이 주제와 꽤 많은 유사한 주제들이 robots.txt가 엄격한 규칙인 것처럼 여기에서 이미지를 주고 있습니다. 아닙니다. robots.txt는 그저 가이드라인일 뿐, 그 이상도 이하도 아닙니다.
그리고 스팸/도용에 대한 두려움은 타당하지만, 쓰레기 봇은 robots.txt에 무엇이 있는지를 신경 쓰지 않습니다. 접근할 수 있는 것이라면 어디든 갑니다.