I want the user profiles on my website to get indexed in Google. I know it might attract a lot of spam but we can handle that, although getting user profiles indexed in Google is more important for us. So is there a way to do this?
I checked the old topics on meta and they suggested to edit robot.txt. Is the answer still the same? And if the yes, then how to edit the robot.txt?
AI-generated summary
The discussion revolves around whether and how to edit robots.txt to allow Google indexing of user profiles, categories, and tags on a Discourse site. thegurjyot seeks to index user profiles despite potential spam, arguing it’s critical for their use case. Stephen warns that indexing profiles wastes crawl budget with low-value pages, a concern echoed in Discourse’s design. thegurjyot reiterates their unique need, while Viki extends the request to categories and tags. Stephen directs users to search meta for documented reasons behind the default restrictions. Jeremie_Leroy asks if it’s safe to override /tags and /tags/ rules via Discourse’s custom robots.txt editor. codinghorror strongly advises against any modifications. Jagster offers a workaround: manually triggering Googlebot indexing by sharing links via Gmail and having admins click them, noting that robots.txt is merely a guideline — malicious bots ignore it anyway, and legitimate crawlers may still index if content is linked. The consensus leans against editing robots.txt, but acknowledges its non-enforceable nature and workarounds for specific needs.
User profiles are disallowed as the version which is seen by the crawler contains no useful information.
It’s not a question of whether the server can handle it or “take the hit”, it’s the tens of thousands of effectively empty pages which will waste the crawl time allocated to your site by the search engines.
You are absolutely right, for most websites your answer makes absolute sense. But, my use case is slightly different and that’s why I wanted to know how to index user profiles. Even if tons of useless pages (user profiles) gets indexed, then also that’s what I need for my use case.
If there’s a way to achieve it then, please let me know. 
I am also want to index the Categories and Tags.
Is there a way to edit the robots.txt?
Search meta, the reasons this is disabled is well documented.
@Stephen, now that we can “play” with robot.txt here → /admin/customize/robots do you guys advise to allow those two → Disallow: /tags - Disallow: /tags/ ?
We do not advise messing with this at all.
Old one, but there is one fast trick: send a gmail where is link to user category, tags etc. and let users — like admins and mods — open those links. Then googlebot doesn’t follow robots.txt anymore and will index those.
Rest is, as usual, matter of Google will it follow deeper or forget.
This topic, and quite many similar ones, gives an image here where robots.txt would a strict rule. No it is not. It is just a guideline, nothing else.
And spam/steeling fear is valid, but trashy bots don’t care what is in robots.txt. If something is reachable they will go there.