A bug has been identified regarding how Discourse handles user profile visibility and robots.txt when ‘Hide user profiles from public’ is enabled. RGJ initially questioned why /u isn’t disallowed in robots.txt, noting that search engines receive 403 errors instead of seeing noindex headers — potentially harming SEO. While Meta is the only known instance with Disallow: /u, Moin suggested it may be customized, bypassing recent migrations. RGJ confirmed this is a bug, referencing a 2022 PR (#27712) that moved noindex header injection to before_action to ensure it triggers even on 403s. However, the fix is incomplete: while /u/username correctly returns x-robots-tag: noindex, /u/username/summary does not — indicating inconsistent behavior. The issue stems from the hardcoded list of paths in robots_txt_controller.rb that are meant to receive noindex headers, but the logic fails to apply universally. This bug affects search engine indexing of user profile subpages and requires further patching to ensure consistent noindex headers across all /u/* routes.
EDIT this turns out to be a bug, see post #3
If “Hide user profiles from public” is checked, shouldn’t /u be disallowed in robots.txt ?
Otherwise search engines will hit a 403 which might affect ranking and visibility.
This was supposedly implemented in 2014 Excluding user profiles in robots.txt (or allow edit of file) - #2 by neil Disallow /users/ in robots.txt · discourse/discourse@8267a45 · GitHub
The only Discourse forum I could find that has Disallow: /u is Meta 
Maybe Meta was customized so the migration in FIX: Remove /u/ from robots by nattsw · Pull Request #30782 · discourse/discourse · GitHub was skipped
Good find @Moin
Currently, we already add
noindexto /u routes. However, due to robots.txt blocking this, search engines are not able to see the header.
So then we are referred to a previous change FIX: Always noindex /u routes - Pull Request #27712 - discourse/discourse - GitHub
which says
Secondly,
SiteSetting.hide_user_profiles_from_publicraises aForbidden, which disallows ourafter_action: add no index headerfrom triggering.
This PR makes sure that the no index header gets added via
before_actioninstead. We may consider removing /u fromdiscourse/app/controllers/robots_txt_controller.rb
Line 24 in 2900cbe
DISALLOWED_WITH_HEADER_PATHS = %w[/badges /u/ /my /search /tag//l /g /t//.rss /c/.rss]
Unfortunately, that does not always work. While /u/rgj does have a x-robots-tag: noindex header, /u/rgj/summary does not, so it seems like the most recent change is having some unwanted side effects.
(Moving to Contribute > Bug)