User profiles / User directory in robots.txt / x-robots-tag header

Contribute Bug
3

Good find @Moin

Currently, we already add noindex to /u routes. However, due to robots.txt blocking this, search engines are not able to see the header.

So then we are referred to a previous change FIX: Always noindex /u routes - Pull Request #27712 - discourse/discourse - GitHub

which says

Secondly, SiteSetting.hide_user_profiles_from_public raises a Forbidden, which disallows our after_action: add no index header from triggering.

This PR makes sure that the no index header gets added via before_action instead. We may consider removing /u from

discourse/app/controllers/robots_txt_controller.rb

Line 24 in 2900cbe

DISALLOWED_WITH_HEADER_PATHS = %w[/badges /u/ /my /search /tag//l /g /t//.rss /c/.rss]

Unfortunately, that does not always work. While /u/rgj does have a x-robots-tag: noindex header, /u/rgj/summary does not, so it seems like the most recent change is having some unwanted side effects.

(Moving to Contribute > Bug)

1 Like