Dear Discourse Team, first things first: We want to salute you for the spirit and endurance on conceiving and maintaining Discourse. It is always a pleasure to use for us and our community members. We summarized some minor observations made today and hope you will find them useful. On the other hand, we will also be happy to hear back about any misunderstandings on our side. With kind regards, Andreas. Introduction While investigating the behavior of our hosted Discourse instance at https://community.crate.io/ with respect to its robots.txt definition file [1], we discovered that Googlebot might not honor the settings as intended. Evaluation [image]Excluding user profiles in robots.txt (or allow edit of file) User profiles are now disallowed from indexing by default. Let’s exercise it on behalf of the robots.txt on Meta [2], with my profile URL https://meta.discourse.org/u/amotl/. The difference can quickly be spotted by comparing the outcomes from this free robots.txt validator: [image] – Evaluate with Googlebot [image] – Evaluate with MSNBot Research On this very topic, we believe to have found the answer already: Thoughts So, we are inclined to figure Googlebot might currently ignore the rules defined within the User-agent: * section [3] completely and only honors the rules within section User-agent: Googlebot [4]. https://community.crate.io/robots.txt ↩︎ https://meta.discourse.org/robots.txt ↩︎ robots.txt, section User-agent: * User-agent: * Disallow: /admin/ Disallow: /auth/ Disallow: /assets/browser-update*.js Disallow: /email/ Disallow: /session Disallow: /user-api-key Disallow: /*?api_key* Disallow: /*?*api_key* Disallow: /badges Disallow: /u Disallow: /my Disallow: /search Disallow: /tag Disallow: /g Disallow: /t/*/*.rss Disallow: /c/*.rss ↩︎ robots.txt, section User-agent: Googlebot User-agent: Googlebot Disallow: /auth/ Disallow: /assets/browser-update*.js Disallow: /email/ Disallow: /session Disallow: /*?api_key* Disallow: /*?*api_key* ↩︎

[image] amotl: Thoughts So, we are inclined to figure Googlebot might currently ignore the rules defined within the User-agent: * section [3] completely and only honors the rules within section User-agent: Googlebot [4]. This is correct and intentionally implemented this way. [image] Jagster: Again — robots.txt is just a request. Therefore Googlebot receives an extra http-header X-Robots-Tag: noindex for pages which really should not be indexed. See: https://github.com/discourse/discourse/commit/758e1608626ce78566dd39004a4a6251d3bbfc60 For your own domains you may use Google Search Console → Inspect URL Then try to add a user-profile url to index – e.g. https://www.example.com/u/jacob [grafik]

If I understod right the answer is not that easy. If a single user has send a link using Gmail, googlebot doesn’t honor robots.txt. Same thing if a link is shared to somewhere (backlinks) where it looks to googlebot as normal everyday link. Again — robots.txt is just a request. There is another but more… Quite many bots identify themselves as googlebot and the reality can be found only via IP.

Dear Ayke, [image] rrit: Therefore Googlebot receives an extra http-header X-Robots-Tag: noindex for pages which really should not be indexed. Confirmed. $ http https://meta.discourse.org/u/amotl --print hH | grep -i robot X-Robots-Tag: noindex [image] rrit: This is correct and intentionally implemented this way. Thank you very much for your quick response and explanation and for referencing the corresponding patch. With kind regards, Andreas.

I moved this to #support, thank you for your delightfully written bug report here. It has taken us so much time to finely tune our rules to keep google happy x-robots-tag noindex is unevenly supported but it is an industry standard. The issue with just banning crawling was that for some reason, under certain conditions, pages could find themselves in the google index and then there was no easy way to remove them cause crawling was banned, a bit of a chicken meets egg problem.

Dear Sam, thank you very much for sharing more details about this matter. As always, I am amazed about the level of awesomeness you are pouring into every detail of Discourse. [image] sam: x-robots-tag noindex is unevenly supported but it is an industry standard I have to admit I haven’t been aware of this until now. So, thanks again! [image] rrit: For [checking this on] your own domains you may use Google Search Console. [image] Great. Thanks. If this has become a widely accepted standard now, will there be hope that some of the free robots.txt-validators might also start honoring that in the future? Are you aware about any which already implement corresponding additional header checks on top of reading the robots.txt today, like Google Search Console’s Inspect URL does? That would probably help people to avoid the same confusion we were running into. With kind regards, Andreas.

Generic rules in "robots.txt" not picked up by Googlebot

Product Support

amotl (Andreas Motl) March 1, 2022, 3:50pm 4

Dear Ayke,

Confirmed.

$ http https://meta.discourse.org/u/amotl --print hH | grep -i robot
X-Robots-Tag: noindex

Thank you very much for your quick response and explanation and for referencing the corresponding patch.

With kind regards,
Andreas.

Topic		Replies	Views
Pages listed in the robots.txt are crawled and indexed by Google Product Support	18	3625	July 30, 2019
Excluding user profiles in robots.txt (or allow edit of file) Feature	4	2636	May 24, 2014
Google changed how they process robots.txt in Discourse? Product Support	20	2031	December 22, 2020
Malformed robots.txt causing issues with indexing Product Support	7	2036	September 14, 2019
Indexing User Profiles Product Support	2	197	September 4, 2025

Generic rules in "robots.txt" not picked up by Googlebot

Related topics