# Google a-t-il modifié la façon dont il traite robots.txt dans Discourse ?

**URL:** <https://meta.discourse.org/t/google-changed-how-they-process-robots-txt-in-discourse/151064>\
**Category:** Product Support\
**Created:** [Mai 11, 2020, 3:37 UTC](https://meta.discourse.org/t/google-changed-how-they-process-robots-txt-in-discourse/151064 "2020-05-11T15:37:06Z")\
**Posts on this page:** 1\
**Showing post:** 20

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [Décembre 22, 2020, 6:01 UTC](https://meta.discourse.org/t/google-changed-how-they-process-robots-txt-in-discourse/151064/20 "2020-12-22T06:01:49Z")

</div>

J’ai créé cette PR pour résoudre ce problème :

> <https://github.com/discourse/discourse/pull/11553>
>
> Googlebot handles no-index headers very elegantly. It advises to leave as many r…outes as possible open and uses headers for high fidelity rules regarding indexes.
> 
> Discourse adds special \`x-robot-tags\` noindex headers to users, badges, groups, search and tag routes.
> 
> Following up on b52143feff8c32f2 we now have it so Googlebot gets special handling.
> 
> Rest of the crawlers get a far more aggressive disallow list to protect against excessive crawling.

Google conserve sa règle spéciale et nous livrons une meilleure protection pour divers robots moins sophistiqués. Le fichier robots par défaut ressemble maintenant à ceci :

```plaintext
# Voir http://www.robotstxt.org/robotstxt.html pour la documentation sur l'utilisation du fichier robots.txt
#
User-agent: mauibot
Disallow: /

User-agent: semrushbot
Disallow: /

User-agent: ahrefsbot
Disallow: /

User-agent: blexbot
Disallow: /

User-agent: seo spider
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /auth/
Disallow: /assets/browser-update*.js
Disallow: /email/
Disallow: /session
Disallow: /user-api-key
Disallow: /*?api_key*
Disallow: /*?*api_key*
Disallow: /badges
Disallow: /u
Disallow: /my
Disallow: /search
Disallow: /tags
Disallow: /g
Disallow: /t/*/*.rss
Disallow: /tags/*.rss
Disallow: /c/*.rss

User-agent: Googlebot
Disallow: /admin/
Disallow: /auth/
Disallow: /assets/browser-update*.js
Disallow: /email/
Disallow: /session
Disallow: /user-api-key
Disallow: /*?api_key*
Disallow: /*?*api_key*

```

---

_[View the full topic](https://meta.discourse.org/t/google-changed-how-they-process-robots-txt-in-discourse/151064)._