# Throttling user agents and settings format

**URL:** https://meta.discourse.org/t/throttling-user-agents-and-settings-format/99499
**Category:** Support
**Created:** [October 14, 2018, 11:47pm UTC](https://meta.discourse.org/t/throttling-user-agents-and-settings-format/99499 "2018-10-14T23:47:29Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Mark\_Walkom](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mark_walkom/32/77411_2.png) [@Mark\_Walkom](https://meta.discourse.org/u/Mark_Walkom)
#### Post date: [October 14, 2018, 11:47pm UTC](https://meta.discourse.org/t/throttling-user-agents-and-settings-format/99499/1 "2018-10-14T23:47:29Z")

</div>

In the `slow down crawler user agents` setting, is the value case insensitive as per the `blacklisted crawler user agents`.

eg is `YandexBot` a valid setting or does it need to be the complete thing - `Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)`.

---

<div class="post-metadata">

### Author: ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [October 15, 2018, 12:42am UTC](https://meta.discourse.org/t/throttling-user-agents-and-settings-format/99499/2 "2018-10-15T00:42:39Z")

</div>

The answer _should_ be no, but I’m not sure what we actually did here. @sam worked on this I think.

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [October 15, 2018, 12:49am UTC](https://meta.discourse.org/t/throttling-user-agents-and-settings-format/99499/3 "2018-10-15T00:49:31Z")

</div>

This is a directive that is added to robots.txt with the same casing that you enter in the setting per: [FEATURE: allow for setting crawl delay per user agent · discourse/discourse@3a7b696 · GitHub](https://github.com/discourse/discourse/commit/3a7b6967035d7dd1a4789e98577b2f3ae6c04663)

So, depending on how the particular bot interprets robots.txt it will be case sensitive or insensitive.

`Yandex` documentation seems to imply you would use [the string Yandex there](https://yandex.com/support/webmaster/controlling-robot/robots-txt.html#crawl-delay).

---

<div class="post-metadata">

### Author: ![Mark\_Walkom](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mark_walkom/32/77411_2.png) [@Mark\_Walkom](https://meta.discourse.org/u/Mark_Walkom)
#### Post date: [October 18, 2018, 11:43pm UTC](https://meta.discourse.org/t/throttling-user-agents-and-settings-format/99499/4 "2018-10-18T23:43:24Z")

</div>

Cool, thanks for that 🙂

---

<div class="post-metadata">

### Author: ![system](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/system/32/443519_2.png) [@system](https://meta.discourse.org/u/system)
#### Post date: [July 13, 2021, 5:23pm UTC](https://meta.discourse.org/t/throttling-user-agents-and-settings-format/99499/5 "2021-07-13T17:23:30Z")

</div>