Throttling user agents and settings format

Support
1

In the slow down crawler user agents setting, is the value case insensitive as per the blacklisted crawler user agents.

eg is YandexBot a valid setting or does it need to be the complete thing - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots).

2

The answer should be no, but I’m not sure what we actually did here. @sam worked on this I think.

3

This is a directive that is added to robots.txt with the same casing that you enter in the setting per: FEATURE: allow for setting crawl delay per user agent · discourse/discourse@3a7b696 · GitHub

So, depending on how the particular bot interprets robots.txt it will be case sensitive or insensitive.

Yandex documentation seems to imply you would use the string Yandex there.

3 Likes
4

Cool, thanks for that :slight_smile:

2 Likes