Throttling user agents and settings format

In the slow down crawler user agents setting, is the value case insensitive as per the blacklisted crawler user agents.

eg is YandexBot a valid setting or does it need to be the complete thing - Mozilla/5.0 (compatible; YandexBot/3.0; +

The answer should be no, but Iā€™m not sure what we actually did here. @sam worked on this I think.

This is a directive that is added to robots.txt with the same casing that you enter in the setting per:

So, depending on how the particular bot interprets robots.txt it will be case sensitive or insensitive.

Yandex documentation seems to imply you would use the string Yandex there.


Cool, thanks for that :slight_smile: