Throttling user agents and settings format

Support
1

In the slow down crawler user agents setting, is the value case insensitive as per the blacklisted crawler user agents.

eg is YandexBot a valid setting or does it need to be the complete thing - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots).

2

The answer should be no, but Iā€™m not sure what we actually did here. @sam worked on this I think.

3

This is a directive that is added to robots.txt with the same casing that you enter in the setting per: https://github.com/discourse/discourse/commit/3a7b6967035d7dd1a4789e98577b2f3ae6c04663

So, depending on how the particular bot interprets robots.txt it will be case sensitive or insensitive.

Yandex documentation seems to imply you would use the string Yandex there.

3 Likes
4

Cool, thanks for that :slight_smile:

2 Likes