Throttling user agents and settings format


(Mark Walkom) #1

In the slow down crawler user agents setting, is the value case insensitive as per the blacklisted crawler user agents.

eg is YandexBot a valid setting or does it need to be the complete thing - Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots).


(Jeff Atwood) #2

The answer should be no, but I’m not sure what we actually did here. @sam worked on this I think.


(Sam Saffron) #3

This is a directive that is added to robots.txt with the same casing that you enter in the setting per: FEATURE: allow for setting crawl delay per user agent · discourse/discourse@3a7b696 · GitHub

So, depending on how the particular bot interprets robots.txt it will be case sensitive or insensitive.

Yandex documentation seems to imply you would use the string Yandex there.


(Mark Walkom) #4

Cool, thanks for that :slight_smile: