Web Crawlers

Sneaky imposters even worse than weasels they can be:

"# How to check that a robot belongs to Yandex

Some robots can disguise themselves as Yandex robots by indicating the relevant User Agent. You can check the authenticity of a robot using reverse DNS lookup.

Just follow these steps:

  1. Determine the IP address of the user agent in question using your server logs.

  2. Use a reverse DNS lookup of the IP address to determine the host domain name.

  3. Check whether the host belongs to Yandex. All Yandex robot names end in yandex.ru, yandex.net or yandex.com. If the host name has a different ending, the robot does not belong to Yandex.

  4. Make sure that the name is correct. Use a forward DNS lookup to get the IP address corresponding to the host name. It should match the IP address used in the reverse DNS lookup. If the IP addresses do not match it means that the host name is fake."