# Web Crawlers

**URL:** <https://meta.discourse.org/t/web-crawlers/271767>\
**Category:** Data & reporting\
**Created:** [July 16, 2023, 6:41am UTC](https://meta.discourse.org/t/web-crawlers/271767 "2023-07-16T06:41:00Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![anon65426961](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@anon65426961](https://meta.discourse.org/u/anon65426961)\
**Post date:** [July 16, 2023, 7:06pm UTC](https://meta.discourse.org/t/web-crawlers/271767/6 "2023-07-16T19:06:14Z")

</div>

Sneaky imposters even worse than weasels they can be:

> **[How to check that a robot belongs to Yandex |](https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html)**
>
> Some robots can disguise themselves as Yandex robots by indicating the relevant User Agent. You can check the authenticity of a robot using a reverse DNS lookup

"# How to check that a robot belongs to Yandex

Some robots can disguise themselves as Yandex robots by indicating the relevant User Agent. You can check the authenticity of a robot using reverse DNS lookup.

Just follow these steps:

1. Determine the IP address of the[user agent in question](https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html#robot-in-logs) using your server logs.

2. Use a reverse DNS lookup of the IP address to determine the host domain name.

3. Check whether the host belongs to Yandex. All Yandex robot names end in [yandex.ru](http://yandex.ru), [yandex.net](http://yandex.net) or [yandex.com](http://yandex.com). If the host name has a different ending, the robot does not belong to Yandex.

4. Make sure that the name is correct. Use a forward DNS lookup to get the IP address corresponding to the host name. It should match the IP address used in the reverse DNS lookup. If the IP addresses do not match it means that the host name is fake."

---

_[View the full topic](https://meta.discourse.org/t/web-crawlers/271767)._