MegaIndex bot did about 4,000 pageviews on one day

Update: 08/13/2022

The bot visited our site again on 08/04/2022 (crawler site)

Report: Consolidated Pageviews

image

Report: Web Crawler User Agents

Report: Top Traffic Source

Clearly letting the bot MegaIndex.ru/2.0 index the site does not appear to be generating traffic to the site.
Note: AFAIK yandex.ru is different from Megaindex.ru.


For blocking crawlers there is robots.txt which as noted

https://<Discoruse site>/admin/customize/robots

but not all crawlers will honor robots.txt. :slightly_frowning_face:


As noted above by IAmGav there are other crawler settings.

4 Likes