Crawler-mania - cosa è successo?

How to dive in and see what happened there?

You can find detailed information about which bots are crawling your site in the web crawler report over at /admin/reports/web_crawlers.

Once you check there, you can block specific crawlers from your site by adding them to the blocked crawler user agents site setting. That setting takes a case insensitive word that exists in the user agent string. For example, to block the rogerbot user agent, add rogerbot to the setting.

4 Mi Piace

Thanks

Most of the visits was from a danish type of netarchive under the Royal Danish Library

Seems pretty fair :slight_smile:

https://en.statsbiblioteket.dk/national-library-division/netarchive

Dont want to block that initiative…

3 Mi Piace

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.