How to dive in and see what happened there?
You can find detailed information about which bots are crawling your site in the web crawler report over at /admin/reports/web_crawlers.
Once you check there, you can block specific crawlers from your site by adding them to the blocked crawler user agents site setting. That setting takes a case insensitive word that exists in the user agent string. For example, to block the rogerbot user agent, add rogerbot to the setting.
4 Likes
Thanks
Most of the visits was from a danish type of netarchive under the Royal Danish Library
Seems pretty fair 
https://en.statsbiblioteket.dk/national-library-division/netarchive
Dont want to block that initiative…
3 Likes
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.