How to dive in and see what happened there?
You can find detailed information about which bots are crawling your site in the web crawler report over at /admin/reports/web_crawlers.
Once you check there, you can block specific crawlers from your site by adding them to the
blocked crawler user agents site setting. That setting takes a case insensitive word that exists in the user agent string. For example, to block the
rogerbot user agent, add
rogerbot to the setting.
Most of the visits was from a danish type of netarchive under the Royal Danish Library
Seems pretty fair
Dont want to block that initiative…
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.