Automated crawler blocking based on reputation

Feature
1

We recently had a spike in bad crawler activity that we’ve subsequently blocked, it was the mauibot and the IP was listed on https://www.abuseipdb.com/. (Props to the support team for guidance there!)

I can see that there has been some work around better handling of crawlers. And it got me thinking that perhaps there might be a way to automate (temporary?) blocking of IPs based on reputation?

Otherwise, is there a list of “recommended”/common crawlers that would be ok to explicitly whitelist?

2 Likes
2

Honestly if you want you can block every single crawler except Google and have no functional change in people visiting, because all other search engines provide negligible traffic. (Exception for Chinese or Russian sites, though)

We don’t do this for political reasons but you have no such limitation.

2 Likes