Smarter handling of random crawler traffic

This is now complete thanks to @neil!

You can see “top crawlers” at https://yoursite/admin/reports/page_view_crawler_reqs

You can blacklist bad crawlers by setting blacklisted crawler user agents

Alternatively, if you wish to only allow particular crawlers, you can set the whitelist with the setting whitelisted crawler user agents

10 Likes