This is now complete thanks to @neil!
You can see “top crawlers” at https://yoursite/admin/reports/page_view_crawler_reqs
You can blacklist bad crawlers by setting blacklisted crawler user agents
Alternatively, if you wish to only allow particular crawlers, you can set the whitelist with the setting whitelisted crawler user agents