About 1/3 of our traffic is from crawlers (about 250K last month). Is there a way to block these but allow Google’s crawlers?
This is now possible via site settings, search for “crawler”, specifically
blacklisted crawler user agents. There is also
whitelisted crawler user agents if you want to be very strict and disallow everything except a specific crawler.