About 1/3 of our traffic is from crawlers (about 250K last month). Is there a way to block these but allow Google’s crawlers?
This is now possible via site settings, search for “crawler”, specifically blocked crawler user agents. There is also allowed crawler user agents if you want to be very strict and disallow everything except a specific crawler.
12 Likes