How to block all crawlers but Google's

About 1/3 of our traffic is from crawlers (about 250K last month). Is there a way to block these but allow Google’s crawlers?

3 Likes

This is now possible via site settings, search for “crawler”, specifically blacklisted crawler user agents. There is also whitelisted crawler user agents if you want to be very strict and disallow everything except a specific crawler.

8 Likes