This is now possible via site settings, search for “crawler”, specifically blocked crawler user agents. There is also allowed crawler user agents if you want to be very strict and disallow everything except a specific crawler.
12 Likes