Troppi crawler, è un problema?

Well, that depends on how large your site is and the user agent for that crawler. You don’t want to block Googlebot, for example.

Have a look here

3 Mi Piace