Well, that depends on how large your site is and the user agent for that crawler. You don’t want to block Googlebot, for example.
Have a look here
3 إعجابات
Well, that depends on how large your site is and the user agent for that crawler. You don’t want to block Googlebot, for example.
Have a look here