You can do what you like but I’d be careful here. There are other legit bots.
Why don’t you dig into the crawler report and see which ones are causing the most noise?
3 Likes
You can do what you like but I’d be careful here. There are other legit bots.
Why don’t you dig into the crawler report and see which ones are causing the most noise?