Try checking /admin/reports/web_crawlers for a detailed report on the crawlers. You can then proceed to block whichever one you do not find helpful or useful by adding them to the blocked crawler user agents site setting. That setting takes a case insensitive word that exists in the user agent string.
Sorry to bump this, but I haven’t quite found what I am looking for. So for one, the crawler stats the reports page has to offer give me info on the number of crawler page views, and their user agents, but little else in terms of what to do with that information. Crawlers in general are not a problem for me, but spikes in crawler access make the dashboard stats almost unusable
You see, one bump on december 16 leads to such a squeeze of all the charts that they become hard to read. Plus, what value does that information even hold? None. That was the day of the Google outage last week, so probably once google came back, all those crawlers ramped up their activity. Big deal.
So my question is this: Can I remove crawlers from the main admin dashboard stats?