Improving how we detect and flag likely crawlers in your traffic data

The challenge is that “likely crawlers” quite often are using legit user agents. Sometimes, the browser version looks quite old, but it is not a strong enough signal to consider it a likely bot. That’s why we gather so many metrics and try to combine them to gauge likelihood.

When the crawler honestly introduces itself via a user agent like Bingbot or Googlebot, it lands directly in “crawlers” category.

1 Like