The admin dashboard can now identify likely crawler traffic — automated bots and scrapers that use real browser JavaScript but aren’t human visitors — and separate those pageviews from your genuine traffic numbers.
In this topic, we’ll share how we’ve approached identifying this traffic and how you can turn this feature on today.
What’s changed
Discourse has always filtered Crawlers that identify themselves with standard bot user agents (Googlebot, for example) into a separate category of site traffic data. But modern automated traffic is sneakier: it runs a full browser, executes JavaScript, and blends into your pageview data as if it were a real visit. Admins would see traffic spikes in Anonymous page views that turned out to be scrapers, or visitor counts inflated by AI bots — with no way to tell the difference between them and true anonymous (i.e. not logged in) human traffic.
This update adds a Likely Crawlers category to your site traffic data. When improved crawler detection is enabled, pageviews that our heuristics flag as likely automated are broken out separately, so your human traffic numbers are more accurate.
How detection works
The system scores each browser pageview using a set of behavioral signals:
- Automation user agent — the UA explicitly names a headless browser or automation tool (HeadlessChrome, Playwright, Puppeteer, Selenium, PhantomJS, jsdom)
- Verified crawler ASN — the IP’s autonomous system number is a known crawler network (Baidu, Ahrefs, Yandex, Internet Archive, etc.)
- Datacenter ASN — the IP is in a major cloud or hosting provider’s address space
- Velocity — the IP + user agent generates unusually many pageviews per hour
- Rapid navigation — the median gap between pageviews is under 5 seconds
- Session churn — many short sessions (mostly one page each) from the same IP + user agent
- Missing engagement — no recorded human-like interaction (mouse movement, keystrokes, scrolling) for a session and its neighbors
- Bad referrer ratio — most pageviews arrive with no referrer or an external one, suggesting direct URL access rather than navigation through the site
- Single direct request — a single unengaged pageview with no referrer; scores higher if the URL carries a locale parameter
- Rapid IP rotation — multiple IPs used within a single session, cycling faster than a human would
- Stale Chromium — an outdated Chrome version with no engagement signals
Scores from multiple signals stack; when a pageview has a certain amount of the above signals, we classify it as likely crawler.
Turning on improved crawler detection in your community
For now, this is considered an experimental change. We’re still tuning the detection heuristics and welcoming feedback.
To turn this on, head to the Upcoming changes page in your admin area and find the Improved crawler detection item. Update the Enabled for… field to opt your site in.
Once enabled, the Site Traffic section of your admin dashboard will show likely crawlers as a separate category.
Important notes:
- This is not a perfect science and we will continue to hone our heuristics over time. However, we will not backfill as we update these calculations.
- Classifying a pageview as a likely crawler does not deny or rate-limit the request.
- Likely crawler pageviews are excluded from referrers, countries, and other traffic breakdowns — the same way known crawlers are handled today.
What do you think?
We’d love your feedback — especially from communities that have noticed unusual traffic patterns or unexplained spikes. Are the numbers looking right for your site? Are there visits being flagged that seem like real users to you? Share what you’re seeing in this topic.
