Are web crawler page views included in the monthly 500K page view limit for the free plan?
As an administrator of a Discourse forum, I’ve observed that web crawlers account for more than 95% of our total page views. Even so, we remain under 500K page views per month. That said, crawler traffic and overall page views have shown a steady upward trend over the past several years.
Given this, understanding how to monitor and manage crawler activity is increasingly important—particularly for administrators who are new to operating a Discourse instance.
It may be beneficial for the initial site setup documentation to include clear guidance on:
- How page views are counted (including whether crawler traffic is included)
- How to monitor crawler activity
- Methods for managing or limiting crawler traffic where appropriate
