Hi all! When we look at our crawler page view count, there seems to be an unidentified entry that is racking up over 500k page views in a month: [Screen Shot 2020-01-04 at 3.13.28 AM] Is there any easy way to find out what this could be? It seems to average some 10,000-15,000+ hits a day.

I just met with our lead SysAdmin in person and figured it out. It’s HAProxy doing a health check every 5 seconds :joy:

We’re a self hosted FLOSS project, so I suppose that’s out of the question :slight_smile: I know I could put some more filtering on there and get our infra guy to look at more logs - I guess I was just wondering if anyone else had seen this before.

Check the nginx logs (access.log) for requests coming with that user agent and the respective IP.

Health checks should set a proper User Agent, so this doesn’t happen again. Also, you can use the /srv/status route for health checking.

“Yeah, it should - but I’m lazy.” - SysAdmin I’ll see what I can do. Thanks!

Unidentified Crawler with High Amounts of Pageviews

Community Building Data & reporting

codinghorror (Jeff Atwood) January 4, 2020, 5:31pm 2

If you are on our hosting email support and we can handle it for you.

Topic		Replies	Views
Possible wrong counter of pageviews Product Support	2	618	February 10, 2019
Consolidated Pageviews accuracy Data & reporting	19	2528	August 25, 2019
Do hits to /srv/status count as crawlers? Data & reporting	13	1077	April 2, 2020
Consolidated Pageviews: exclude uptime crawler from report without blocking? Product Support dashboard-reports	4	175	August 29, 2024
Can I ignore some user agents? Product Support	6	981	August 23, 2022

Unidentified Crawler with High Amounts of Pageviews

Related topics