Unidentified Crawler with High Amounts of Pageviews

Community Building Data & reporting

AI-generated summary

In the discussion Unidentified Crawler with High Amounts of Pageviews, anthonylavado reported an unidentified crawler with over 500k page views in a month, averaging 10,000-15,000+ hits a day. After codinghorror offered assistance, anthonylavado explained that they are a self-hosted FLOSS project and cannot utilize the support.

Falco suggested checking the nginx logs for requests with the respective user agent and IP. anthonylavado later revealed that they had identified the issue - HAProxy performing health checks every 5 seconds. Falco recommended that health checks should set a proper User Agent to prevent similar issues in the future.

1

Hi all!

When we look at our crawler page view count, there seems to be an unidentified entry that is racking up over 500k page views in a month:

Screen Shot 2020-01-04 at 3.13.28 AM
Screen Shot 2020-01-04 at 3.13.28 AM1790×594 64.6 KB

Is there any easy way to find out what this could be? It seems to average some 10,000-15,000+ hits a day.

2

If you are on our hosting email support and we can handle it for you.

3

We’re a self hosted FLOSS project, so I suppose that’s out of the question :slight_smile:

I know I could put some more filtering on there and get our infra guy to look at more logs - I guess I was just wondering if anyone else had seen this before.

4

Check the nginx logs (access.log) for requests coming with that user agent and the respective IP.

5

I just met with our lead SysAdmin in person and figured it out. It’s HAProxy doing a health check every 5 seconds :joy:

6

Health checks should set a proper User Agent, so this doesn’t happen again. Also, you can use the /srv/status route for health checking.

7

“Yeah, it should - but I’m lazy.” - SysAdmin

I’ll see what I can do. Thanks!