# Do hits to /srv/status count as crawlers?
**URL:** https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431
**Category:** Data & reporting
**Created:** [5 במרץ, 2020, 6:00pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431 "2020-03-05T18:00:22Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [5 במרץ, 2020, 6:00pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/1 "2020-03-05T18:00:22Z")
I’ve got a site running 3 kubernetes pods (it’s what they wanted!). The crawler stats show \> 120K pageviews/day. There are . . . very few . . . users and fewer than 1000 posts, so there isn’t much crawling to be done. Staring at `production.log` of one of the pods for a few minutes shows only traffic to `/srv/status` that k8s is using as a health monitor. Should that be getting counted as a crawler? Is there something else I might be missing?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [6 במרץ, 2020, 6:19pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/2 "2020-03-06T18:19:01Z")
No, that should not be counted.
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [6 במרץ, 2020, 8:35pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/3 "2020-03-06T20:35:00Z")
120K is right on the mark for a constant 5k/hr, so I’d say they are being counted even though they shouldn’t.
Maybe something’s going wrong in the detection?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [6 במרץ, 2020, 8:36pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/4 "2020-03-06T20:36:08Z")
I can confirm that they are being counted (and they are logged from 127.0.0.1)
```plaintext
SELECT * from web_crawler_requests
where date = '2020-03-05'
order by count desc
```
EDIT: also, I’m bad with numbers, apparently, and it’s ~12K/day.
And that is confusing too because if I `grep -c /srv/status` in the 3 pods they each have ~15K lines matching `/srv/status` in the 20 hours that they have been up. So that would be close to 50K/day.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [6 במרץ, 2020, 8:49pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/5 "2020-03-06T20:49:39Z")
They are counted and if you don’t want them counted you have a clear option you can use
>
Set the Discourse-Track-View HTTP header to 0 on the originating request
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [6 במרץ, 2020, 8:57pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/6 "2020-03-06T20:57:28Z")
> [@sam](#):
>
> you have a clear option
I appreciate that you over-estimate what’s clear to me. 😉
> [@sam](#):
>
> Set the Discourse-Track-View header to 0
That’s almost clear.
OK, so I **do** need to set `HTTP_DISCOURSE_TRACK_VIEW=false` in the ENV passed to the container when it cranks up? And/Or I contrive to get the thing that’s hitting `/srv/status` to include a `Discourse-Track-View: 0` header? (or maybe it’s an `=`, but presumably I can figure that out).
EDIT: I think that did the trick (thanks, @Falco!) but I’ll report back tomorrow when I can tell for sure that it worked.
EDIT2: And GKE has not only health checks from k8s, but also health checks from the load balancer, so both of those need to be configured with the `Discourse-Track-View: 0` header.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [6 במרץ, 2020, 8:58pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/7 "2020-03-06T20:58:24Z")
The later. Assuming the thing has the feature to allow you to use custom headers 😅
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [6 במרץ, 2020, 11:10pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/8 "2020-03-06T23:10:08Z")
Yep, custom headers:
```plaintext
pods/probe/http-liveness.yaml
apiVersion: v1
kind: Pod
metadata:
labels: {...}
name: liveness-http
spec:
containers:
- name: ...
image: ...
readinessProbe:
httpGet:
path: /srv/status
port: 80
httpHeaders:
- name: Discourse-Track-View
value: "0"
periodSeconds: 3
livenessProbe:
httpGet:
# sent https://github.com/discourse/discourse/pull/9136
path: /srv/status?shutdown_ok=1
port: 80
httpHeaders:
- name: Discourse-Track-View
value: "0" # note the quotes, to avoid yaml making it an integer
initialDelaySeconds: 3
periodSeconds: 3
```
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [6 במרץ, 2020, 11:14pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/9 "2020-03-06T23:14:06Z")
Thanks, Kane! I really appreciate that. That’s what I’m trying, and it appears that it’s working, but I was waiting until tomorrow to see if it’s really working. 😉
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [8 במרץ, 2020, 2:26am UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/10 "2020-03-08T02:26:30Z")
> [@sam](#):
>
> They are counted and if you don’t want them counted you have a clear option you can use
I think by default @sam this route should _not_ be counted, I see no value in ever counting “pageviews” to this route.. do you?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [9 במרץ, 2020, 6:56am UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/11 "2020-03-09T06:56:29Z")
Sure we can add a bypass here it is just a bit tricky to test and we don’t have a clean pattern for bypassing.
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [23 במרץ, 2020, 7:57pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/13 "2020-03-23T19:57:37Z")
The responses should already be excluded from view tracking because the responses are `text/plain` and not `text/html`.
What’s the exact path the GoogleHC was hitting? If it was hitting `/`, that’s your problem; and the view counting was correct.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [23 במרץ, 2020, 8:54pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/14 "2020-03-23T20:54:02Z")
Hmm. I think that I added the above-described headers to both K8s and the load balancer and that they’re both hitting `/srv/status` I’m pretty sure. and I’m still getting exactly 12,960 hits a day most days. I’ll give it another look shortly.
Thanks for your help!
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [2 באפריל, 2020, 5:32pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/15 "2020-04-02T17:32:41Z")
> [@riking](#):
>
> If it was hitting `/` , that’s your problem; and the view counting was correct.
That was the problem. I thought that I’d configured both the load balancer and the k8s health check to hit /srv/status, but one of them was hitting `/`. So when I finally figured out how to fix that, the problem went away.
The great news is that adding the `Discourse-Track-View` is not necessary!
Now to get all of my uptime robot health checks using `/srv/status`.
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [2 באפריל, 2020, 5:34pm UTC](https://meta.discourse.org/t/do-hits-to-srv-status-count-as-crawlers/143431/16 "2020-04-02T17:34:53Z")