# Seeing anonymous user and crawler traffic, even though site is private

**URL:** https://meta.discourse.org/t/seeing-anonymous-user-and-crawler-traffic-even-though-site-is-private/181051
**Category:** Data & reporting
**Created:** [February 24, 2021, 9:30pm UTC](https://meta.discourse.org/t/seeing-anonymous-user-and-crawler-traffic-even-though-site-is-private/181051 "2021-02-24T21:30:10Z")
**Posts on this page:** 1
**Showing post:** 2

<div class="post-metadata">

### Author: ![awesomerobot](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/awesomerobot/32/142900_2.png) [@awesomerobot](https://meta.discourse.org/u/awesomerobot)
#### Post date: [February 24, 2021, 10:41pm UTC](https://meta.discourse.org/t/seeing-anonymous-user-and-crawler-traffic-even-though-site-is-private/181051/2 "2021-02-24T22:41:37Z")

</div>

There shouldn’t be anything additional you need to do… that crawler traffic is likely from crawlers hitting `community.yoursite.com/login`. If you check `community.example.com/admin/reports/web_crawlers` you can see how often specific crawlers are hitting your site.

There are a couple things you can do to reduce the crawler traffic…

- Try disallowing `/login` from crawlers within robots.txt (`community.example.com/admin/customize/robots`)… you’d probably see some crawler traffic drop (though probably not completely because there are crawlers out there not obeying robots.txt)

- Take a look at the worst offenders from `/admin/reports/web_crawlers` and add their user-agents to the `blocked crawler user agents` site setting

---

_[View the full topic](https://meta.discourse.org/t/seeing-anonymous-user-and-crawler-traffic-even-though-site-is-private/181051)._