For some reason, over the past week my Anonymous users statistics have exploded. According to Google Analytics, I’ve only had around 200 Sessions during this period.
No idea if it has anything to do with crawlers but I have the default “blacklisted crawler user agents” set to mauibot, Yandex, Daum & BLEXBox already.
I’m hoping someone can help explain:
Thanks
Have you looked at the origin IPs for this traffic?
Where would I find that data?
If you’ve got SSH access you can:
./launcher enter app
and then
cd /var/log/nginx
Which has access logs for the nginx instance within the Discourse container.
You can throw them at your log analyser of choice to look at where requests are coming from and their frequency.
It’s likely to be one or more crawlers and/or spam bots. How new is your community?
Thanks, I learned something new today 
Initially, some IP from Mexico looks like the culprit. I’ll dig through some more logs.
Thank you
Usually the crawler reports by user agent under Admin cover this just fine cc @Stephen.
From what I can see, there’s considerably high numbers of Mozilla crawlers.
So, you methodology allowed me to identify a rogue IP address that was spamming my server. I blacklisted that address 22 hours ago and my latest stats look like this under my Anonymous Report:
| Date | Count |
|---|---|
| March 6, 2019 | 28 |
| March 5, 2019 | 2665 |
| March 4, 2019 | 3076 |
| March 3, 2019 | 1172 |
| March 2, 2019 | 1448 |
| March 1, 2019 | 3179 |
Thanks again
Yeah, from the numbers @digitalstartup was querying (3979 vs 62) I didn’t point him in that direction because I assume the crawler list that goes into that report to be pretty up-to-date.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.