Sudden Explosion in Crawler Views in 2.0 beta 1

[Falco]

@RyanK

Did someting change

Yes!

Smarter handling of random crawler traffic

Following: https://github.com/discourse/discourse/commit/7b562d2f46c60df5323ac06731cf341d95d85027 We now have much improved crawler/bot detection. In the past we have seen many customers complain that pageviews in Google analytics are way off from pageviews in Discourse. This change much more accurately splits up the traffic so both reports are more inline. While analysing this problem I noticed that we are allowing a gigantic amount of crawling from many probably pointless sources. DotBot …