We noticed a spike in anonymous (not bot) page views at forum.makerforums.info and discovered that they were associated with crawl-$IP.googlebot.com hostnames in the 66.249/16 netblock. I found that Google recently announced that they are going to change their user agent strings:
Two concerns, related:
Reporting (minor): The reporting was just wrong, since the graph didn’t show it as “Crawlers” activity but as “Anonymous”
Indexing (important): Does the present-the-whole-page turn-off-infinite-scrolling code work with the new user agents? (If not, we might just have been badly un-indexed…)
I just tested the crawler detection with the user agents mentioned in the blog post and everything looks fine. You can test it yourself by using a browser plugin of your choice, setting one of the user agents and visiting your site.
The reporting uses data from the RequestTracker which relies on the same crawler detection, so I’m not sure why you are seeing bots counted as anonymous traffic.
Aha! While I thought I saw the new user agents, I was wrong. Bad log search. The new Chrome version strings I saw came from google-proxy-$IP.google.com and were real traffic going through google proxy.
I’m very sorry, I should have dug deeper. But I guess now there is a record here that the new user agent strings are OK for the next confused person who comes to search!
