New googlebot detection?

AI-generated summary

The discussion revolves around a reported spike in anonymous page views on forum.makerforums.info, initially attributed to Googlebot activity. mcdanlj noted that the traffic was linked to crawl-$IP.googlebot.com hostnames in Google’s 66.249/16 netblock, coinciding with Google’s announcement about updating Googlebot’s user agent strings. Concerns were raised about reporting accuracy—traffic was misclassified as ‘Anonymous’ instead of ‘Crawlers’—and potential indexing issues due to the new user agents. gerhard confirmed that crawler detection works correctly with the new user agents, suggesting the issue might lie in data reporting. mcdanlj later clarified that the observed user agents were not from Googlebot but from google-proxy-$IP.google.com, indicating real traffic via Google’s proxy, and apologized for the confusion, noting the discussion will serve as a reference for future inquiries.

We noticed a spike in anonymous (not bot) page views at forum.makerforums.info and discovered that they were associated with crawl-$IP.googlebot.com hostnames in the 66.249/16 netblock. I found that Google recently announced that they are going to change their user agent strings:

https://webmasters.googleblog.com/2019/10/updating-user-agent-of-googlebot.html

Two concerns, related:

  • Reporting (minor): The reporting was just wrong, since the graph didn’t show it as “Crawlers” activity but as “Anonymous”
  • Indexing (important): Does the present-the-whole-page turn-off-infinite-scrolling code work with the new user agents? (If not, we might just have been badly un-indexed…)

I just tested the crawler detection with the user agents mentioned in the blog post and everything looks fine. You can test it yourself by using a browser plugin of your choice, setting one of the user agents and visiting your site.
The reporting uses data from the RequestTracker which relies on the same crawler detection, so I’m not sure why you are seeing bots counted as anonymous traffic.

Aha! While I thought I saw the new user agents, I was wrong. Bad log search. The new Chrome version strings I saw came from google-proxy-$IP.google.com and were real traffic going through google proxy.

I’m very sorry, I should have dug deeper. But I guess now there is a record here that the new user agent strings are OK for the next confused person who comes to search! :smiling_face:

:blush: