Has anyone seen the OpenAI web crawler GPTBot visit their site?

Yes, first time to use the crawler report too, and lo and behold. There it was.

My take is it appeared in August, and it’s the largest crawler of them all.

Here is an example of a 24 hour period and the kind of ratio

#1 ChatGPT 18K pageviews
#2 mj12bot 1.8K pageviews
…
#4 Google 1.7K pageviews

This discourse deployment was put into long_required specially to stop dead the crawler getting at the content, so it must only be hitting the login_required page to clock up those hits right?

Could it use a user?

I assume that is technically possible but not likely probable, and if so I would expect such a user to suddenly have a really high post read count.

Right now it looks to be close to 100K pageviews far in excess of the next highest which is approx. less than half.

chapgpt crawler is a monster

1 Like