On a login-required site that I manage I’m getting ~80K crawlers per day.
I noticed that sometime in late September traffic jumped significantly and have wondered what the explanation was
I suppose I should go look at the NGINX logs and see if I can figure out what the heck is hitting the site 80K times a day.