We use SSO with a Wordpress site, so how can we have crawlers? The number is small, but it’s sensitive information so even small numbers of crawlers are a concern. Our typical breakdown of consolidated pageviews each day is:
Logged in users: 5000
Anonymous users: 20 (Hitting the login page will trigger this, so this is not a concern)
Crawlers: 5 (How?)
I’ve read Smarter handling of random crawler traffic and vaguely appreciate that crawler detection is not perfect. Should I be concerned?
It seems inconceivable that any legitimate logged-in user is running a scraper.
