Suggestion
Have AI identify web crawlers that are regularly crawling site but that are not generating visits. Flag them for review by admins.
Additionally modify the Consolidated Page View report to break out crawlers into more detail
- crawlers that generate visits
- one time crawlers
- others
There are many web crawlers visiting sites daily or repeatedly and for sites such as SWI-Prolog many run up the page views but then there is no benefit to the site for such. Most often these are search engine sites, but if the search engine is not generating visits then they should be excluded from being allowed to crawl the site.
Yes I know there is no ideal way to stop a badly behaved web crawler but knocking down the number of unnecessary page views from such web crawlers does add up to real money and time in the long run.