FYI "Using GPT-4 for content moderation" - An OpenAI blog entry

[Lilly]

good idea here

Have AI identify and flag web crawlers

Suggestion Have AI identify web crawlers that are regularly crawling site but that are not generating visits. Flag them for review by admins. Additionally modify the Consolidated Page View report to break out crawlers into more detail crawlers that generate visits one time crawlers others There are many web crawlers visiting sites daily or repeatedly and for sites such as SWI-Prolog many run up the page views but then there is no benefit to the site for such. Most often these are search en…