FYI - The Cloudflare Blog - Pay per crawl

5 Likes

That would be only fair.

Since it seems Clouldflare has implemented some of these crawler restrictions, posting what happens on the SWI-Prolog site for reference.

Same report just later in the same day.

List of crawlers and count of page views as CSV for SWI-Prolog forum on July 03 2025 Note: Day has not ended so more pages could be viewed which would increase counts. This does show which crawlers are active and how much they are downloading.

web-crawlers-250703-213320.csv (4.4 KB)

1 Like