FYI - The Cloudflare Blog - Pay per crawl

General
1
5 Likes
2

That would be only fair.

3

Since it seems Clouldflare has implemented some of these crawler restrictions, posting what happens on the SWI-Prolog site for reference.

This bar chart displays the Legacy Consolidated Pageviews from June 20th to July 3rd, 2025, broken down into logged-in users, anonymous users, and crawlers. (Captioned by AI)
This bar chart displays the Legacy Consolidated Pageviews from June 20th to July 3rd, 2025, broken down into logged-in users, anonymous users, and crawlers. (Captioned by AI)1096×303 8.46 KB

Same report just later in the same day.

A bar graph displays the legacy consolidated pageviews from June 24, 2025, to July 2, 2025, distinguishing between logged-in users, anonymous users, and crawlers. (Captioned by AI)
A bar graph displays the legacy consolidated pageviews from June 24, 2025, to July 2, 2025, distinguishing between logged-in users, anonymous users, and crawlers. (Captioned by AI)1087×302 8.4 KB

List of crawlers and count of page views as CSV for SWI-Prolog forum on July 03 2025 Note: Day has not ended so more pages could be viewed which would increase counts. This does show which crawlers are active and how much they are downloading.

web-crawlers-250703-213320.csv (4.4 KB)

1 Like