Webcrawler

[JammyDodger]

There’s also a guide here you can refer to:

Controlling Web Crawlers For a Site

When a site notices a spike in their pageviews it’s important to check how web crawlers fit into the mix. How to see if your site is hit by crawlers Head to the Consolidated Pageviews report (/admin/reports/consolidated_page_views) on your admin dashboard. Here you will see a breakdown of pageview numbers from Logged in users, Anonymous Users, and Crawlers. A site where crawlers work normally: [image] A site where crawlers are out of control: [image] Which crawlers are hitting your site …