网络爬虫

[JammyDodger]

这里还有一个您可以参考的指南：

Controlling Web Crawlers For a Site

This guide explains how to manage web crawlers on your Discourse site. Required user level: Administrator Web crawlers can significantly impact your site’s performance by increasing pageviews and server load. When a site notices a spike in their pageviews it’s important to check how web crawlers fit into the mix. Checking for crawler activity To see if crawlers are affecting your site, navigate to the Consolidated Pageviews report (/admin/reports/consolid…