ウェブクローラー

[JammyDodger]

こちらにも参照できるガイドがあります。

Controlling Web Crawlers For a Site

This guide explains how to manage web crawlers on your Discourse site. Required user level: Administrator Web crawlers can significantly impact your site’s performance by increasing pageviews and server load. When a site notices a spike in their pageviews it’s important to check how web crawlers fit into the mix. Checking for crawler activity To see if crawlers are affecting your site, navigate to the Consolidated Pageviews report (/admin/reports/consolid…