There are a few crawlers on our site. Is there any risk that they have access to the content? What is an « acceptable crawler load/risk » before I would have to undertake blocking procedures with which I have few, if any, expertise?

クローラーは何を見ることができますか？

HAWK (Hawk) 2024 年 2 月 4 日午前 2:41 6

「いいね！」 1

トピック		返信	表示
How to protect myself from bots crawling my Discourse instance? Support	6	1587	2022 年 1 月 17 日
Controlling Web Crawlers For a Site Site Management how-to	10	2268	2025 年 7 月 19 日
Smarter handling of random crawler traffic Feature	2	3494	2018 年 3 月 29 日
MegaIndex bot did about 4,000 pageviews on one day Community	40	4472	2023 年 12 月 2 日
How to allow user-agent access to private discourse? Support	10	2008	2018 年 11 月 25 日