There are a few crawlers on our site. Is there any risk that they have access to the content? What is an « acceptable crawler load/risk » before I would have to undertake blocking procedures with which I have few, if any, expertise?

מה יכולים crawlers לראות?

HAWK (Hawk) 4 בפברואר,‏ 2024,‏ 2:41am 6

לייק 1

נושא		תגובות	צפיות
How to protect myself from bots crawling my Discourse instance? Support	6	1599	17 בינואר,‏ 2022
Controlling Web Crawlers For a Site Site Management how-to	10	2311	19 ביולי,‏ 2025
Smarter handling of random crawler traffic Feature	2	3496	29 במרץ,‏ 2018
MegaIndex bot did about 4,000 pageviews on one day Community	40	4496	2 בדצמבר,‏ 2023
How to allow user-agent access to private discourse? Support	10	2012	25 בנובמבר,‏ 2018