There are a few crawlers on our site. Is there any risk that they have access to the content? What is an « acceptable crawler load/risk » before I would have to undertake blocking procedures with which I have few, if any, expertise?

What can crawlers see?

HAWK (Hawk) February 4, 2024, 2:41am 6

1 Like

Topic		Replies	Views
How to protect myself from bots crawling my Discourse instance? Support	6	1576	January 17, 2022
Smarter handling of random crawler traffic Feature	2	3484	March 29, 2018
Controlling Web Crawlers For a Site Site Management how-to	9	2132	September 14, 2024
MegaIndex bot did about 4,000 pageviews on one day Community	40	4433	December 2, 2023
How to allow user-agent access to private discourse? Support	10	1993	November 25, 2018