There are a few crawlers on our site. Is there any risk that they have access to the content? What is an « acceptable crawler load/risk » before I would have to undertake blocking procedures with which I have few, if any, expertise?

What can crawlers see?

HAWK (Hawk) February 4, 2024, 2:41am 6

1 Like

Topic		Replies	Views
Seeing anonymous user and crawler traffic, even though site is private Data & reporting	3	866	September 9, 2021
Controlling Web Crawlers For a Site Site Management how-to	9	1958	September 14, 2024
How to disable indexing by crawlers Support	17	5037	April 24, 2019
How to prevent community content from being used to train LLMs like ChatGPT? Community	71	3663	October 14, 2023
Site is accessible on the IP address when not following th install guide Feature	18	2002	June 30, 2020