There are a few crawlers on our site. Is there any risk that they have access to the content? What is an « acceptable crawler load/risk » before I would have to undertake blocking procedures with which I have few, if any, expertise?

They can only crawl public sites, meaning there is no security breach. But yes, they can access public content.

When the load is so high it has negative impact and you should buy more CPU and/or RAM. Well, I don’t know how easily that can happend on Discourse because solution is different, but PHP-based WordPress is quite easy to put on its knees. But Discourse is serving static and lightweight content for bo…

Also note that you can easily block crawlers via your admin settings.

I would be so grateful to know how…

Controlling Web Crawlers For a Site

Hopefully that isn’t just editing robots.txt because it works only with good behaving ones. There is actually only one effective but a bit more difficult solution: reverse proxy.

That approach is effective – we use it ourselves and recommend it to those on our hosting.

Should I read that Discourse is using filtering?

I’m not sure what you’re asking. We don’t block anything by default but we provide admins with the tools to be selective.

So you are trusting bots will a) read robots.txt and b) follow rules. Well, bad behaving ones don’t do either of those. And we are back where we started: if bots are anykind issue reverse proxy is the best solution. Thanks. That I would like to know.

ماذا يمكن للزواحف رؤيته؟

مجتمع البيانات والتقارير

HAWK (Hawk) 4 فبراير 2024، 2:41ص 6

التحكم في زواحف الويب لموقع

الموضوع		الردود	مرات العرض
How to protect myself from bots crawling my Discourse instance? Support	4	1642	21 يناير 2019
Controlling Web Crawlers For a Site Site Management how-to	10	2590	19 يوليو 2025
Smarter handling of random crawler traffic Feature	1	3556	29 مارس 2018
MegaIndex bot did about 4,000 pageviews on one day Community Building	40	4708	2 ديسمبر 2023
How to allow user-agent access to private discourse? Support	8	2071	15 نوفمبر 2015

ماذا يمكن للزواحف رؤيته؟

الموضوعات ذات الصلة