# How self-hosted here are dealing with bad crawlers?

**URL:** <https://meta.discourse.org/t/how-self-hosted-here-are-dealing-with-bad-crawlers/361280>\
**Category:** Community Building\
**Created:** [April 10, 2025, 4:02pm UTC](https://meta.discourse.org/t/how-self-hosted-here-are-dealing-with-bad-crawlers/361280 "2025-04-10T16:02:49Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![Jagster](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jagster/32/192154_2.png) [@Jagster](https://meta.discourse.org/u/Jagster)\
**Post date:** [April 11, 2025, 8:57am UTC](https://meta.discourse.org/t/how-self-hosted-here-are-dealing-with-bad-crawlers/361280/3 "2025-04-11T08:57:28Z")

</div>

Setting `Blocked crawler user agents` is the admin’s friend. The trash traffic isn’t such a big issue with Discourse because the load isn’t that heavy. But I’ve banned a handful of the worst ones because I really dislike their business model. Everyone is crying about how AI companies are stealing content, which they are actually doing, but SEO companies are much worse — and their bots are really greedy.

But I’m using geo-blocking too, because I can. There are at least half a dozen countries that are sources of knockers and other malicious actors. But if a forum is for a global audience, that isn’t possible, of course.

With my WordPress sites, the same thing is done using Nginx with the help of Varnish.

At the moment, the ratio of humans vs. bots is something like 50/50 in my forum.

---

_[View the full topic](https://meta.discourse.org/t/how-self-hosted-here-are-dealing-with-bad-crawlers/361280)._