An admin, spdegabrielle, reports that spiders are consuming the majority of their free Discourse instance’s traffic allocation, raising concerns about resource abuse. They ask if others face similar issues and seek mitigation suggestions. Jagster notes that while banning most bots helps, Discourse is more resilient than platforms like WordPress — and humorously admits humans are harder to catch. supermathie thanks the admin for their appreciation and confirms this is a growing concern across customers. They recommend checking the bot report and blocking aggressive crawlers using this guide as a first step, while noting that Discourse is developing better tools — though for now, unless the issue becomes drastic, there’s no need to worry. Jagster adds that the current UI for blocking bots is cumbersome, especially given the volume of SEO bots and content stealers, and wryly suggests a bulk-upload feature would be helpful.
I’m an admin on a free instance for open source kindly provided by the lovely people at Discourse.
I see part of being a good person is not abusing this kindness by using resources beyond the conditions of the free service.
Sadly, spiders have been taking the majority of our allocation of views and only seem to be getting worse. (Does anyone else suffer from this?)
Can anyone make any suggestions to help mitigate this issue?
Best regards
Stephen
You can ban most of bots. It helps. But Discourse doesn’t care when a WordPress would be on its knees.
How to catch humans… well, tell to all of us when you figure out it.
Thanks for the appreciation; with the increase in scraping going on this has been a concern for many of our customers.
You can check your bot report and block the most egregious crawlers from your site, that’d be a good start.
We’re looking at developing better tools for managing this, but for the next while unless it becomes a DRASTIC problem, don’t worry about it. 
The only issue with that is UI. Adding those Discourse way is kind of pain somewhere, because there is quite many of those useless SEO-bots and other content stealers. If we had some way to send list or something 