I’m an admin on a free instance for open source kindly provided by the lovely people at Discourse. I see part of being a good person is not abusing this kindness by using resources beyond the conditions of the free service. Sadly, spiders have been taking the majority of our allocation of views and only seem to be getting worse. (Does anyone else suffer from this?) Can anyone make any suggestions to help mitigate this issue? Best regards Stephen

Thanks for the appreciation; with the increase in scraping going on this has been a concern for many of our customers. You can check your bot report and block the most egregious crawlers from your site, that’d be a good start. We’re looking at developing better tools for managing this, but for the next while unless it becomes a DRASTIC problem, don’t worry about it. :heart:

The only issue with that is UI. Adding those Discourse way is kind of pain somewhere, because there is quite many of those useless SEO-bots and other content stealers. If we had some way to send list or something :smirking_face:

Spiders are making up the majority of our traffic!

Product Support

Jagster (Jakke Flemming) August 14, 2024, 1:01pm 2

You can ban most of bots. It helps. But Discourse doesn’t care when a WordPress would be on its knees.

How to catch humans… well, tell to all of us when you figure out it.

Topic		Replies	Views
Too many Crawlers, is that a problem? Data & reporting	6	2676	June 25, 2020
MegaIndex bot did about 4,000 pageviews on one day Community Building	40	5112	December 2, 2023
How to protect myself from bots crawling my Discourse instance? Product Support	4	1739	January 21, 2019
Why semrushbot and ahrefsbot are blocked by default? Product Support	10	4777	December 3, 2020
Smarter handling of random crawler traffic Feature	1	3680	March 29, 2018

Spiders are making up the majority of our traffic!

Related topics