How self-hosted here are dealing with bad crawlers?

Cloudflare has always been nice to me and I’ve never had to pay for anti-bot services. That and their newer stuff like anti-AI is great and what keeps me a customer and shill for them I guess. Don’t want AI scrapers stealing your data? Just use one of their managed rules (given it’s entirely possible using just normal robotstxt like I do on my site)

.. vs a generic managed one, way better..

Whether or not these startups actually listen to and respect the file is another story, but good on them for trying at least. None of my sites have issues with bots in the past and I’m still repeatedly happy with the ability to block common wordpress exploits directly there after reading my logs.

1 Like