This is not so much a bug, as it is an attempt to figure out more information about what appears to be an aggressive AI scraper.
In my Cloudflare analytics dashboard, I see 60% of http requests coming from a user-agent with “Discourse” in the name, notably: lfc-discourse-public-crawler. The full user-agent is lfc-discourse-public-crawler/0.1 (+public Discourse API archival crawl). I’ve never seen this user-agent before.
Additional information:
Only accesses .json version of a topic.
It appears to be operating out of Singapore.
The IP leads back to speedypage.com.
Browser: unknown/others.
OS: unknown/others.
And it doesn’t appear affiliated with Discourse. But with a name like that, perhaps it’s good that you’re also aware of this.
Thank you @awesomerobot. This could indeed be the tool used here. Since I’m not archiving anything with such a tool, someone else seems to be making (or trying to make) a copy of all my content.
As a measure to try and stop this, I’ve blocked the user-agent via robots.txt. And I’ve also blocked any and all requests coming from Singapore. Will re-evaluate in seven days.
robots.txt is only a pollite suggestion. It doesn’t stop anything per se. But if a user agent follows those rules, it tells that bot isn’t awful malicious.
