User-agent: lfc-discourse-public-crawler

Contribute Fout
1

Good afternoon,

This is not so much a bug, as it is an attempt to figure out more information about what appears to be an aggressive AI scraper.

In my Cloudflare analytics dashboard, I see 60% of http requests coming from a user-agent with “Discourse” in the name, notably: lfc-discourse-public-crawler. The full user-agent is lfc-discourse-public-crawler/0.1 (+public Discourse API archival crawl). I’ve never seen this user-agent before.

Additional information:

  • Only accesses .json version of a topic.
  • It appears to be operating out of Singapore.
  • The IP leads back to speedypage.com.
  • Browser: unknown/others.
  • OS: unknown/others.

And it doesn’t appear affiliated with Discourse. But with a name like that, perhaps it’s good that you’re also aware of this.

3 likes
2

Hmm could be anything, but perhaps a fork of A basic Discourse archival tool?