Continuing the discussion from Discourse Etiquette: take actions when users intend to post inappropriate remark:
Google’s Perspective API is an experimental machine learning project, trained by moderation data from New York Times.
It’s designed to analyse online comments and highlight anything toxic.
I built an app that loads all posts from a Discourse forum backup and processes them using the Perspective API (instructions in README.md):
The results are stored in a DB for analysis.
When run on my forum, the “most toxic posters” list corresponded well with members our moderators are keeping an eye on. Aside from “toxicity” the measures of “inflammatory”, “attack on author”, “attack on commenter” were also reasonably accurate - at least enough to highlight the most problematic behaviour on the forum.
There were one or two anomalies. For example, one of our “most toxic” members turned out to be a wonderfully kind chap who was one of our forum’s earliest joiners and freely donated money to help with our running costs. Why so toxic? It turns out that he signs off posts with his nickname, “Dick,” and as far as the Perspective API is concerned, that’s rather uncivil.
Please give my app a try on your forum and let me know how you get on. I’d be grateful for any PRs and/or feedback on the code as I’m new to Postgres, Akka-HTTP, Doobie, Docker and Spray-JSON. Cheers!