# Pre-emptively warning a contributor about the toxicity of their post **URL:** \ **Category:** Feature\ **Created:** [2017年四月3日 09:04 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357 "2017-04-03T09:04:40Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![deevolution](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/deevolution/32/70206_2.png) [@deevolution](https://meta.discourse.org/u/deevolution)\ **Post date:** [2017年四月3日 09:04 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/1 "2017-04-03T09:04:40Z")
I was perusing through the Discourse feature and moderation lists while thinking about various enhancements that could be made to the platform. I stumbled across [this post](https://meta.discourse.org/t/measures-taken-against-bad-behaviour-in-league-of-legends/26084/3) that was focused on how implementing various moderation features to the League of Legends chat dropped the number of offenders down 40-50% and generally improved the chat space. This got me thinking - could you use machine learning to proactively moderate a post? As a user is typing, their text could be analyzed for toxicity, and a warning might let them know that what they are typing may be flagged after they post, perhaps helping them to re-word their post. Has a feature like this been proposed already? Is machine learning being used anywhere else on Discourse? ---
**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\ **Post date:** [2017年四月3日 13:50 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/2 "2017-04-03T13:50:06Z")
I know @angus has been playing around in this exact area (a machine learning plugin), but nothing is released quite yet. ---
**Author:** ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)\ **Post date:** [2017年四月3日 15:44 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/3 "2017-04-03T15:44:24Z")
It is unlikely to work any better than a simple word blacklist. There are a billion ways to be rude, many of which involve completely innocuous language. See also > **[Google's new product identifies whether a comment could be perceived as...](https://bbs.boingboing.net/t/googles-new-product-identifies-whether-a-comment-could-be-perceived-as-toxic-to-a-discussion/95703)** > > Originally published at: http://boingboing.net/2017/02/23/googles-new-product-identif.html … Read the comments and examples… ---
**Author:** ![angus](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/angus/32/341715_2.png) [@angus](https://meta.discourse.org/u/angus)\ **Post date:** [2017年四月4日 17:46 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/4 "2017-04-04T17:46:41Z")
Yes, I am working on a machine learning plugin that will provide the infrastructure if you want to run a model to detect this kind of thing. If you want to take a look, the unfinished plugin is here: [https://github.com/angusmcleod/discourse-machine-learning](https://github.com/angusmcleod/discourse-machine-learning) But as Jeff points out, I’m not sure there are any decent machine learning models for this use case that would work better than a simple word blacklist. That said, here’s a model that’s in the ballpark: > **[Implementing a CNN for Text Classification in TensorFlow](https://dennybritz.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/)** > > Denny's Blog ---
**Author:** ![deevolution](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/deevolution/32/70206_2.png) [@deevolution](https://meta.discourse.org/u/deevolution)\ **Post date:** [2017年四月4日 22:07 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/5 "2017-04-04T22:07:43Z")
I’m going to request access to Google’s perspectiveapi and see what I can get out of it. ---
**Author:** ![tophee](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/tophee/32/73406_2.png) [@tophee](https://meta.discourse.org/u/tophee)\ **Post date:** [2017年四月4日 22:26 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/6 "2017-04-04T22:26:45Z")
I think there is huge potential in a Clippy 2.0 plugin for improving the quality of online discussions 😋 ![](https://global.discourse-cdn.com/meta/original/3X/0/c/0c3b7123656a49a905836283f3c58a2e60ac3d7d.jpg) ---
**Author:** ![deevolution](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/deevolution/32/70206_2.png) [@deevolution](https://meta.discourse.org/u/deevolution)\ **Post date:** [2017年四月4日 22:50 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/7 "2017-04-04T22:50:15Z")
Except this plugin won’t provide you with an option, it will forcefully do it for you. Or it will just employ mechanical turk workers to check your every post for threats against trump! 😱 ---
**Author:** ![rimian](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rimian/32/120658_2.png) [@rimian](https://meta.discourse.org/u/rimian)\ **Post date:** [2017年四月4日 23:27 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/8 "2017-04-04T23:27:21Z")
Take a look at this! [https://www.perspectiveapi.com/](https://www.perspectiveapi.com/) “Perspective is an API that makes it easier to host better conversations. The API uses machine learning models to score the perceived impact a comment might have on a conversation. Developers and publishers can use this score to give realtime feedback to commenters or help moderators do their job, or allow readers to more easily find relevant information, as illustrated in two experiments below. We’ll be releasing more machine learning models later in the year, but our first model identifies whether a comment could be perceived as “toxic” to a discussion." ---
**Author:** ![codinghorror](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/codinghorror/32/110067_2.png) [@codinghorror](https://meta.discourse.org/u/codinghorror)\ **Post date:** [2017年四月5日 01:39 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/9 "2017-04-05T01:39:53Z")
Yes that was already covered in my link above. ---
**Author:** ![erlend\_sh](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/erlend_sh/32/119475_2.png) [@erlend\_sh](https://meta.discourse.org/u/erlend_sh)\ **Post date:** [2017年四月5日 20:36 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/10 "2017-04-05T20:36:04Z")
I applied for an invite to Perspective API and they shared some links that aren’t very easily accessible to the public yet, but they are public knowledge none the less: - [https://github.com/conversationai/perspectiveapi/blob/master/api\_reference.md](https://github.com/conversationai/perspectiveapi/blob/master/api_reference.md) - [https://github.com/conversationai/perspectiveapi/blob/master/quickstart.md](https://github.com/conversationai/perspectiveapi/blob/master/quickstart.md) - [Redirecting to Google Groups](https://groups.google.com/forum/#!forum/perspective-announce) - [Conversation-AI - Related Tools & Research](https://docs.google.com/forms/d/e/1FAIpQLSdiU2Wdpq2NKy_A_FrntQwG1j6L3Jgo9oKkd-8xOsrqg0-gdw/viewform?c=0&w=1) @deevolution and @angus definitely keep us in the loop on your progress! It only took me a couple of days to get whitelisted for API access. If you’ve waited more than a week, let me know and I could provide a reference on behalf of Discourse. ---
**Author:** ![deevolution](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/deevolution/32/70206_2.png) [@deevolution](https://meta.discourse.org/u/deevolution)\ **Post date:** [2017年四月5日 20:58 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/11 "2017-04-05T20:58:16Z")
jealous! I applied a few days ago, but haven’t heard anything back yet ☹ I’ll comb through these links tonight. Thanks @erlend_sh! ---
**Author:** ![deevolution](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/deevolution/32/70206_2.png) [@deevolution](https://meta.discourse.org/u/deevolution)\ **Post date:** [2017年四月6日 18:02 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/12 "2017-04-06T18:02:03Z")
Just saw this post on hackernews: [Https://blog.openai.com/unsupervised-sentiment-neuron/](Https://blog.openai.com/unsupervised-sentiment-neuron/) Their neural network on github: [https://github.com/openai/generating-reviews-discovering-sentiment](https://github.com/openai/generating-reviews-discovering-sentiment) Looks like they’ve trained a high accuracy sentiment nueral network model. Maybe something can be pulled from this. ---
**Author:** ![erlend\_sh](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/erlend_sh/32/119475_2.png) [@erlend\_sh](https://meta.discourse.org/u/erlend_sh)\ **Post date:** [2017年四月6日 18:05 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/13 "2017-04-06T18:05:45Z")
They have an open Discourse forum, so you might be able to pitch Discourse to them as a suitable test subject for that library 😉 [https://discuss.openai.com/](https://discuss.openai.com/) ---
**Author:** ![bookaddict](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@bookaddict](https://meta.discourse.org/u/bookaddict)\ **Post date:** [2017年四月7日 17:01 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/14 "2017-04-07T17:01:29Z")
This is a very interesting idea. I’ve been reading the Meta for some time since our community moved to Discourse. One of the issues the moderators run into is with a handful of people who are often OK contributors but can’t seem to stop themselves from adding a toxic comment on occasion. They are typically already at least level 2, some have even made it to 3, but they just have that tendency to type first and think later. It keeps some from ever being considered for level 3 and keeps the moderators busy watching this small group. I finally signed up today since I keep reading but never post or comment, so I thought it was time to at least be able to get involved in discussion. ---
**Author:** ![deevolution](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/deevolution/32/70206_2.png) [@deevolution](https://meta.discourse.org/u/deevolution)\ **Post date:** [2017年四月7日 18:32 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/15 "2017-04-07T18:32:25Z")
Awesome! That’s good to hear an actual customer use case. I definitely want to move ahead with development soon (likely around may/june after I finish up thesis). Thanks for chiming in. When we have a working version, I think it would help a lot if we use your site for testing and feedback. I would definitely be interested in getting some help/guidance on the project from @angus. ---
**Author:** ![angus](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/angus/32/341715_2.png) [@angus](https://meta.discourse.org/u/angus)\ **Post date:** [2017年四月7日 18:57 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/16 "2017-04-07T18:57:01Z")
Would be happy to help out. I don’t have a lot of time right now, but hopefully will have a few spare moments here and there. The plugin I’m building is purely infrastructure to support training, testing and using any model with any discourse instance. It’s meant to be model / framework agnostic (albeit it’s quite biased towards tensorflow atm). The biggest remaining ‘to do’ for an ‘mvp’ is to implement a docker network to properly retrieve the output from the container running the model. Currently the output is being parsed from a log stream. Although, when I talked with @sam about this, he suggested putting it all in the same container, so perhaps a network is not necessary. This is what the UI looks like: ![](https://global.discourse-cdn.com/meta/original/3X/b/0/b033293c6a0cba84498941671719fab91e7c1c5c.png) ---
**Author:** ![deevolution](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/deevolution/32/70206_2.png) [@deevolution](https://meta.discourse.org/u/deevolution)\ **Post date:** [2017年四月10日 22:30 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/17 "2017-04-10T22:30:05Z")
I just [posted](https://discuss.openai.com/t/request-for-research-use-discourse-as-a-training-set/1365) under their Request for Research topic about considering Discourse as a training set. ---
**Author:** ![downey](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/downey/32/166878_2.png) [@downey](https://meta.discourse.org/u/downey)\ **Post date:** [2017年四月17日 22:38 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/18 "2017-04-17T22:38:51Z")
Not to steer things too far from this excellent discussion of tools in progress, but I wonder if [Mozilla’s Coral Project](https://coralproject.net/) might be interested in maybe doing some work together, perhaps writing a plugin for their [Talk community commenting product](https://coralproject.net/products/talk.html)? They are already using Discourse and are obviously interested in this space: [https://community.coralproject.net/t/please-read-this-guide-to-running-comment-spaces-and-tell-us-what-you-think/649](https://community.coralproject.net/t/please-read-this-guide-to-running-comment-spaces-and-tell-us-what-you-think/649) ---
**Author:** ![erlend\_sh](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/erlend_sh/32/119475_2.png) [@erlend\_sh](https://meta.discourse.org/u/erlend_sh)\ **Post date:** [2017年九月27日 10:30 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/19 "2017-09-27T10:30:17Z")
Continued here: > [@Pre Emptive Striker plugin development log](https://meta.discourse.org/t/pre-emptive-striker-plugin-development-log/70845): > > Hey, all As suggested by @erlend_sh, I have created a development log for a plugin I’ve begun developing for Discourse. I have started work on building what I call the pre-emptive striker plugin (if anyone has a better suggestion for the name, throw it at me!). It was proposed at the beginning of the summer and I’m now officially developing the plugin. Link to original proposal is [here](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357). Basically, the plugin checks on the user as they are writing for toxicity using [Google’s perspective API.](https://www.perspectiveapi.com/#/) a… ---
**Author:** ![erlend\_sh](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/erlend_sh/32/119475_2.png) [@erlend\_sh](https://meta.discourse.org/u/erlend_sh)\ **Post date:** [2017年九月27日 10:30 UTC](https://meta.discourse.org/t/pre-emptively-warning-a-contributor-about-the-toxicity-of-their-post/60357/20 "2017-09-27T10:30:22Z")