# Discourse Disorder
**URL:** https://meta.discourse.org/t/discourse-disorder/246587
**Category:** Plugin
**Tags:** deprecated
**Created:** [December 16, 2022, 9:10pm UTC](https://meta.discourse.org/t/discourse-disorder/246587 "2022-12-16T21:10:01Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [December 16, 2022, 9:10pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/1 "2022-12-16T21:10:01Z")
> âš This feature is now part of [Discourse AI](https://meta.discourse.org/t/discourse-ai/259214). This plugin is deprecated.
| | | |
| --- | --- | --- |
| :discourse2: | **Summary** | **Disorder** helps moderators by automatically flagging potential toxic content on your Discourse forum. |
| đź› | **Repository Link** | [https://github.com/xfalcox/disorder](https://github.com/xfalcox/disorder) |
| đź“– | **Install Guide** | [How to install plugins in Discourse](https://meta.discourse.org/t/install-plugins-in-discourse/19157) |
> If you are using our official hosting service, please contact our support to register your interest for this plugin.
# Toxicity
As @hawk said in [Dealing with Toxicity in Online Communities](https://meta.discourse.org/t/dealing-with-toxicity-in-online-communities/248301), managing toxicity in your community is fundamental.
> [@Dealing with Toxicity in Online Communities](https://meta.discourse.org/t/dealing-with-toxicity-in-online-communities/248301/1):
>
> Occasionally though, cracks can start to appear and if they are not addressed quickly, a culture of toxicity can creep in. If that is allowed to gain momentum, the community becomes a hostile environment which marginalises people.
While Discourse ships out of the box with many different tools to help manage toxicity in your community, we are always researching ways to improve it further. In particular, I’ve been researching about possible applications of Machine Learning and AI in online forums. Which is now an experimental plugin, available to all communities.
## The plugin
Disorder leverages Artificial Intelligence and Machine Learning to help you moderate you community, making it easier for your moderation team to be on top of potentially problematic content and even optionally nudging your users to revise toxic posts before posting.
This is a first foray into using self-hosted ML models in Discourse, and while it’s a simple model it sets a pattern that can be reused to apply more complex models down the road.
### Features
#### Background Flagging
This is Disorder’s main mode of operation, as it’s completely transparent to your users, that will not be aware of any changes.
Whenever a new post (or chat message using [Discourse Chat](https://meta.discourse.org/t/discourse-chat/230881)) is created, it will be put in a classification queue asynchronously. In case that classification comes back above a configurable threshold, the post/chat message will be flagged so your moderation team is warned about it, and can make the final decision on the flag.
#### New post intervention
If you think that prevention is the best medicine, you may be interested in this more active option.
You can enable a synchronous classification of any new post, that if above a configurable threshold of toxicity, will trigger an intervention on the new post flow, asking the user to review and make amends to the message that may be outside of the boundaries set by your community rules.
This will only happen once, and after closing the modal the user will be able to post normally.
### How does it work?
This plugin integrates the open source models from Detoxify, using a remote API call model to allow admins to properly scale the inference rate to each community needs.
We provide a simple image that provides an thin HTTP API that Discourse will call to perform content classification, which can be ran either in the same server where you run Discourse, or in a different server altogether.
The Discourse plugin listens to new post / new chat messages events, and enqueue a classification job in the background queue. Results are stored in the database so you can extract reports, and we flag content using a separate bot user so we can track it’s flag accuracy over time.
### Options
First, the plugin ships working out of the box, so it’s not necessary to change any setttings right away. However, if you want to change the plugin behavior, there are a few knobs you can use.
We provide 3 different classification models that you can pick on the plugin options:
- **unbiased** (default): A model that tries to reduce the unintended model bias in toxicity classification
- **multilingual** : A model that can classify Italian, French, Russian, Portuguese, Spanish and Turkish.
- **original** : Most simple model.
You can also tweak if the plugin will:
- automatically flag
- enable sync intervention on toxic posts with warning (experimental)
- enable sync intervention on toxic posts (not recommended)
All the above only happen when the comment is classified to be above the thresholds for each classification type:
- toxicity
- severe\_toxicity
- identity\_attack
- insult
- threat
- sexual\_explicit
You can tweak each of the classification thresholds for automatic actions.
#### Classification Service
The plugin comes pre-configured working out of the box. For that, it’s contacting a service ran by Discourse (CDCK) to classify the user content. That [classifier API service is open-source](https://github.com/xfalcox/detoxify-api), and you can run your own copy of the service if necessary.
---
### Author:  [@Decorbuz](https://meta.discourse.org/u/Decorbuz)
#### Post date: [December 18, 2022, 4:39pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/2 "2022-12-18T16:39:38Z")
Just out of curiosity, what are the differences between “Disorder” and Discourse’s implementation of the Google Perspective API?
> [@Discourse Google Perspective API](https://meta.discourse.org/t/discourse-google-perspective-api/98733):
>
> discourse2Summary Google Perspective API is the official Google Perspective API plugin for Discoursehammer_and_wrenchRepository Link [https://github.com/discourse/discourse-perspective-api](https://github.com/discourse/discourse-perspective-api)open_bookInstall Guide [How to install plugins in Discourse](https://meta.discourse.org/t/install-plugins-in-discourse/19157)Features What is the Perspective API? From [the official site](https://www.perspectiveapi.com/), “Perspective is an API that makes it easier to host better conversations.The API uses machine learning models to score the perceived impact a comment might h…
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [December 18, 2022, 6:59pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/3 "2022-12-18T18:59:31Z")
Code wise, they are completely different plugins.
From a ten foot view cover they same need, but they are engineered differently:
- Disorder works with chat and posts, Perspective only with posts
- Perspective relies on a proprietary and third party API, with all the privacy, reliability and transparence implications of that.
- Disorder sets a pattern that allows the addition of new models easily, so we can evolve the service or even add brand new features
- Disorder self-hostable API gives flexibility and freedom from pay-per API call and rate limits.
- Disorder front-end surface is quite smaller, so it _should_ be more resilient across Discourse updates.
---
### Author:  [@fyodorovich](https://meta.discourse.org/u/fyodorovich)
#### Post date: [December 23, 2022, 2:42am UTC](https://meta.discourse.org/t/discourse-disorder/246587/4 "2022-12-23T02:42:40Z")
> [@Falco](#):
>
> If you are using our official hosting service, please contact our support to register your interest for this plugin.
Cool. Where and how do we do that?
---
### Author:  [@HAWK](https://meta.discourse.org/u/HAWK)
#### Post date: [December 23, 2022, 3:17am UTC](https://meta.discourse.org/t/discourse-disorder/246587/5 "2022-12-23T03:17:16Z")
Email [team@discourse.org](mailto:team@discourse.org) 🙂
---
### Author:  [@simon](https://meta.discourse.org/u/simon)
#### Post date: [December 27, 2022, 12:47am UTC](https://meta.discourse.org/t/discourse-disorder/246587/6 "2022-12-27T00:47:28Z")
> [@Falco](#):
>
> Disorder sets a pattern that allows the addition of new models easily, so we can evolve the service or even add brand new features
No need to reply here, but if you’re looking for suggestions about where to go next, an AI tag suggester based on a topic’s text could be useful. I’m imagining something similar to how Soundcloud suggest musical genre tags after they run an analysis on an upload. It’s useful for organizing user generated content on a busy site.
---
### Author:  [@kinetiksoft](https://meta.discourse.org/u/kinetiksoft)
#### Post date: [January 16, 2023, 10:30pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/8 "2023-01-16T22:30:46Z")
Do I understand correctly the disorder API instance should be launched to companion the plugin? There is a pre-filled setting **disorder inference service api endpoint** with [https://disorder-testing.demo-by-discourse.com](https://disorder-testing.demo-by-discourse.com) pre-set. Yet there is **disorder inference service api key** which is empty by default.
We are interested in giving this plugin a test as we face a lot of toxic behavior between users, which eventually gets resolved by flagging and help of Leaders, yet we would like to pro-actively prevent users from spreading negative posts if it’s possible, and this plugin seems to fit into such role.
Can we use any ready endpoint to give it a try? Fair warning, we have ~150k page views daily and it might clog up some unprepared servers.
We are standalone.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [January 16, 2023, 10:34pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/9 "2023-01-16T22:34:43Z")
While you _can_ run your own API server, the plugin comes pre-configured pointing to `https://disorder-testing.demo-by-discourse.com/` so it works out of the box.
Please do use this endpoint for your instance, as it’s provided exactly for your use case of self-hosted instances wanting to give this plugin a try. **In the default configuration** , all the API calls happen in the background, so the API being down won’t impact your site in any way, so it’s safe to use it.
The `api key` setting is optional, and only needed if your API server have it enabled. The public instance at `https://disorder-testing.demo-by-discourse.com/` doesn’t have it enabled.
---
### Author:  [@kinetiksoft](https://meta.discourse.org/u/kinetiksoft)
#### Post date: [January 16, 2023, 11:25pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/10 "2023-01-16T23:25:09Z")
Thank you! Sounds perfect and will give it a try in upcoming days ❤
---
### Author:  [@piffy](https://meta.discourse.org/u/piffy)
#### Post date: [January 17, 2023, 1:25am UTC](https://meta.discourse.org/t/discourse-disorder/246587/11 "2023-01-17T01:25:45Z")
> [@Falco](#):
>
> This is a first foray into using self-hosted ML models in Discourse
Are there other ML applications planned for the future?
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [January 22, 2023, 5:14am UTC](https://meta.discourse.org/t/discourse-disorder/246587/15 "2023-01-22T05:14:31Z")
I tried this for a week, and it was absurdly aggressive at flagging posts. I recommend using this only if you have a huge site without enough mods. Hope the AI gets better, but it’s just not there yet.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [January 22, 2023, 2:04pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/16 "2023-01-22T14:04:48Z")
This is great feedback! Would you be willing to share some debugging stats to help me understand exactly what went down?
Something like the result of
```sql
SELECT
pcf.value,
p.raw
FROM
post_custom_fields AS pcf
INNER JOIN
posts AS p ON p.id = pcf.post_id
WHERE
pcf.name = 'disorder'
```
here or in a PM would be immensely helpful.
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [January 24, 2023, 7:12pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/17 "2023-01-24T19:12:41Z")
Ahh yes, I forgot all about that! Here you go. There really weren’t that many, but they were just unnecessary and members and mods found them annoying. I also am unsure about it scanning DMs, I know there could be value there if someone is harassing someone via DM, but most of the time it’s just going to trigger people knowing that we’re looking at their DMs
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [January 24, 2023, 7:15pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/18 "2023-01-24T19:15:52Z")
Do you use chat? Were all the annoying flags in posts / PMs ?
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [January 24, 2023, 7:33pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/19 "2023-01-24T19:33:24Z")
We do use chat but I’m pretty sure all the flags were in posts and pms
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [January 26, 2023, 6:07pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/20 "2023-01-26T18:07:56Z")
> [@davidkingham](#):
>
> I tried this for a week, and it was absurdly aggressive at flagging posts. I recommend using this only if you have a huge site without enough mods. Hope the AI gets better, but it’s just not there yet.
First of all, I’m very grateful for both the feedback and the data you shared that allowed me to debug this further.
Now to my findings!
During this week, you had 1942 new posts from non-staff users. Quite an active community! However I would not say that the AI is " absurdly aggressive at flagging posts", as **only 7 posts were flagged**.
That said, of those 7, half are clearly false positives triggered by too low defaults thresholds, other half are trickier for AI to understand the context (calling your interlocutor a jerk vs telling a story about how someone was a jerk to you today while you were shopping) and one is, IMO, a correct hit.
If you are willing to give it another try, moving all the thresholds to 85 and moving to the `original` model may solve almost all trigger-happy flagging issues you had so far. I’ll add a site setting to allow skipping PMs as I can see how that can be annoying for some communities too.
---
### Author:  [@davidkingham](https://meta.discourse.org/u/davidkingham)
#### Post date: [January 27, 2023, 2:32pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/21 "2023-01-27T14:32:30Z")
Thanks Falco, I apologize for saying it was absurdly agressive. I had a lot of drama happening on the site already and the flagging just added to that and I was quite annoyed at the time.
I appreciate the suggestions and will give it another try. Question, what happens when you disable `disorder flag automatically`? Will I still be notified somehow if a post is deemed disorderly? This would be nice to test it out and figure out what settings work without having posts flagged.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [January 27, 2023, 2:56pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/22 "2023-01-27T14:56:22Z")
Without that setting it will runs the posts against the AI but won’t take any actions. You can leave it like that and then run that [Data Explorer](https://meta.discourse.org/t/32566?silent=true) query to do some analysis of the false positive/false negative rates.
There is also another setting that allows you to add groups to a skip list, where you could, for example, skip posts from TL4/3 from being classified. That may also help.
---
### Author:  [@kinetiksoft](https://meta.discourse.org/u/kinetiksoft)
#### Post date: [February 19, 2023, 10:25pm UTC](https://meta.discourse.org/t/discourse-disorder/246587/23 "2023-02-19T22:25:19Z")
Dear @Falco,
We started testing Disorder out. The overall feedback is positive - it really does detect inappropriate things, while flagging a lot of things which our community accepts. Due to nature of the forum where we test this plugin (Adult), the communication involves several aspects which trigger Disorder to flag many many posts. Your [SQL Query](https://meta.discourse.org/t/disorder-automatic-toxicity-detection-for-your-community/246587/16) really does help checking out which thresholds to adjust, but may I suggest adding those to Reviewable Scoring table for each flagged post?
> **This one**
>
> 
I don’t know if it’s possible for a plugin to introduce it’s own data to this view, but it would help staff a lot to understand which criteria to adjust to reduce false-positive results for us. The way I see it is adding dropdown with a breakdown per criteria triggered within this view. No need to include criteria equaling 0. Those which are above 0, should be present there, but only those which exceed the current config thresholds should be marked bold/red.
> **Disorder Scoring example**
>
> - Toxicity 65% \[1\]
> - Insult 73% \[2\]
> - Threat 12% \[3\]
> - Sexual explicit 2% \[4\]
If needed, I can provide you with SQL Query results. We are far from finishing reviewing Flag Queue..
We are using multilingual model and haven’t tried others. Decided it would be a good to start with considering we have some users who prefer posting using their original language.
* * *
1. exceeding, red font
2. exceeding, red font
3. normal, normal font
4. normal, normal font
---
### Author:  [@kinetiksoft](https://meta.discourse.org/u/kinetiksoft)
#### Post date: [March 1, 2023, 10:56am UTC](https://meta.discourse.org/t/discourse-disorder/246587/24 "2023-03-01T10:56:41Z")
Hi again,
Wanted to let you know that we get Errors in logs related to Disorder using “original” model. I just switched it back to multilingual to see if it will make difference.
> Job exception: undefined method `\>=’ for nil:NilClass @classification[label] \>= SiteSetting.send(“disorder\_flag\_threshold\_#{label}”) ^^
> **Details**
>
> /var/www/discourse/plugins/disorder/lib/classifier.rb:39:in `block in consider\_flagging’
>
> /var/www/discourse/plugins/disorder/lib/classifier.rb:38:in `filter’
>
> /var/www/discourse/plugins/disorder/lib/classifier.rb:38:in `consider\_flagging’
>
> /var/www/discourse/plugins/disorder/lib/classifier.rb:25:in `classify!’
>
> /var/www/discourse/plugins/disorder/app/jobs/regular/classify\_post.rb:14:in `execute’
>
> /var/www/discourse/app/jobs/base.rb:249:in `block (2 levels) in perform’
>
> rails\_multisite-4.0.1/lib/rails\_multisite/connection\_management.rb:80:in `with_connection' /var/www/discourse/app/jobs/base.rb:236:in `block in perform’
>
> /var/www/discourse/app/jobs/base.rb:232:in `each’
>
> /var/www/discourse/app/jobs/base.rb:232:in `perform’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:202:in `execute\_job’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:170:in `block (2 levels) in process’
>
> sidekiq-6.5.8/lib/sidekiq/middleware/chain.rb:177:in `block in invoke’
>
> /var/www/discourse/lib/sidekiq/pausable.rb:134:in `call’
>
> sidekiq-6.5.8/lib/sidekiq/middleware/chain.rb:179:in `block in invoke’
>
> sidekiq-6.5.8/lib/sidekiq/middleware/chain.rb:182:in `invoke’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:169:in `block in process’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch’
>
> sidekiq-6.5.8/lib/sidekiq/job\_retry.rb:113:in `local’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch’
>
> sidekiq-6.5.8/lib/sidekiq.rb:44:in `block in module:Sidekiq’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:263:in `stats’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch’
>
> sidekiq-6.5.8/lib/sidekiq/job\_logger.rb:13:in `call’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch’
>
> sidekiq-6.5.8/lib/sidekiq/job\_retry.rb:80:in `global’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:124:in `block in dispatch’
>
> sidekiq-6.5.8/lib/sidekiq/job\_logger.rb:39:in `prepare’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:123:in `dispatch’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:168:in `process’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:78:in `process\_one’
>
> sidekiq-6.5.8/lib/sidekiq/processor.rb:68:in `run’
>
> sidekiq-6.5.8/lib/sidekiq/component.rb:8:in `watchdog’
>
> sidekiq-6.5.8/lib/sidekiq/component.rb:17:in `block in safe\_thread’
> **Details 2**
>
> | hostname | |
> | --- | --- |
> | process\_id | 65460 |
> | application\_version | 2f8ad17aed81bbfa2fd20b6cc9210be92779bd74 |
> | current\_db | default |
> | current\_hostname | |
> | job | Jobs::ClassifyPost |
> | problem\_db | default |
> | time | 1:52 pm |
> | | |
> | opts | |
> | post\_id | 604063 |
> | current\_site\_id | default |
P.S. Yes, multilingual mode does not produce these errors. Unbiased model does not produce errors either
[Next page](https://meta.discourse.org/t/discourse-disorder/246587.md?page=2)