In the discussion titled ‘AI flagging too sensitive’, Shauny reports that the AI toxicity flagging in Discourse is overly sensitive—flagging harmless posts like calling a TV episode ‘silly’—even at 80% sensitivity, and asks for guidance on optimal settings. Falco acknowledges the issue and informs that a new feature is now available, allowing community managers to use large language models as customizable classifiers for content flagging, offering better control over sensitivity. Although no guide exists yet, Falco promises to notify Shauny when one is published. Later, Saif shares that a guide for spam detection has been published, noting that its prompts can be adapted via Enable AI Bot to detect sensitive content as well: Setting Up Spam Detection in Your Community. The discussion centers on improving AI moderation precision to balance community safety with minimizing false positives.
I can’t find any specific information about this.
When auto flagging toxic posts (something I want, because I want my community to be respectful and safe), it is being far too sensitive, flagged a post where someone called a tv episode “silly”.
It was set at 80. I’ve just increased it to 90. But there’s no documentation on what is the maximum (I imagine that’s 100) and what precisely I need to set this at so it doesn’t do super simple false positives but still protects my community?
Thanks!
Hey @Shauny, we have received reports that the current toxicity module provider in the Discourse AI plugin is indeed too sensitive for most communities.
To address that issue, we just made it possible to use the new large language models as classifiers for the means of flagging content, which will give community managers the means to customize the sensitivity of the flagging to the one appropriate for their communities.
We don’t have a guide for this brand new feature yet, as we are working on it, but I will ping you here when we have one to share.
We just published a guide on detecting spam. The prompts could be customized with Enable AI Bot help to look for sensitive content!