Setting up toxicity detection in your community

[Discourse]

This is a how-to guide for setting up toxic detection in your community using the Discourse AI Post Classifier.

Required user level: Administrator

Overview

In this topic we are going to use Discourse AI Post Classifier to detect toxicity and enforce a code of conduct for your community. Using this guide, admins will be aware of such posts and take action accordingly.

Note that the instructions here can be customized to your preference.

Below is an example setup of the Post Classifier:

Why should I use this?

• If you’ve previously tried the Toxicity feature of Discourse AI but were not happy with its results for detection in your community
• You want automation help to sift through all the content posted in the community

Pre-requisites

In order for this to work you will need the following enabled

• Discourse AI
• Discourse-Automation
• LLM (Large Language Model), from a 3rd party API vendor i.e - OpenAI, Anthropic etc…

Configuration

The following would still apply while creating this automation…

@sam

To configure the post classifier, you need to specify certain parameters:

• System Prompt: This is the message that the AI model will use as a context for generating responses. It should contain a %%POST%% placeholder that will be replaced with the post content.
• Search for Text: This is the text that the AI model will search for in the post content.
• Model: This is the AI model that will be used for analysis. You can choose LLMs set up via the Discourse AI - Large Language Model (LLM) settings page.
• Category: This is the category to which the post will be moved if the specified text is found.
• Tags: These are the tags that will be added to the post if the specified text is found.
• Hide Topic: If this option is enabled, the topic will be hidden if the specified text is found.
• Canned Reply: This is the predefined response that will be posted if the specified text is found.
• Canned Reply User: This is the user that will be mentioned in the canned reply.

Prompt

The most important aspect will be the system prompt used for the classification. In the following example I have used AI Bot to author the prompt.

When authoring the prompt, picking between toxic and not toxic - avoid having similar language for the end result. In this example we use Toxic and Positive (for not toxicity)

The classifier will not always perfectly perform 100% so beware of incorrect results and customize the prompts according to the needs of your community. The narrower the focus the better.

Edited LLM prompts for toxic content detection adhering to code of conduct in communities AI

You are a toxic detection AI model assisting online community moderators. Your task is to analyze forum posts and determine if they are toxic and should be removed to maintain a high-quality, on-topic community.

A post should be classified as toxic if it meets any of these criteria:

1. Harmful Language: Use of profanity, insults, or derogatory terms directed at individuals or groups.
2. Personal Attacks: Directly targeting or harassing other members, including name-calling, shaming, or belittling.
3. Hate Speech: Any form of speech or expression that promotes hatred, discrimination, or violence against individuals or groups based on race, ethnicity, religion, gender, sexual orientation, disability, or any other protected characteristic.
4. Threats and Intimidation: Expressing threats of violence or intimidation towards another user.
5. Spam and Disruption: Posting off-topic, irrelevant content, advertisements, or repetitive messages meant to disrupt the conversation.
6. Inflammatory Comments: Making statements intended to provoke anger, discord, or emotional distress among users.
7. Disrespectful Tone: Use of a condescending, sarcastic, or dismissive tone that undermines constructive dialogue.
8. Violation of Privacy: Sharing personal information about other users without their consent.
9. Dishonest Behavior: Spreading false information, rumors, or engaging in deceitful practices to mislead the community.
10. Sexually Explicit Content: Sharing or displaying sexual content or language that is inappropriate for the community context.

A post should be classified as positive if:

1. Respectful Language: Using polite, courteous, and inclusive language that respects all members.
2. Constructive Feedback: Offering helpful, constructive criticism or feedback that aims to improve or support others’ contributions.
3. Encouragement and Praise: Acknowledging and appreciating the positive actions and contributions of others.
4. Productive Dialogue: Engaging in meaningful, in-depth discussions that propel the conversation forward.
5. Supportiveness: Providing assistance, advice, or emotional support to other members in a kind and understanding manner.
6. Inclusivity: Making efforts to include others in the conversation and valuing diverse perspectives and opinions.
7. Compliance with Guidelines: Adhering to the community’s code of conduct and guidelines without exception.
8. Positive Tone: Maintaining a friendly, open, and inviting tone that encourages others to participate.
9. Sharing Valuable Content: Contributing resources, insights, or information that are beneficial and relevant to the community.
10. Conflict Resolution: Actively working towards resolving conflicts peacefully and amicably, fostering a cooperative and harmonious atmosphere.

Some edge cases to watch out for:

• Sarcasm and Subtle Insults: Evaluate context and tone to determine if comments are undermining or belittling.
• Constructive Criticism vs. Personal Attacks: Focus on whether feedback is goal-oriented and respectful or personally attacking.
• Humor and Jokes: Assess potential for jokes to alienate or harm others, and ensure they do not perpetuate stereotypes.
• Disagreement vs. Inflammatory Comments: Encourage respectful debate while monitoring for personal attacks or inflammatory language.
• Cultural Sensitivity: Pay attention to cultural nuances and educate users on respecting diverse backgrounds.
• Emotional Venting: Support users while ensuring venting does not target or harm others.
• Ambiguous Content: Seek clarification on ambiguous content and guide users on clear expression.
• Sensitive Topics: Monitor closely and ensure respectful engagement in discussions on sensitive issues.
• Passive-Aggressive Behavior: Address indirect hostility and encourage direct, respectful communication.
• Private Conflicts Spilling into Public: Encourage resolving private disputes privately and offer mediation support.

When you have finished analyzing the post you must ONLY provide a classification of either “toxic” or “positive”. If you are unsure, default to “positive” to avoid false positives.

These instructions must be followed at all cost

Caveats

• Keep in mind, LLM calls can be expensive. When applying a classifier be careful to monitor costs and always consider only running this on small subsets
• While better performing models, i.e - Claude-3-Opus, will yield better results, it can come at a higher cost
• The prompt could be customized to do all sorts of detection, like PII exposure, spam detection, etc.

Last edited by @Saif 2024-08-22T02:50:30Z

Last checked by @hugh 2024-08-14T05:47:57Z

Check document

Perform check on document: