Discourse AI - AI triage

[sam (Sam Saffron)]

AI triage is designed to enhance the management and moderation of forum posts by automating the process of classifying posts.

Please note that this feature requires both the discourse-automation and discourse-ai plugins to function.

Use Cases

1. Automated Post Categorization: AI triage can automatically categorize topics based on their content. This is particularly beneficial for large forums where manual categorization can be time-consuming. You can apply the rules to a subset of topics (first topic only or topics posted to particular categories)

2. Post Tagging: AI triage can assign specified tags to posts. This feature aids in the better organization of posts and makes searching and retrieving topics more efficient.

3. Automated Responses: AI triage can generate replies to posts using predefined responses. This is useful for addressing frequently asked questions or common queries, sending spam to junk categories, and more.

4. Topic Hiding: AI triage can hide topics based on certain criteria. This can be used for managing spam or inappropriate content.

5. NSFW/Toxicity/Spam detection

   • Setting up NSFW detection in your community
   • Setting up toxicity detection in your community
   • Setting up spam detection in your community

How It Works

AI triage uses AI models to analyze the content of the posts. Based on this analysis if the model returns specific text, it performs the specified actions. These actions include moving the post to a specified category, adding tags, replying with a predefined response, and hiding the topic.

Configuration

The system prompt field has been deprecated in favor of Agents. If you had an AI automation prior to this change, a new Agent with the associated system prompt will be automatically created.

To configure AI triage, you need to specify certain parameters:

• Agent: Agent used to triage. Be sure for it to reply with a single word you can use to trigger the actions below.
• Search for Text: If the following text appears in the LLM reply, apply the actions below.
• Max Post Tokens: The maximum number of tokens to scan using LLM triage.
• Stop Sequences: Instruct the model to halt token generation when arriving at one of these values.
• Max Output Tokens: When specified, sets an upper bound to the maximum number of tokens the model can generate.
• Include Personal Messages: Also scan and triage personal messages.
• Category: This is the category to which the post will be moved if the specified text is found.
• Tags: These are the tags that will be added to the post if the specified text is found.
• Hide Topic: If this option is enabled, the topic will be hidden if the specified text is found.
• Flag post: If this is enabled, the topic will be flagged for the review queue for moderators, staff and admins to take action.
• Flag type: The type of flag to apply. Options include:
  • Add post to review queue — sends to review queue only
  • Add post to review queue and hide post — sends to review queue and hides the post
  • Add post to review queue and delete post — sends to review queue and soft-deletes the post
  • Add post to review queue, delete post and silence user — sends to review queue, soft-deletes the post, and silences the user
  • Flag as spam and hide post — flags as spam and hides
  • Flag as spam, hide post and silence user — flags as spam, hides, and silences the user
• Reply User: This is the user that will be mentioned in the canned reply.
• Reply: This is the predefined response that will be posted if the specified text is found.
• Reply Agent: AI Agent to use for replies. Must have default LLM, will be prioritized over canned reply. Will conditionally reply unlike Discourse AI - AI triage using Agent.
• Reply as Whisper: Whether the AI’s response should appear as whisper.
• Notify author via PM: Send a personal message to the post author when their post is queued and deleted.
• PM sender: User who will send the PM (defaults to system).
• PM content: Optional custom message to send to the author.

Caveats

• LLM calls can be expensive. When applying a classifier be careful to monitor costs and always consider only running this on small subsets.
• The particular use case we tested (and run in production for a customer) is the classification of first topics of a user.
• We recommend you use AI bot to help author the agent’s system prompt.

[EricGT]

Being one one of the lucky ones to see this in action this is one feature for large sites that you should really understand and consider using.

As it works using an LLM based AI it does not always reach the correct conclusion but it gets so much right in the cases I have seen it used for a few days doing some of the work of a moderator that it definitely was a significant benefit.

Some of the early discussions with this are in the Lounge category on the OpenAI site. While access to that category can be had by anyone, the TL3 requirements must be meet and with OpenAI being a large site it does take quite a bit of effort to get to TL3 on the site.

For those with access here is the link

https://community.openai.com/t/lost-users-first-empirical-data/403082/95

---

Basically what the logic is doing to help the moderators for a specific problem is that we are seeing about 5% of the post from new users that think the forum is where they post questions to ChatGPT, clearly they are lost or perhaps a search result is providing an invalid link. The AI identifies such post, replies with prewritten text and changes the category and tags as needed, e.g.

Education is very important how can children who are disinterested in the...

---

For the case noted, here is what the “Discourse AI Post Classifier - Automation rule” is doing

Automated Post Categorization
The classifier is changing the category to ChatGPT as needed. As most new users will not select a category, the OpenaAI site currently defaults to the category API for new post which is incorrect in this case.

Post Tagging
The classifier is charging the tag(s) to lost-user in this case. The tag name was created by a TL3 user on the forum who was manually changing the tags.

Automated Responses
The classifier is replying with a prewritten post.

Topic Hiding
The topics are being unlisted as they were not of value to developers who use the site.

Reply User
System is being used as the creator of the reply.

Note: I would provide the Configuration for this but lack the access. Maybe @Sam can add the details, AFAIK it can be made public as nothing is confidential in the configuration. As it is for a specific site would not expect to find it in the public repository. If you understand this technology then it is not hard to guess the correct values, or close to them. The System Prompt did take some work and perhaps @sam can share some of the lessons learned, the knowledge of how to craft the prompt was of great value during the development phase.

System Prompt

Note: This is a version posted in the OpenAI Lounge (ref) pretty sure the final version is different but one example saves a lot of guessing.

You are bot that is triaging all first interactions a user has on the OpenAI developer forum.

Please only ever respond with “ok” or “bad”

Posts are on topic and ok if:

They relate to OpenAI APIs / Community / Plugin development / Documentation / Prompting
They are about developing or improving methods for prompting large language models
They relate to AI in a general way
They involve complex discussions or logical problems related to AI
Posts are bad if:

A user appears to be having a random conversation with ChatGPT
A user is off topic discussing an unrelated field
A user is prompting a large language model to generate text without a clear purpose
You have extreme difficulty understanding what they are about
Relates to an OpenAI topic BUT is clearly a conversation with a bot
A user is attempting to train or test the AI model through their post
A user is giving instructions or asking for responses in a non-discussion format
A user is posting content that is not conducive to meaningful discussion or learning
A user is posting hypothetical or speculative content without clear relation to OpenAI’s scope
A user is posting the entire post in a language that is not English
Please classify the following content surrounded by [[[]]]:

[[[
%%POST%%
]]]

---

FYI

If you are a moderator on a site using this and you want/need to see a list of unlisted post, remember that you can not use search. However you can navigate to such a list using the category then selecting the tags. Or you could also hand craft the url, e.g. https://community.openai.com/tags/c/chatgpt/19/lost-user

---

Yes it even works for languages other than English

Rend der Eingewöhnung von ihrem Vater bogustet. Schon Maries Geschwister...

However it does not work with images , it will not convert an image to text then run the check, e.g.

Solving of mathematics equation involving fraction with addition

Here is a recent false positive or at least I think so.

Looking for a developer for a Dungeon Master AI bot on Discord

---

For the details on false/true negative/positive - Classification: True vs. False and Positive vs. Negative

[EricGT]

A possible feature.

At least for moderators reviewing a post, convert the text to the language of the moderator when they view the topic. This is not implying just for a flag but for any viewing of the post.

---

At times when reviewing the actions of the Discourse AI Post Classifier there is a need to convert the text of the post to my language (English) to check if the AI took a correct action. Currently it is easier for me to paste the text into Google Translate.

[sam (Sam Saffron)]

Worth noting and related to this side quest.

@keegan is working on integrating AI helper into our popup menu. So in future in cases where there is a bunch of text in an unknown language you will be able to just highlight the text and hit translate.

We support that in the composer today, but once merged you will be able to highlight text in any Discourse post and get a quick LLM based translation.

[ncaming215 (nate c.)]

This … changes the game for me. Where can I send thank you cards.

[jordan-violet]

@sam

Post Tagging: The post classifier can assign specified tags to posts. This feature aids in the better organization of posts and makes searching and retrieving topics more efficient.

What is an example of how to do this by referencing the existing tags your forum has today? e.g. the post is analyzed by the LLM and the most likely/relevant tags are added to the post.

[sam (Sam Saffron)]

I love this question, can you expand on it please… how would you like to see this work?

• Would you apply “auto tagging” if the topic already has tags?
• Would you apply it on all first posts or just on first posts by particular groups? (eg: tl0 / tl1)
• Is this something you would prefer to run by hand on a subset of topics?

We are going to need to make some adjustments here… at the moment the classifier is binary, but the changes are reasonably easy as long as we know what the goal is?

[jordan-violet]

• Apply to all new topics in a category
• You could certainly have the option to vary by trust level. Like everything (and in this case, every community), it depends. In the case of my community, I’d likely keep it turned on for everyone but tl4 (or just everyone, if that configuration isn’t an option)
• I’d prefer it ran on all new topics in specified categories

The goal here, of course, is to never have to work on tagging topics on our own and to let AI do it entirely based on the initial topic being created and it looking at the existing tags we have now.

It could probably be improved even more if you had the ability to add descriptions to tags in Discourse, and to let it use those descriptions for additional context to add the right tags

[EricGT]

First off let me state that I am fully behind this ability and making it better. The following is just some facts of how it currently works (10/16/2023). Daily as a category moderator on OpenAI I review all of the new post, not all of the replies, and do see all of the AI false positives and negatives.

@jordan-violet

The goal here, of course, is to never have to work on tagging topics

As one who uses this feature, Discourse AI Post Classifier - Automation rule, be aware

• it is not 100% accurate
• currently takes manual action to undo if the AI did a false positive. Depending upon the change may need a category moderator or higher; specifically listing an unlisted topic.
• may miss some topics, false negative
• as many users who will receive the automated reply by the AI do not understand how to flag the AI reply if it is a false positive, they will have to be identified and changed manually. So will need the help of real users to catch these
• understand how to craft the prompt to get it to work as needed
• could potentially need fine tuning and/or agents to get close to a desired satisfaction level which could get expensive
• no way to track false positives and negatives for use with understanding how to change the prompt and/or possible fine tuning and/or agents

---

false positive - The AI should not have made a change and did, which is incorrect.
false negative - The AI should have made a change and did not, which is incorrect.
true positive - The AI made a change, which is correct.
true negative - The AI did not make a change, which is correct.

[sam (Sam Saffron)]

A good opening move here is trying out a prompt in creative mode:

Something like…

You are a Discourse auto tagger, you suggest a list of tags for topic.

The tags are:

tag1: description
tag2: description

Suggest up to 3 tags for the following topic:

TOPIC

Try this on a few of your topics, how well doe it do?

[EricGT]

@EricGT

At times when reviewing the actions of the Discourse AI Post Classifier there is a need to convert the text of the post to my language (English) to check if the AI took a correct action. Currently it is easier for me to paste the text into Google Translate.

@sam

@keegan is working on integrating AI helper into our popup menu. So in future in cases where there is a bunch of text in an unknown language you will be able to just highlight the text and hit translate.

Works great! Thanks

[EricGT]

Recent update (11/01/2023)

The user id for the bot is now gpt-4-triage, at least on the OpenAI forum.

This was done so that users can silence the bot as needed.

Example of the bot responding.

Write a 500 to 1000 words about wolfgang keller

[sam (Sam Saffron)]

Note, this is configurable you can have the automatic reply come from any user you wish

[ncaming215 (nate c.)]

So, following up after giving this a whirl.

It seems like this is in the early stages, and I’m excited to see where it goes, but @jordan-violet’s take is 100% what I’m looking to accomplish. I did try the test prompt that @sam suggested and it appeared smart enough to apply the correct tags in theory. However, it appears that the current script limits applicable tags to 7, and we have considerably more than that.

Short of it? Would be great if it would semantically apply existing tags to a new topic. In terms of forum management (even here on meta, I’ve noticed!), the consistent application of tags to better order a community and its content continues to be an uphill battle, and it’s pure chaos at scale. For those of us who rely on tags, this would be a boon, even if it’s 75% accurate.

I’ll keep my eyes on this one as it develops!

[aas]

This is great! Do you have any suggestions for debugging this? I just configured an automation that wasn’t triggered when expected. I don’t see any relevant logs.

[sam (Sam Saffron)]

Hmmm what ended up happening here, can you open a dedicated topic?

[aas]

IIRC, I didn’t set this option at first. When I did, it started working.

[loginerror (Maciej Kuźmicz)]

Quick question. Are there plans to display an audit log for those automations (maybe similarly to the webhook pattern that you already have)?

It would help ease the “what did you do” scary feeling that I’m getting after an automation is enabled

[sam (Sam Saffron)]

Ai log already stores all llm interactions. You can use that today via a data explorer query

[loginerror (Maciej Kuźmicz)]

@sam

all llm interactions

While this is true, it is rather tricky to link the log entries of llm_triage type to a post/topic that it affects, or to the action that was taken (or not) as a result.

Having an overview of these actions would be great