Discourse AI - AI triage

Just following up here to note that AI triage no longer needs the following thanks to @Roman_Rizzi work on improving the prompts

3 Likes

hey all! This is more of a kudos comment to becauseI wanted to first thank you all for supplying the information provided here in addition to the posts that I had perused regarding this feature.

It provided some much-needed context into how this work has been implemented. I am currently studying the measurement and identification of online Toxicity models in enterprise, social media, and especially smaller or growing communities and I have a lot of questions but nothing huge to report as of yet.

2 Likes

Quick follow-up - does it mean that the existing prompts that were already running should be updated?

2 Likes

Yes, existing prompts should be updated now to reflect the changes

2 Likes

AI Triage is fantastic! Iā€™m looking forward how far we can go with it from classical ā€˜bag of wordsā€ algorithm. Post classifier is a great feature. I wonder if it there are plans to allow to use it with post approval queue. ( approve post count, approve unless allowed groups).

So basically use a classifier not only to add to queue for review but to automatically approve or reject from the queue. (I donā€™t mean handling flags only approvals)

I see AI post triaging and post review queue as complementary.

4 Likes

So to add tags or to change categories, there are currently the following options (correct me if I am wrong)

  • the user uses the AI helper to suggest tags and category from within the composer based on nearest neighbor embedding vectors, comparing it with previous posts.
  • the administrator sets up a separate AI post classifier that runs the topic through an LLM for every possible tag and every possible category, which adds the tag or changes the category when the LLM output says so. It will not add arbitrary tags, each classifier will only add a specific tag yes or no. This is not only cumbersome to set up and maintain, it is also expensive because you need to run them all.

What I would like to have is a classifier that takes an instruction like

You are a tagging assistant for a forum. The forum is about X, Y and Z.

Your task is to tag forum posts with one or more of the following tags:

  • one tag that specifies whether the post is selling for something or looking for something. Tags: for_sale, wanted, advice
  • one tag that specifies the kind of item the post is about: cars, services, tickets, kids_stuff, home_and_garden, other.

If you come across a message that is NSFW, tag it NSFW.

You can also set actions in order to moderate a post.
If you come across a message that is on topic for this forum but might be seen as offensive, set the action to ā€˜flagā€™.
If you come across a message that is off topic, set the action to ā€˜hideā€™

This can then be accompanied by the great new OpenAI structured outputs option to return a json structure that specifies the tags to be added or the action to be taken, which will then be processed by the classifier.

Is anything like this on the roadmap and/or is a PR welcome? Or would this be plugin territory?

6 Likes

I think the safest way to go about this would be to define a brand new automation for this. It is structurally very different to the current one.

I think we would be open to a well tested PR to Discourse AI.

Keep in mind Completion Prompt in AI does not support structured outputs at the moment, this concept would need to be added and implemented for OpenAI models, we have a ton of models, many do not have native support, I would be open for a PR just to provide OpenAI support for starters as long as it is not confusing to end users and we allow them to select a model not supporting the feature.

It is a reasonably big change though, but if you are feeling braveā€¦ go for it.

Would recommend going about it in multiple PRs starting from giving structured output support to completion prompts (tested)

8 Likes

In the process of creating an ā€˜Auto-tag postsā€™ automation, however I am not able to select a tag that has been added to a tag group. Please let me know if there are any workarounds for this. I prefer to keep the existing tag group. Thanks!

1 Like

Hey everyone,

Do I understand this correctly? If I have 50 categories I am going to use AI triage to auto categorize topics into those categories I will need 50 unique automations to accomplish?

Is the only time a call will be made to LLM is when any and all new topics are created for that initial post? Seems the AI usage is not excessive for this scenario.

I think I read that someone was considering another approach for this but might be consider only idea stage I suppose.

2 Likes

I think the answer to my question is it depends on how granular you want to auto categorize.

Perhaps someone can answer when using automation to automatically categorize topics does the search input box being matched take priority over using AI?

In other words if I set Banana as the search term in automation and a user creates a topic using Banana within the post, Match, No AI called. Categorize as banana.

If the user creates topic which references yellow fruit monkeys eat, No Match, AI prompt runs and may determine that description is a banana :banana:.

For this scenario Banana would also be an existing category.

1 Like

some of this ties back to what @RGJ talked about and some ideas I have been kicking around with @Falco .

I am kind of opening up to allowing a persona to act as an agent, then you could let the agent loose to do various things for you via tools, like categorize topics and so on.

The mechanics here are all in the air, but we are thinking about this.

50 automations is clearly a terrible solution.

6 Likes

Hello

Can AI Triage perform searches? I intend to use a prompt that avoids topics with similar titles if the creation date of the other topic is less than one day old. Is this possible with AI Triage?

Not at the moment, it sounds like you are describing ā€œagentā€ like behaviors. This is certainly something we are thinking about!

3 Likes

Wow, this works like a charm!
Iā€™m seeking advice: what should I put as the cap on the number of tokens? This is for a low-traffic forum.

Copying this over here for relevancy