Auto-categorize topics using AI

תיעוד ניהול אתר
, ,
1

:bookmark: This guide explains how to automatically categorize topics using the AI triage feature in the Discourse AI plugin combined with Discourse Automation.

:person_raising_hand: Required user level: Administrator

Auto-categorize topics using AI

This automation uses an LLM to read topic content, decides whether the selected category is appropriate, and updates it if the AI Agent considers another category suits the topic better.

Prerequisites

Reference Agent configurations

  • System prompt:
You are an expert content classifier for this forum.

  Your task is to analyze posts and suggest the appropriate category based on content, images, and the list of categories available.

  Guidelines:
  - Be conservative - only categorize what you're confident about
  - Consider both content topic and post intent
  • Default language model: Select the LLM configured in the Prerequisites section.

    :spiral_notepad: Discourse hosted customers can select the CDCK Hosted Small LLM when configuring AI Agents.

  • Enabled tools: Read, Categories, Edit Categories
  • Forced tools: Read, Categories
  • Forced tools strategy: Apply to first reply only
  • Execution Mode: Default (Fixed Limits)
  • (Optional) Turn on the Require approval for moderation tools option
  • User: Create user

Automation configurations

  1. Navigate to Admin → Plugins → Automations.
  2. Click + Add Automation.
  3. Select the Triage posts using AI Agent script.
  4. Configure the settings from the When/What section as you see fit.
  5. Select the Agent defined for this automation in the AI Agent setting from the Script options section.
  6. Enable the Silent Mode setting.
  7. Click on the Update automation button.
  8. Enable the automation.

Script options
Script options2066×688 45.8 KB

Best practices

  • Start with a narrow scope: Apply the automation only to a specific category or to first posts by new users (trust level 0 or 1) before expanding it. This limits LLM costs and reduces the risk of widespread mis-categorizing.
  • Write clear category descriptions in your prompt: The more precise your descriptions of each category’s meaning, the more reliably the LLM will classify correctly.
  • Monitor for false positives and negatives: The LLM won’t be 100% accurate. Build a workflow for reviewing and correcting mis-categorized topics, and adjust your prompt over time.
  • Keep token limits sensible: The “Max Post Tokens” setting limits how much of a post is sent to the LLM. For most auto-categorizing tasks, the first few hundred tokens (the opening of the post) is sufficient.

Common issues and solutions

  • The appropriate category is not being applied by the automation
    Review the security settings of the category. If the category that should be applied is a private category, make sure the Read private tool option from the Agent settings is enabled.
  • The automation runs but doesn’t categorize topics
    Review the site’s logs to identify any errors associated with the Discourse AI or Discourse Automation plugins.

Additional resources