# Run Discourse AI evals
**URL:** https://meta.discourse.org/t/run-discourse-ai-evals/390080
**Category:** Developer Guides
**Tags:** ai
**Created:** [December 1, 2025, 6:09pm UTC](https://meta.discourse.org/t/run-discourse-ai-evals/390080 "2025-12-01T18:09:55Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [December 1, 2025, 6:09pm UTC](https://meta.discourse.org/t/run-discourse-ai-evals/390080/1 "2025-12-01T18:09:56Z")
# Overview
The Discourse AI plugin ships a Ruby CLI under `plugins/discourse-ai/evals` that exercises AI features against YAML definitions and records results. Use it to benchmark prompts, compare model outputs, and regression-test AI behaviors without touching the app database.
## Core concepts (what users need to know)
- **Eval case** : A YAML definition under `evals/cases//.yml` that pairs inputs (`args`) with an expected outcome. Evals can check exact strings, regexes, or expected tool calls.
- **Feature** : The Discourse AI behavior under test, identified as `module:feature_name` (for example, `summarization:topic_summaries`). `--list-features` shows the valid keys.
- **Agent prompt** : The system prompt wrapped around the LLM call. Runs default to the built-in prompt unless you pass `--agent-keys` to load alternate prompts from `evals/agents/*.yml`. Add multiple keys to compare prompts in one run.
- **Judge** : A rubric embedded in some evals that requires a second LLM to grade outputs. Think of it as an automated reviewer: it reads the model output and scores it against the criteria. If an eval defines `judge`, you must supply or accept the default judge model (`--judge`, default `gpt-4o`). Without a judge, outputs are matched directly against the expected value.
- **Comparison modes** : `--compare agents` (one model, many agent prompts) or `--compare llms` (one agent prompt, many models). The judge picks a winner and reports ratings; non-comparison runs just report pass/fail.
- **Datasets** : Instead of YAML cases, pass `--dataset path.csv --feature module:feature` to build cases from CSV rows (`content` and `expected_output` columns required).
- **Logs** : Every run writes plain text and structured traces to `plugins/discourse-ai/evals/log/` with timestamps and agent keys. Use them to inspect failures, skipped models, and judge decisions.
## Prerequisites
- Have a working Discourse development environment with the Discourse AI plugin present. The runner loads `config/environment` (defaulting to the repository root or `DISCOURSE_PATH` if set).
- LLMs are defined in `plugins/discourse-ai/config/eval-llms.yml`; copy it to `eval-llms.local.yml` to override entries locally. Each entry expects an `api_key_env` (or inline `api_key`), so export the matching environment variables before running, for example:
- `OPENAI_API_KEY=...`
- `ANTHROPIC_API_KEY=...`
- `GEMINI_API_KEY=...`
- From the repository root, change into `plugins/discourse-ai/evals` and run `bundle exec ruby ./run --help` to confirm the CLI is wired up. If `evals/cases` is missing it will be cloned automatically from `discourse/discourse-ai-evals`.
## Discover available inputs
- `bundle exec ruby ./run --list` lists all eval ids from `evals/cases/*/*.yml`.
- `bundle exec ruby ./run --list-features` prints feature keys grouped by module (format: `module:feature`).
- `bundle exec ruby ./run --list-models` shows LLM configs that can be hydrated from `eval-llms.yml`/`.local.yml`.
- `bundle exec ruby ./run --list-agents` lists agent keys defined under `evals/agents/*.yml` plus the built-in `default`.
## Run evals
- Run a single eval against specific models:
- Run every eval for a feature (or the whole suite) against multiple models:
- Some evals define a `judge` block. When any selected eval requires judging, the runner defaults to `--judge gpt-4o` unless you pass `--judge name`. Invalid or missing judge configs cause the CLI to exit before running.
## Agent prompts and comparison modes
- Supply custom prompts with `--agent-keys key1,key2`. Keys resolve to YAML files in `evals/agents`; each needs `key` (optional, defaults to the filename), `system_prompt`, and an optional `description`.
- Minimal agent prompt example (`evals/agents/topic_summary_eval.yml`):
- `--compare agents` runs one model against multiple agents. The built-in `default` agent is automatically prepended so you can compare YAML prompts against stock behavior, and at least two agents are required.
- `--compare llms` runs one agent prompt (default unless overridden) across multiple models and asks the judge to score them side by side.
- Non-comparison runs accept a single agent prompt; pass one `--agent-keys` value or rely on the default prompt.
## Dataset-driven runs
- Generate eval cases from a CSV instead of YAML by passing `--dataset path/to/file.csv --feature module:feature`. The CSV must include `content` and `expected_output` columns; each row becomes its own eval id (`dataset--`).
- Minimal CSV example:
- Example:
## Writing eval cases
- Store cases under `evals/cases//.yml`. Each file must declare `id`, `name`, `description`, and `feature` (the `module:feature` key registered with the plugin).
- Provide inputs under `args`. Keys ending in `_path` (or `path`) are expanded relative to the YAML directory so you can reference fixture files. For multi-case files, `args` can contain arrays (for example, `cases:`) that runners iterate over.
- Expected results can be declared with one of:
- `expected_output`: exact string match
- `expected_output_regex`: treated as a multiline regular expression
- `expected_tool_call`: expected tool invocation payload
- Set `vision: true` for evals that require a vision-capable model. Include a `judge` section (`pass_rating`, `criteria`, and optional `label`) to have outputs scored by a judge LLM.
## Results and logs
- CLI output shows pass/fail per model and prints expected vs actual details on failures. Comparison runs also stream the judge’s winner and ratings.
- Example pass/fail snippet:
- Comparison winner snippet:
- Each run writes plain logs and structured traces to `plugins/discourse-ai/evals/log/` (timestamped `.log` and `.json` files). The JSON files are formatted for [ui.perfetto.dev](https://ui.perfetto.dev) to inspect the structured steps.
- On completion the runner echoes the log paths; use them to audit skipped models, judge decisions, and raw outputs when iterating on prompts or features.
## Common features (what to try first)
- `summarization:topic_summaries`: Summarize a conversation.
- `spam:inspect_posts`: Spam/ham classification.
- `translation:topic_title_translator`: Translate topic titles while preserving tone/formatting.
- `ai_helper:rewrite`: Prompt the AI helper for rewrites.
- `tool_calls:tool_calls_with_no_tool` and `tool_calls:tool_call_chains`: Validate structured tool call behavior.
* * *
This document is version controlled - suggest changes [on github](https://github.com/discourse/discourse/blob/main/docs/developer-guides/docs/06-general-guides/04-ai-evals.md).
---
### Author:  [@Moin](https://meta.discourse.org/u/Moin)
#### Post date: [December 1, 2025, 9:12pm UTC](https://meta.discourse.org/t/run-discourse-ai-evals/390080/2 "2025-12-01T21:12:09Z")
This seems to be about the AI plugin. Therefore, I would like to suggest adding the #ai tag.