# Improving accuracy of AI reporting through validation

**URL:** https://meta.discourse.org/t/improving-accuracy-of-ai-reporting-through-validation/408794
**Category:** Extras
**Tags:** ai, workflows
**Created:** [July 29, 2026, 6:44am UTC](https://meta.discourse.org/t/improving-accuracy-of-ai-reporting-through-validation/408794 "2026-07-29T06:44:18Z")
**Posts on this page:** 1
**Showing post:** 1

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [July 29, 2026, 6:44am UTC](https://meta.discourse.org/t/improving-accuracy-of-ai-reporting-through-validation/408794/1 "2026-07-29T06:44:18Z")

</div>

Discourse AI has shipped with [AI reporting](https://meta.discourse.org/t/discourse-ai-periodic-summary-reports/290236?tl=en) for 2 and half years!

This extremely powerful feature allows you to compress large amounts of information on the forum into a more manageable report. You can see this in action on Meta:

> [@Daily Summary Feedback](https://meta.discourse.org/t/daily-summary-feedback/291853):
>
> With the creation of our newly minted #site-feedback:forum-summaries category, we now have an AI-generated Daily Summary that you can all check out to get a snapshot of what’s been happening here on meta over the last 24 hours. partying_face As with all things AI, there is a learning curve… slight_smile And this is the topic where you can share what you like and what you don’t like about the Daily Summary, and offer up any suggestions on how we could make it better. Just to note, this topi…

Over the years, models have become smarter, and the quality of the reporting improved. We have all watched the quality of reports improve every time a frontier lab released a new model. No changes needed on our end.

[Discourse workflows](https://meta.discourse.org/t/discourse-workflows/407100) now allows for a new way of improving report quality without needing to upgrade language models.

 ![image](https://global.discourse-cdn.com/meta/original/4X/1/b/0/1b03f51aef6c936abd0d83e8875c8be5a6d238f3.png)

The original Discourse AI report pipeline was static:

1. Gather context
2. Run LLM call on context
3. Create post

This simple process works surprisingly well, frontier models are spectacular at compressing large amounts of data.

However, even the smartest models out there can sometimes make mistakes.

Workflows allow us to chain multiple AI agents to achieve better results.

- Our Report Runner agent is responsible for creating a draft report.
- Our Fact checker scans the draft report and corrects mistakes using an agentic loop

### Zooming in to the nodes that make a system like this work

 ![image](https://global.discourse-cdn.com/meta/original/4X/9/3/4/934fdf641ad0ac737d081b964a47efed1e8fb3d1.png)

 ![image](https://global.discourse-cdn.com/meta/original/4X/4/e/b/4eb6a4b98d06299b7adaf91d7297a9bda30efd2e.png)

The schedule node, kicks off the workflow, important note is that it is in UTC. Timezone support will be added in future.

 ![image](https://global.discourse-cdn.com/meta/original/4X/7/c/c/7cc29a7c22c08c45ce58ce254896fdef2c4f92d7.png)

 ![image](https://global.discourse-cdn.com/meta/original/4X/9/b/1/9b1c255bbaebd86fa9988d5a9fbbe25b88c6fc48.png)

The list posts node, allows us to select a post filter, post filters are similar to [topic filters](https://meta.discourse.org/t/filtering-topic-lists-in-discourse/375558) except that they operate at the post level, so you can get multiple results per topic.

Including raw content of the post is critical for our reporting (and limiting amount of information returned per post) to keep context manageable

 ![image](https://global.discourse-cdn.com/meta/original/4X/5/1/4/5149ae1fb064b35379b6251f7964b13245033263.png)

Formatting the report so the agent can understand it is done using a custom JavaScript block

 ![image](https://global.discourse-cdn.com/meta/original/4X/4/b/5/4b5d443df70394673ad5e89a6f3e104eae4a88af.png)

```javascript
const items = $input.all();

function asArray(value) {
  if (!value) {
    return [];
  }

  return Array.isArray(value) ? value : [value];
}

function formatDate(value) {
  if (!value) {
    return "";
  }

  const date = new Date(value);
  if (Number.isNaN(date.getTime())) {
    return String(value);
  }

  return date.toISOString().slice(0, 16).replace("T", " ");
}

function truncate(text, maxLength) {
  if (!text) {
    return "";
  }

  const value = String(text);
  if (value.length <= maxLength) {
    return value;
  }

  return `${value.slice(0, maxLength).trimEnd()}...`;
}

function postFromItem(item) {
  const json = item.json || {};
  return json.post || json;
}

const posts = items
  .map((item) => postFromItem(item))
  .filter((post) => post && post.id && post.topic_id);

const summary = {
  returned_posts: posts.length,
  returned_topics: new Set(posts.map((post) => post.topic_id)).size,
  total_likes: posts.reduce((sum, post) => sum + Number(post.like_count || 0), 0),
  top_users: [],
};

const users = new Map();

for (const post of posts) {
  const username = post.username || "unknown";
  const existing = users.get(username) || {
    username,
    posts: 0,
    likes: 0,
  };

  existing.posts += 1;
  existing.likes += Number(post.like_count || 0);

  users.set(username, existing);
}

summary.top_users = Array.from(users.values())
  .sort((left, right) => {
    if (right.likes !== left.likes) {
      return right.likes - left.likes;
    }

    if (right.posts !== left.posts) {
      return right.posts - left.posts;
    }

    return left.username.localeCompare(right.username);
  })
  .slice(0, 10);

const topicMap = new Map();

for (const post of posts) {
  const topicId = post.topic_id;

  if (!topicMap.has(topicId)) {
    topicMap.set(topicId, {
      id: topicId,
      title: post.topic_title || `Topic ${topicId}`,
      slug: post.topic_slug,
      category_id: post.category_id,
      category_name: post.category_name,
      tags: asArray(post.tags),
      post_likes: 0,
      posts: [],
    });
  }

  const topic = topicMap.get(topicId);

  topic.post_likes += Number(post.like_count || 0);

  const tags = asArray(post.tags);
  for (const tag of tags) {
    if (!topic.tags.includes(tag)) {
      topic.tags.push(tag);
    }
  }
  if ((post.raw || '') !== '') {
      topic.posts.push({
        id: post.id,
        topic_id: post.topic_id,
        post_number: post.post_number,
        reply_to_post_number: post.reply_to_post_number,
        post_url: post.post_url,
        username: post.username,
        user_id: post.user_id,
        created_at: post.created_at,
        created_at_formatted: formatDate(post.created_at),
        updated_at: post.updated_at,
        excerpt: post.excerpt,
        raw: post.raw,
        cooked: post.cooked,
        like_count: Number(post.like_count || 0),
        reply_count: Number(post.reply_count || 0),
        score: Number(post.score || 0),
      });
  }
}

const topics = Array.from(topicMap.values())
  .map((topic) => {
    topic.posts.sort((left, right) => {
      return Number(left.post_number || 0) - Number(right.post_number || 0);
    });

    return topic;
  })
  .sort((left, right) => {
    if (right.post_likes !== left.post_likes) {
      return right.post_likes - left.post_likes;
    }

    return String(left.title).localeCompare(String(right.title));
  });

let context = "";

context += "## Summary\n\n";
context += `Returned posts: ${summary.returned_posts}\n`;
context += `Returned topics: ${summary.returned_topics}\n`;
context += `Total likes in returned posts: ${summary.total_likes}\n\n`;

context += "Top users in returned posts:\n";
for (const user of summary.top_users) {
  context += `@${user.username} (${user.likes} likes, ${user.posts} posts)\n`;
}

context += "\n## Topics\n";

for (const topic of topics) {
  context += `\n### ${topic.title}\n\n`;
  context += `topic_id: ${topic.id}\n`;

  if (topic.category_name) {
    context += `category: ${topic.category_name}\n`;
  }

  if (topic.tags.length > 0) {
    context += `tags: ${topic.tags.map((tag) => `#${tag}`).join(", ")}\n`;
  }

  context += `sample_likes: ${topic.post_likes}\n`;

  let lastPostNumber = 0;

  for (const post of topic.posts) {
    if (post.post_number && lastPostNumber && post.post_number > lastPostNumber + 1) {
      context += "\n...\n";
    }

    context += "\n";
    context += `post_id: ${post.id}\n`;
    context += `post_number: ${post.post_number}\n`;

    if (post.post_url) {
      context += `url: ${post.post_url}\n`;
    }

    if (post.created_at_formatted) {
      context += `${post.created_at_formatted}\n`;
    }

    if (post.username) {
      context += `user: @${post.username}\n`;
    }

    context += `likes: ${post.like_count}\n`;

    const body = post.raw || post.excerpt || "";
    context += `${truncate(body, 4000)}\n`;

    lastPostNumber = Number(post.post_number || lastPostNumber);
  }
}

return [
  {
    json: {
      summary,
      topics,
      context,
    },
  },
];

```

 ![image](https://global.discourse-cdn.com/meta/original/4X/b/9/5/b95e9058e825311ef3ec35f081a5d0969accf169.png)

 ![image](https://global.discourse-cdn.com/meta/original/4X/5/2/7/527a895337a6d56b24948f7d33e4863393c3fade.png)

Notably when creating reports you have complete control over the shape of the data passed to the agent and System Prompt it uses.

 ![image](https://global.discourse-cdn.com/meta/original/4X/8/2/c/82c7025e24f59c27ad794ea7352d1a34fd4a53b4.png)

 ![image](https://global.discourse-cdn.com/meta/original/4X/8/8/0/88000cbf25458533d186fcd6b09a68170b027669.png)

 ![image](https://global.discourse-cdn.com/meta/original/4X/1/f/2/1f221dcadedbecc058e1f05f53fce8ce3d5beab0.png)

The merge node allows us to take 2 reports and merge them into a single report.

Our internal weekly update is composed from staff updates and general activity on our internal community. By merging both into a single stream we can create a single cohesive report from 2 different sources.

Finally the heavy lifting of correcting the report is left to a fact checker agent:

 ![image](https://global.discourse-cdn.com/meta/original/4X/8/9/b/89bd2931798a2637605933925f1fc19eeaf56333.png)

 ![image](https://global.discourse-cdn.com/meta/original/4X/e/2/c/e2c05f3b2b839a168b799cbb9a5ef44777b8f365.png)

Notably a single read post tool with an agentic loop is able to do the major lift of walking through every fact on the report and ensuring it is not hallucinated.

* * *

This strategy allows us to get better intelligence for less money. Our weekly report used to be generated with Claude Opus 4.8 which is a great model that is more capable than DeepSeek v4 Flash. Yet by properly defining the pipeline and verification loop we are able to get better results for less money.

Hope you find this tutorial helpful, let me know if you have any questions.

---

_[View the full topic](https://meta.discourse.org/t/improving-accuracy-of-ai-reporting-through-validation/408794)._