Observability for Discourse AI

Contribute Feature
,

AI-generated summary

The discussion revolves around the importance of observability for Discourse AI, particularly in monitoring and evaluating LLMs. aas emphasizes the need for robust evaluation systems, citing LangSmith as a potential tool for LLM tracing and experimentation. Falco counters that Discourse already logs all LLM requests and responses in a database, accessible via Data Explorer, and shares a screenshot of the logged data. aas acknowledges the existing logging but notes that LLM calls are made from Ruby, making direct integration with LangSmith challenging. Despite this, aas sees potential in exporting the logs to create traces in LangSmith for evaluation purposes. The team confirms that creating evaluations for features like Related Topics and Summarization is on the roadmap for version 3.4.

1

Monitoring and evaluating LLMs is critical:

I started working with language models five years ago when I led the team that created CodeSearchNet, a precursor to GitHub CoPilot. Since then, I’ve seen many successful and unsuccessful approaches to building LLM products. I’ve found that unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems.

If Discourse AI is to power business-critical LLM tasks, I think supporting monitoring tools like LangSmith should be prioritized.

Using LangSmith is as simple as running yarn add langchain langsmith and adding a few environment variables.

Has team Discourse thought about how we can configure LLM tracing? Also, any thoughts on how we can implement this prior to discourse-ai official supporting it?

2 Likes
2

Hahahaha, I wish.

We log every single request and response to LLMs in a table, and allow admins to query those at any time via Data Explorer. Have you tried this already?

image
image1370×979 62.4 KB

{
  "max_tokens": 2000,
  "model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
  "temperature": 0,
  "stop": [
    "\n</output>"
  ],
  "messages": [
    {
      "role": "system",
      "content": "You are a markdown proofreader. You correct egregious typos and phrasing issues but keep the user's original voice.\nYou do not touch code blocks. I will provide you with text to proofread. If nothing needs fixing, then you will echo the text back.\nYou will find the text between <input></input> XML tags.\nYou will ALWAYS return the corrected text between <output></output> XML tags.\n\n"
    },
    {
      "role": "user",
      "content": "<input>We log every single request and response to LLMs in a table, and allow admins to query those at any time via Data Explorer. Have you tried already?</input>"
    }
  ]
}

{
  "id": "chat-45cd241b6e0f4a58840fcc9f49dfa56a",
  "object": "chat.completion",
  "created": 1722528517,
  "model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "<output>We log every single request and response to LLMs in a table, and allow admins to query those at any time via Data Explorer. Have you tried this already?</output>",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 135,
    "total_tokens": 174,
    "completion_tokens": 39
  }
}

Creating evals for our features is certainly on our roadmap for 3.4, in special for tweak on our Related Topics and Summarization features.

1 Like
3

I didn’t say that was all there was to it. (:wink:) But I guess it doesn’t matter since I think LLM calls are made from Ruby.

I haven’t yet, but this is brilliant - thank you! Theoretically, I could export these and programmatically create traces in LangSmith for evals and experiments.

1 Like