Observability for Discourse AI

Contribute Feature
,
2

Hahahaha, I wish.

We log every single request and response to LLMs in a table, and allow admins to query those at any time via Data Explorer. Have you tried this already?

image
image1370×979 62.4 KB

{
  "max_tokens": 2000,
  "model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
  "temperature": 0,
  "stop": [
    "\n</output>"
  ],
  "messages": [
    {
      "role": "system",
      "content": "You are a markdown proofreader. You correct egregious typos and phrasing issues but keep the user's original voice.\nYou do not touch code blocks. I will provide you with text to proofread. If nothing needs fixing, then you will echo the text back.\nYou will find the text between <input></input> XML tags.\nYou will ALWAYS return the corrected text between <output></output> XML tags.\n\n"
    },
    {
      "role": "user",
      "content": "<input>We log every single request and response to LLMs in a table, and allow admins to query those at any time via Data Explorer. Have you tried already?</input>"
    }
  ]
}

{
  "id": "chat-45cd241b6e0f4a58840fcc9f49dfa56a",
  "object": "chat.completion",
  "created": 1722528517,
  "model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "<output>We log every single request and response to LLMs in a table, and allow admins to query those at any time via Data Explorer. Have you tried this already?</output>",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 135,
    "total_tokens": 174,
    "completion_tokens": 39
  }
}

Creating evals for our features is certainly on our roadmap for 3.4, in special for tweak on our Related Topics and Summarization features.

1 Like