# LLM prompt evals - Leuk om te weten

**URL:** https://meta.discourse.org/t/llm-prompt-evals-nice-to-know/345867
**Category:** Community Building
**Tags:** ai
**Created:** [6 januari 2025 om 20:32 UTC](https://meta.discourse.org/t/llm-prompt-evals-nice-to-know/345867 "2025-01-06T20:32:54Z")
**Posts on this page:** 1
**Showing post:** 1

<div class="post-metadata">

### Author: ![EricGT](https://avatars.discourse-cdn.com/v4/letter/e/f1d935/32.png) [@EricGT](https://meta.discourse.org/u/EricGT)
#### Post date: [6 januari 2025 om 20:32 UTC](https://meta.discourse.org/t/llm-prompt-evals-nice-to-know/345867/1 "2025-01-06T20:32:54Z")

</div>

For those of us who create [prompts](https://en.wikipedia.org/wiki/Prompt_engineering) for LLMs, understanding how effective our prompts are is of importance.

In LLM lingo, it is known as `evals`, short for evaluations. For those of us used to [Unit Testing](https://en.wikipedia.org/wiki/Unit_testing) in programming, the similarities are so close that I often just mentally equate the two.

During the 12 days of OpenAI, this question was asked:

> What are we as developers not doing as much as you think we should? What do you wish we did differently, or more or less of?

Michelle Pokrass of OpenAI [replied](https://community.openai.com/t/ama-on-the-17th-of-december-with-openais-api-team-post-your-questions-here/1057527/203?u=ericgt):

> One big one is evals! I see tons of developers not using evals at all and relying on vibes for rolling out changes to prod. Would highly recommend creating some simple evals using our evals product (or open source offerings) so you can update with confidence when we release new models.

On Twitter, Amanda Askell @AnthropicAI [notes](https://x.com/amandaaskell/status/1866207266761760812):

> The boring yet crucial secret behind good system prompts is test-driven development. You don’t write down a system prompt and find ways to test it. You write down tests and find a system prompt that passes them.

What many do not know and it is now starting to gain traction with the LLM model creators are tools to help end users evaluate their prompts.

OpenAI playground:

> **[Working with evals | OpenAI API](https://developers.openai.com/api/docs/guides/evals)**
>
> Learn how to test and improve AI model outputs through evaluations.

Note: This is new and in the OpenAI playground, this is not the evals we have seen for years in OpenAI GitHub ([evals](https://github.com/openai/evals))

Anthropic console:

> **[Evaluate prompts in the developer console | Claude](https://claude.com/blog/evaluate-prompts)**
>
> Generate, test, and evaluate prompts directly in the Anthropic Console with automatic test case generation and side-by-side output comparison. When building AI-powered applications, prompt quality significantly impacts results.

Microsoft .Net framework on Azure:

> **[Quickstart - Evaluate the quality of a model's response - .NET](https://learn.microsoft.com/en-us/dotnet/ai/tutorials/llm-eval)**
>
> Learn how to create an MSTest app to evaluate the AI chat response of a language model.

Disclosure: I have not used any of these automated evaluations, but I have done many simpler evaluations manually by trying different prompts. This will just make it easier.

* * *

For more details on the method of asking another (ideally larger or more powerful) model to analyze a review, rather than comparing the model output to human-created output, I recommend this lesson from Colin Jarvis.

[Lesson 6: Metaprompting with o1](https://learn.deeplearning.ai/courses/reasoning-with-o1/lesson/7/meta-prompting)  
part of the DeepLearning.AI course: [Reasoning with o1 - DeepLearning.AI](https://www.deeplearning.ai/short-courses/reasoning-with-o1/)

* * *

FYI

Wanted to add a tag `evals`, but I lack permission to create it.

---

_[View the full topic](https://meta.discourse.org/t/llm-prompt-evals-nice-to-know/345867)._