Prompt tools: funnel, orbit, and flux charts

Just learned about these tools for use creating better prompts

Maybe someday in the future they could be created and incorporated for those creating prompts with Discourse. :slightly_smiling_face:


Funnel: decomposes each eval from a binary outcome of pass/fail into a series of cascading steps, each with its own pass/fail criteria.

Flux: Flux is our quantitative measure of movement through the funnel. We look at flux both in aggregate, to quantify the net outcome of a treatment on our funnel, and broken out by stage, to see how evals are transitioning from stage to stage.

Orbit: The orbit chart visualizes individual evals as they move through “orbits” representing the funnel, with earlier stages closer to the center. It’s an extremely information-dense view of an experimental result.


Images

Funnel

Flux

Orbit