Prompt tools: funnel, orbit, and flux charts

Contribute Feature
,

AI-generated summary

EricGT introduces three analytical tools—Funnel, Flux, and Orbit—originally from Hex Tech’s blog, aimed at improving prompt evaluation. These tools could potentially be integrated into Discourse for prompt creators. The Funnel breaks down binary eval outcomes (pass/fail) into sequential stages with individual pass/fail criteria. Flux quantifies movement through the funnel, measuring net outcomes of treatments and stage-to-stage transitions. Orbit visualizes individual evals as they progress through concentric “orbits,” with earlier stages closer to the center, offering a dense, visual summary of experimental results. Accompanying images illustrate each tool: a bar chart for Funnel showing BAD/PARTIAL/GOOD levels, a heatmap for Flux highlighting stage performance (e.g., Stage 3 with 33 evaluations), and a radial diagram for Orbit depicting eval movement through stages. These tools aim to provide deeper, structured insights into prompt evaluation beyond simple pass/fail metrics.

1

Just learned about these tools for use creating better prompts

Maybe someday in the future they could be created and incorporated for those creating prompts with Discourse. :slightly_smiling_face:

Funnel: decomposes each eval from a binary outcome of pass/fail into a series of cascading steps, each with its own pass/fail criteria.

Flux: Flux is our quantitative measure of movement through the funnel. We look at flux both in aggregate, to quantify the net outcome of a treatment on our funnel, and broken out by stage, to see how evals are transitioning from stage to stage.

Orbit: The orbit chart visualizes individual evals as they move through “orbits” representing the funnel, with earlier stages closer to the center. It’s an extremely information-dense view of an experimental result.

Images

Funnel

The image depicts a bar chart with four bars of varying heights and shaded portions, indicating different levels of performance or data, labeled as "BAD," "PARTIAL," and "GOOD." (Captioned by AI)
The image depicts a bar chart with four bars of varying heights and shaded portions, indicating different levels of performance or data, labeled as "BAD," "PARTIAL," and "GOOD." (Captioned by AI)740Ă—482 43.1 KB

Flux

The heatmap displays evaluation counts across five stages, with stage 3 (Correct Data) having the highest evaluations of 33. (Captioned by AI)
The heatmap displays evaluation counts across five stages, with stage 3 (Correct Data) having the highest evaluations of 33. (Captioned by AI)690Ă—494 8.4 KB

Orbit

The image is a circular diagram composed of intersecting lines and dots in red, green, and grey, arranged in a segmented pattern radiating outward from the center. (Captioned by AI)
The image is a circular diagram composed of intersecting lines and dots in red, green, and grey, arranged in a segmented pattern radiating outward from the center. (Captioned by AI)465Ă—475 63.6 KB

2 Likes