Stop Ranking Agent Configs by Average Score

Stop Treating Agent Eval Like a Leaderboard

eventually hits the same question: which version should I actually ship?

You change the model. You rewrite the prompt. You swap a retrieval tool. You run a batch. The average score moves a little. Maybe one version wins by two points. Maybe the judge prefers one output on Monday and another on Tuesday. So you ship the “best” average.

That is often the wrong decision frame. Agent performance is rarely just better model + better prompt + better tool. The useful question is how those pieces behave together. A prompt that helps a smaller model may slow down a stronger one. A semantic search tool may pair beautifully with an open-ended prompt and poorly with a rigid step-by-step prompt.

This article shows a small, practical way to test that. Pick a few agent levers. Run competing configurations on the same examples. Ask a judge to choose only the best and worst outputs. Then fit a ranking model that turns those choices into utility scores for each configuration and each interaction.

The opinion: if you only score every answer in isolation, rank the averages, and take the top config, you are leaving signal on the table. Direct comparison is why golden sets are valuable in the first place. Best-worst comparison forces the judgment where the variation actually matters.

You can use the method deeply, with a Plackett-Luce model and factorial interaction terms. Or, if you are building agent-first, you can hand the same structure to an evaluation agent and let it run the loop. The model is not the point by itself. The point is cleaner evidence for the next engineering decision.

For Agent-First Builders

You do not have to run this by hand.

If your normal workflow is to give an agent a goal, a test set, expected outputs, a few models to choose from, and a judge agent to review the results, this method still applies. It just gives the agent a better evaluation protocol.

Prompt to try:
Create a small factorial grid of candidate agent configurations. For each test example, sample several configs and run them independently. Anonymize the outputs. Ask a judge to select exactly one best and one worst response. Fit a Plackett-Luce model, MaxDiff model, or equivalent utility ranking. Report which configs should be deployed, pruned, or tested next, and explain which model/prompt/tool interactions appear to matter.

That is the practical value for agent-first development. The agent can generate candidates, run comparisons, inspect utility results, and propose the next set of configurations. The human still decides what matters, but the system gets better evidence with each cycle.
That said, in some cases you will need to be able to walk through the details in order to create a repeatable structure that fully captures the nuance of high value decisions.

The Experiment

The task. An agent pipeline extracts structured JSON from invoice images. Invoice extraction is not glamorous, but it is exactly the kind of work that actually runs in production. The schema is fixed, correctness matters, and errors have downstream consequences. That makes it a useful evaluation surface: the agent either got the right fields or it did not.

I used the shubh303/Invoice-to-Json dataset from HuggingFace: 100 invoice documents, 5 sampled configurations per document, and 499 total production runs after one invalid run was excluded. Of those, 488 achieved valid schema compliance.

The design. Three binary factors define the configuration space, producing eight distinct pipelines:

Factor	Choice 1	Choice 2
Model	Claude Haiku 4.5	GPT-5.4-mini
Prompt	Systematic Planner	Contextual Leaper
Tool	stream table rows	semantic search

The two prompt styles represent different extraction philosophies. The Systematic Planner is rule-following: it gives the agent an explicit sequence and constrains tool use. The Contextual Leaper is semantic and open: it asks the agent to understand the document first, then extract what is needed with more freedom.

Both tools are deterministic stubs. This matters. The experiment is not testing OCR quality, live retrieval quality, or parsing reliability. It is testing how the agent navigates tool handles and manages context execution when the tool affordances change.

The evaluation protocol. For each invoice, a random subset of 5 configs from the 8 processed the document independently. A judge model (claude-sonnet-4-6) then selected the single best and single worst output from the anonymized batch.

This is Best-Worst Scaling, also known as Maximum Difference or MaxDiff-style comparison. The judge does not need to score every answer on an artificial 1–10 scale. It only has to answer two questions: which output is best in this batch, and which is worst?

What Average Scores Tell You — and What They Miss

Before reaching for a ranking model, look at what a standard analysis would show. Here are the macro score averages for each factor, collapsed across all 499 runs:

The gaps are small. A team looking only at this table might reasonably conclude that model matters a little, prompt barely matters, and tool choice is almost a coin flip.

Now look at the raw best-worst win rates by configuration:

Config	Model	Prompt	Tool	Appearances	Wins	Win Rate
7	GPT	Leaper	Semantic	71	21	29.6%
1	Haiku	Planner	Semantic	66	17	25.8%
3	Haiku	Leaper	Semantic	58	13	22.4%
4	GPT	Planner	Stream	62	12	19.4%
6	GPT	Leaper	Stream	63	11	17.5%
2	Haiku	Leaper	Stream	56	9	16.1%
0	Haiku	Planner	Stream	61	9	14.8%
5	GPT	Planner	Semantic	62	8	12.9%

This is more useful, but still incomplete. It shows that Config 7 is strong. It also makes Config 1 look like a clear second-place option and Config 4 look less interesting than it really is.

The missing piece is competition quality. A config that wins against weak batches should not get the same credit as a config that wins against strong ones. Beating a top config means something. Beating the discount config does not mean much.

That is why isolated scores and raw win rates are both limited. They contain useful signal, but they do not fully account for who each config had to beat.

Turning Best/Worst Picks into Utility Scores

The model I used for this is called Plackett-Luce. Its part of a family of models for best-worst comparison. The job is simple: take a collection of best/worst choices and estimate an underlying utility score for each option.

In plain English, the model asks: given the configs that appeared together in each batch, which configs were consistently preferred after accounting for the strength of the alternatives?

The useful extension here is effects coding. Each factor level is coded as +1 or −1, which lets the model estimate two kinds of signal:

Main effects estimate how much a factor appears to help on its own. Interaction effects estimate whether a combination works better or worse than the individual parts suggest.

That distinction is the heart of the article. Sometimes you can just swap in a better model and get most of the benefit. In other cases, the model only works well with a specific prompt/tool pairing. Each case is different. The point is to measure the configuration as a unit instead of assuming the parts add cleanly.

Results: The Useful Signals

Feature Importance

The first question is not “which coefficient looks clever?” It is: which variation actually matters?

Effect	Importance	95% Bootstrap CI
Triple interaction (Model × Prompt × Tool)	0.31	[0.023, 0.635]
Model (main)	0.25	[0.018, 0.572]
Model × Prompt	0.15	[0.009, 0.528]
Tool (main)	0.13	[0.006, 0.451]
Prompt × Tool	0.07	[0.006, 0.319]
Prompt (main)	0.03	[0.005, 0.338]
Model × Tool	0.01	[0.004, 0.341]

In this particular experiment, the biggest signal is the three-way interaction between model, prompt, and tool. That does not mean every agent eval will behave this way. In some systems, model choice will dominate. In others, retrieval quality or OCR quality may dwarf everything else.

Here, though, the result is clear enough to act on: evaluating model, prompt, and tool independently would miss a large part of the story. Prompt as a global main effect is tiny, but prompt appears inside the interaction structure. That means prompt value is conditional. It depends on which model and tool it is paired with.

Configuration Rankings

Here are all 8 configurations ranked by their Plackett-Luce total utility score, with every term shown explicitly. The β (beta) terms applies to single factors like the model choice. The γ (gamma) terms shows the how interactions between choices, like the way the model and tool work together.

Config	7	1	3	4	6	2	0	5
Model	GPT	GPT	Haiku	GPT	GPT	Haiku	Haiku	Haiku
Prompt	Leaper	Planner	Planner	Leaper	Planner	Leaper	Planner	Leaper
Tool	Semantic	Stream	Semantic	Stream	Semantic	Stream	Stream	Semantic
β model	+0.13	+0.13	−0.13	+0.13	+0.13	−0.13	−0.13	−0.13
β prompt	−0.01	+0.01	+0.01	−0.01	+0.01	−0.01	+0.01	−0.01
β tool	+0.06	−0.06	+0.06	−0.06	+0.06	−0.06	−0.06	+0.06
γ mp	+0.07	−0.07	+0.07	+0.07	−0.07	−0.07	+0.07	−0.07
γ mt	−0.01	+0.01	+0.01	+0.01	−0.01	−0.01	−0.01	+0.01
γ pt	+0.03	+0.03	−0.03	−0.03	−0.03	−0.03	+0.03	+0.03
γ mpt	+0.16	+0.16	+0.16	−0.16	−0.16	+0.16	−0.16	−0.16
Total	+0.43	+0.20	+0.15	−0.06	−0.07	−0.16	−0.23	−0.27

The rankings shift once competition quality is accounted for. Config 1 drops from the raw-win-rate second place to third. Config 4 jumps from raw-win-rate fourth to second. Config 5 is the cautionary case: it has the strongest β-only baseline on paper, but its total score falls below zero once interaction penalties are included.

That is exactly the kind of signal average scoring tends to bury. The question is not only “which components are strongest?” It is “which config wins against serious alternatives?”

Operational Decisions

Deploy: Config 7

Config 7 — GPT-5.4-mini / Contextual Leaper / semantic_search · Total: +0.431

Config 7 is the clear winner at +0.431. Its individual component baseline is solid, but not enough to explain the result by itself. What makes it stand out is that the model, prompt, and tool reinforce each other. The Contextual Leaper and semantic tool are specifically good with GPT-5.4-mini in this setup.

The operational lesson is not “semantic tools always win” or “open prompts always win.” The lesson is narrower and more useful: this configuration works as a unit. Ship it as a dedicated workflow, and do not assume the same interaction gain survives a model swap.

Deploy: Config 4

Config 4 — GPT-5.4-mini / Systematic Planner / stream_table_rows · Total: +0.196

Config 4 reaches +0.196 by a different path. It pairs GPT-5.4-mini with the Systematic Planner and stream-based tool. If your architecture makes stream integration simpler than semantic search, this is not just a fallback. It is a legitimate production candidate.

This matters because real deployment decisions include cost, latency, integration complexity, and reliability. The statistical winner is not always the business winner. The value of the model is that it gives you a better map of the trade space.

Investigate: Config 1

Config 1 — Haiku 4.5 / Systematic Planner / semantic_search · Total: +0.152

Config 1 is the interesting case. It is the only Haiku configuration that clears zero, despite carrying the model penalty. That makes it useful evidence, not just an also-ran.

The plausible story is complementarity. Haiku appears to benefit from the structure and tool affordance in a way GPT does not. The weaker model may need the scaffold and semantic tool more, while the stronger model may experience the same scaffold as friction.

This is why component strength and configuration strength are not the same thing. A weaker component can still belong in a strong configuration when the surrounding pieces compensate for it.

Prune: Config 5

Config 5 — GPT-5.4-mini / Systematic Planner/ semantic_search · Total: −0.066
β-only sum: +0.201 (highest of any config). All four interaction terms flip negative.

Config 5 looks good if you only add up the parts: strong model, positive prompt, positive tool. But its total utility score is −0.066.

That is the trap. The parts look attractive in isolation, but the combination does not work as well as expected. In this setup, the Systematic Planner and semantic tool appear to create friction under GPT-5.4-mini rather than leverage.

Configs 0 and 3 also sit below the action line. In a production system, I would remove these from rotation, preserve them as reference cases, and spend evaluation budget on stronger or more informative competitors.

What to Optimize Next

The feature-importance table answers a question teams often skip: before optimizing anything, which lever is worth more attention?

In this experiment, prompt sweeps in isolation would be a low-leverage move. Prompt does not show much global lift. Its value shows up through the combinations. That means the next useful test is not “try ten more prompts.” It is “test prompt strategies inside the model/tool pairings where they are likely to matter.”

The same warning applies to model swaps. In some cases, swapping in a stronger model will get you most of the way there. In other cases, it can destroy the interaction structure that made a cheaper or smaller configuration work. This method helps you see which case you are in.

The highest-leverage engineering work is usually at the orchestration layer: how the model, prompt, tool, retrieval context, and judge feedback behave as a system. A small factorial design will not answer every question, but it is a much better first pass than isolated leaderboard scoring.

Close the Loop: Feed Eval Back into the Agent

The most useful version of this is not a one-time benchmark.

Once the utilities are estimated, the results can feed back into the agent system itself. Winning configs become default workflows. Bad configs are removed from rotation. Edge cases become targeted follow-up tests. If a router is choosing between model/prompt/tool combinations, the fitted utilities become context for that router instead of tribal knowledge living in a notebook.

The loop: test configs → judge outputs → estimate utility → deploy winners → prune failures → update routing context → test again.

That changes the role of evaluation. It is no longer just a report card after the fact. It becomes part of the agent’s learning cycle.

Limits and Next Steps

This is a demonstration, not an academic paper. The sample is intentionally small, the tools are stubs, and the confidence intervals are wide. That is fine for the purpose of the article: the goal is to show how much signal direct comparison can capture, not to claim a universal law about these specific models, prompts, or tools.

The natural extensions are straightforward: add OCR as a fourth factor, test a broader range of model families, include cost or latency explicitly, and run more trials to tighten uncertainty. The same structure can also be implemented as a recurring evaluator in LangSmith or a similar eval stack.

The larger point is the one worth carrying forward: agent eval should not be only about scoring isolated answers and ranking averages. When configs compete head-to-head on shared examples, you get a cleaner view of what is actually working. The ranking model is just the interpreter. The real value is the comparison structure.

@misc{Invoice-to-Json,
title={Invoice-to-Json: A Document Understanding and Information Extraction Dataset},
year={2024}
}

About the Author

Doster Esh is an information architect focused on context structure and statistical evaluation. Much of his work centers on adding structure to data in ways that simplify the decision space — through what he calls cultivated partitioning. The core idea is that both humans and agents make better choices when they are not cognitively overwhelmed by undifferentiated information. He has a history of running a quantitative electricity trading firm and working as a data scientist in retail and financial services. If you want to talk about using AI/ML to make your context more structured, reach out.