in the eval harness. Then it got shut down.
I was in the room for the review that killed it. A mid-market SaaS company had built a customer-operations agent to handle billing and account tickets. They had instrumented it against the twelve-metric evaluation framework I published on this site two months ago. Task completion, faithfulness, retrieval precision, tool-call accuracy, latency, hallucination rate. Every metric was inside its target band. The agent was, by every measure the harness tracked, working.
The CFO opened her laptop and put one number on the screen. The cost per resolved ticket, including the tickets the agent tried and failed to resolve, was higher than what the company had been paying humans to do the same work. Not by a rounding error. The agent-assisted workflow was costing the company more per resolved ticket than the fully-human workflow it was supposed to replace.
The head of engineering pushed back. Every quality metric is green, he said. The agent is more accurate than the team was on the same ticket types. That was true. It was also, in that room, irrelevant. The agent was accurate and economically underwater, and the eval harness had no metric that could see the second half of that sentence.
I built that harness. I stand by the twelve metrics. But I got something wrong, and two months of watching teams deploy against it has made the gap impossible to ignore. The harness measures whether the agent is right. It does not measure whether the agent pays for itself. In 2026, that second question is the one that decides which agents stay in production, and it is not in the framework.
This piece is the metric I should have included, why it is harder to measure than it looks, and what teams do when the number comes back underwater.
What the harness measures, and the half it misses
The twelve-metric harness was built to answer a specific question: is this agent behaving correctly in production? It covers the failure modes that make an agent wrong. Retrieval that surfaces the wrong context. Generation that hallucinates. Tool calls that construct bad arguments. Reasoning loops that never terminate. Outputs that drift from the user’s intent. Each metric isolates a way the agent can produce a wrong result, and together they give you a defensible read on quality.
Every one of those metrics is a quality metric. Not one of them is an economic metric.
This did not feel like a gap when I wrote the framework. Quality was the hard problem. Cost felt like an operational detail you could read off a billing dashboard whenever you wanted it. Token costs were falling every quarter. The narrative in early 2026 was that inference was getting cheaper and the real challenge was reliability. Measure quality, the thinking went, and the economics will take care of themselves.
The economics did not take care of themselves. Inference costs per token did keep falling, but agent architectures got hungrier faster than tokens got cheaper. A single agent resolution in 2026 is not one model call. It is a plan step, several tool calls, a few reasoning passes, sometimes a self-critique loop, sometimes a retry after a failed tool call, and a final synthesis. The token price went down and the tokens per resolution went up, and for a lot of workflows the second effect won.
That is the structural reason a quality-only harness misleads. It tells you the agent is good at the job. It says nothing about whether the job, done this way, is worth doing. An agent can sit comfortably inside every quality band in the framework and still lose money on every unit of work it completes. The harness will show you all green while the unit economics bleed, because the harness was never pointed at the unit economics.
The metric: cost per successful outcome
The metric the harness needed is cost per successful outcome. Not cost per call. Not cost per token. Not accuracy. The fully-loaded cost of producing one successful business outcome, divided by the number of successful outcomes.
The word that does the work in that definition is successful. Cost per call is easy and misleading. Cost per successful outcome is hard and honest, and the difference between them is where the SaaS company’s economics went wrong.
Here is the arithmetic that killed their agent, with the numbers rounded to protect the client but the structure intact.
The agent attempted every incoming ticket. Each attempt cost about $3.40 in fully-loaded inference: the reasoning passes, the tool calls to their billing system, the retries, the synthesis. The agent resolved 71 percent of tickets to a quality bar the eval harness confirmed was high. The other 29 percent, it escalated to a human after attempting them.
Take a thousand tickets. The agent spends $3,400 attempting all of them. It resolves 710 and escalates 290. Those 290 escalations then go to a human agent at roughly $4.20 of loaded cost each, another $1,218. The company paid $4,618 to resolve a thousand tickets that would have cost $4,200 with humans alone. The agent-assisted workflow was 10 percent more expensive than the workflow it replaced.
Now compute the metric that matters. Cost per successful agent outcome is not the $3.40 per attempt that shows up on the inference dashboard. It is the total agent spend divided by the successful resolutions: $3,400 over 710, which is $4.79 per resolved ticket. Every ticket the agent successfully resolved effectively cost $4.79, because the cost of the attempts that failed and escalated has to be carried by the attempts that succeeded. A human resolved the same ticket for $4.20. The agent was more accurate and fifty-nine cents more expensive on every success, before you even counted the double-payment on the escalations.
The eval harness saw a 71 percent resolution rate at high quality and reported success. The CFO saw the true cost of a resolved ticket sitting above the human baseline and reported failure. They were both right. They were measuring different things, and only one of those things determined whether the agent survived.
Why it is harder to measure than it looks
If cost per successful outcome is the metric that decides survival, the obvious question is why teams do not already track it. The answer is that it is genuinely hard to compute, in three specific ways that a cost-per-call dashboard hides.
The first is attribution of failed attempts. The cost of a successful outcome includes the cost of the failures on the way to it. When an agent tries a ticket, burns $3.40 in reasoning and tool calls, and then escalates, that $3.40 did not produce a successful outcome, but it is a real cost of running the system that produces successful outcomes. You cannot honestly compute cost per success without spreading the failed-attempt cost across the successes. Most teams instrument cost per call, see a comfortable number, and never do the division that reveals the real figure. The cost-per-call number and the cost-per-success number can differ by more than a factor of two, and the gap grows as the resolution rate drops.
The second is the double-payment on escalation. In any human-in-the-loop workflow, a failed agent attempt does not replace the human cost. It adds to it. The company pays for the agent to try and then pays for the human to finish. This is the opposite of the intuition that agents reduce cost by deflecting work. Below a certain resolution rate, the deflection saves less than the failed attempts cost, and the agent becomes a tax on the human workflow rather than a substitute for it. Whether you are above or below that line is invisible unless you are measuring cost per successful outcome against the human baseline.
The third is that the business value of an outcome is often not a clean number. Cost per successful outcome only means something compared to what a successful outcome is worth. For a support ticket, the value is the loaded cost of the human alternative, which is knowable. For an agent that qualifies a sales lead, or drafts a contract clause, or triages a security alert, the value of one successful outcome is a modeling exercise, not a lookup. Teams avoid the metric partly because the denominator is hard and the value side is harder. But avoiding it does not make the economics go away. It just means you find out from a CFO in a review instead of from a dashboard in time to act.
None of these are reasons the metric does not matter. They are reasons it gets skipped, and skipping it is exactly how an accurate agent ends up shut down.
The threshold that decides survival
Once you can compute cost per successful outcome, the survival test is a single comparison. Cost per successful outcome has to sit below the business value of that outcome, with enough margin to justify the operational overhead of running the agent at all.
For the SaaS company, the business value of a resolved ticket was the loaded human cost it displaced, about $4.20. Their cost per successful outcome was $4.79. The agent was underwater by fifty-nine cents on every successful resolution, and the system as a whole was underwater once you added the escalation double-payment. There was no margin to discuss. The agent was losing money on its wins.
The threshold has a property that makes it dangerous to ignore early. It moves against you as the workflow scales, not with you. The intuition from traditional software is that unit costs fall with scale. Agent unit costs often do not, because each unit of work is its own inference run and does not get cheaper because you are doing more of them. If your cost per successful outcome is marginally above the value line at pilot scale, scaling the deployment does not rescue it. It multiplies the loss. This is the opposite of the SaaS scaling curve most teams carry in their heads, and it is why an agent that looks close-enough at pilot can become a six-figure line item that finance flags before engineering notices.
The teams that keep their agents in production are not the ones with the best quality metrics. They are the ones whose cost per successful outcome sits comfortably below the value of the outcome, with room to spare. That is a different selection criterion than the eval harness applies, and it explains why the agents that survive a year in production are frequently not the most sophisticated ones. They are the ones deployed on workflows where a successful outcome is worth enough to cover what a success costs.
What teams do when the number comes back underwater
An underwater cost per successful outcome is not automatically a death sentence. It is a signal that the workflow, the architecture, or the deployment scope needs to change. There are real levers, and there is one specific trap.
The first lever is reducing the cost per attempt without touching the resolution rate. Most agents in 2026 are over-provisioned for the median task. They run the same deep reasoning chain and the same full tool suite on the easy tickets as on the hard ones. Trimming reasoning depth on the tasks that do not need it, caching the retrieval and tool results that repeat, and cutting the retry budget where retries rarely help can pull cost per attempt down meaningfully. The SaaS company got their cost per attempt from $3.40 to about $2.30 through this kind of work, without measurably moving the resolution rate.
The second lever is raising the resolution rate without raising the cost per attempt, which spreads the same spend across more successes and lowers cost per successful outcome directly. This is genuine agent improvement, and it is slow, but it is the lever that compounds.
The third lever is narrowing the deployment scope to the ticket types where the agent’s cost per successful outcome is already below the value line, and routing the rest to humans from the start rather than letting the agent attempt and fail. If the agent is economically strong on billing questions and underwater on complex disputes, deploying it only on billing questions can flip the whole system positive.
That third lever is where the trap lives, and it is the trap I wrote about in a separate piece on this site. The obvious way to cut cost is to route the hard work to a cheaper model or away from the agent entirely. Done with a naive classifier, that reintroduces a quality failure the eval harness will eventually catch, and you trade an economics problem for a quality problem. The cost lever and the quality lever are coupled. You cannot pull one hard without watching the other. The teams that get this right optimize cost per successful outcome as a joint constraint with the quality metrics, not as a separate dial. For teams building AI-powered workflow automation at production scale, that joint optimization is the actual engineering problem, and it is the one the single-metric dashboards on either side hide.
The SaaS company pulled the first and third levers. They got cost per attempt to $2.30, narrowed the deployment to the ticket types where the agent was economically strong, and routed complex disputes to humans from intake instead of after a failed agent attempt. Cost per successful outcome came down to about $3.10 against the $4.20 value line. The agent went from ten percent more expensive than humans to roughly twenty-five percent cheaper, on a narrower scope, at a quality bar the harness still confirmed. It stayed in production. It survived because someone finally measured the thing that decides survival, and then changed the deployment until the number cleared the line.
The same metric, pointed the other way
The reason I want cost per successful outcome in the framework is not that it kills agents. It is that it is the only metric that tells you the truth in both directions, and the second direction is the one a cost-per-call dashboard gets catastrophically wrong.
A few weeks after the SaaS review, I looked at a very different deployment. A legal operations team had built an agent to review commercial contracts for a specific class of risk clauses before a human lawyer signed off. The agent ran an expensive pipeline. Long-context ingestion of the full contract, multiple reasoning passes over the clause library, a retrieval step against prior redlines, and a self-critique pass before it produced its findings. The cost per attempt was about eighteen dollars. On a cost-per-call dashboard, eighteen dollars a run is the kind of number that gets an agent flagged for cost review and quietly throttled.
It had been flagged. On the quarterly AI-spend review I sat in on, the contract agent was the line circled in red at the top of the cut list: the single most expensive per-call workload in the budget. The instinct in the room was to move it to a cheaper model and a shorter pipeline. That instinct, followed, would have destroyed the thing that made the agent worth running.
Cost per successful outcome told the opposite story. The agent flagged a genuine risk clause, later confirmed by a lawyer, on roughly one in nine contracts. Price it conservatively and count only those catches as outcomes worth paying for. Nine runs at eighteen dollars is $162 of agent spend per caught clause. A single missed clause of that class ran into five and six figures of downstream exposure, and the legal team could quantify from their own claims history how often the human-only review had been letting them through. Against a value per successful outcome measured in tens of thousands of dollars, $162 was not expensive. It was one of the highest-return workloads in the company.
The cost-per-call view said this was their most expensive agent and a candidate for cost-cutting. The cost-per-successful-outcome view said it was their most valuable agent and a candidate for expansion. Same agent, same spend, opposite conclusions, and only one of the two views was connected to what the work was worth. The team that had been about to throttle it instead widened its scope to two more clause classes, accepted the higher inference bill without flinching, and treated the eighteen-dollar-per-run number as exactly what it was: cheap insurance.
This is why the metric belongs next to the quality metrics and not in a footnote. An agent’s cost per call tells you what it spends. Only its cost per successful outcome, measured against the value of an outcome, tells you whether that spend is the best or the worst money the company is putting into AI. The SaaS agent looked affordable per call and was underwater per outcome. The legal agent looked expensive per call and was one of the best returns in the building. Neither of those truths is visible without the metric, and getting them backwards, which the per-call dashboard invites you to do, is how companies throttle the agents they should scale and scale the agents they should throttle.
Instrumenting it without a rebuild
The practical objection I hear is that this sounds like a measurement project the team does not have time for. It is lighter than it looks, because most of the inputs are already being logged.
The cost side needs three things joined on a single identifier. The per-attempt inference cost, which your provider billing or an instrumentation layer already captures at the call level. The outcome label for each attempt, which is whether the attempt produced a successful business outcome, escalated, or failed. And a stable identifier that ties every call in a multi-step attempt back to the single attempt it belonged to, so the reasoning passes and tool calls and retries of one ticket sum into one attempt cost rather than scattering across the dashboard as unrelated calls. That trace identifier is usually the one piece teams have not wired, and it is a small addition to instrumentation you are mostly already running.
The value side needs one number per workflow: what a successful outcome is worth, defended honestly. For a workflow that displaces human labor, it is the loaded cost of the human alternative. For a workflow that avoids a loss, it is the expected value of the loss avoided, taken from your own historical data rather than a vendor’s benchmark. It does not need to be precise to three decimal places. It needs to be defensible enough that the comparison to cost per successful outcome means something in a review. A rough but honest value number beats a precise cost number with nothing to compare it against.
Once those are in place, cost per successful outcome is a single query: total agent spend over a window, divided by successful outcomes in that window, held up against the value line. The teams that run this number monthly stop being surprised in review meetings, because they are watching the metric that the review meeting is actually about.
The metric I should have included
The twelve-metric harness answers whether an agent is behaving correctly. That is a necessary question and I would not remove a single metric from it. But it is half the question production teams actually face, and the framework presented the half as the whole.
The complete question has a second clause. Is the agent behaving correctly, and does a successful outcome cost less than it is worth? An eval that measures the first clause and ignores the second will keep passing agents that finance later kills, and the engineers who built those agents will keep being surprised in review meetings by a number that was knowable all along.
If I were publishing the harness today, cost per successful outcome would be the thirteenth metric, and it would not be an afterthought at the bottom of the list. It would sit next to task completion, because in 2026 the two together decide whether an agent is a capability or a liability. Accuracy tells you the agent can do the work. Cost per successful outcome tells you whether the work, done this way, is worth doing. The agents that survive the year are the ones where both answers are yes, and the fastest way to find out which agents those are is to measure the number before the CFO does.
Pratik Rupareliya is Co-Founder and Head of Strategy at Intuz. He has spent 18+ years deploying enterprise AI, IoT, and cloud platforms into production across 700+ projects. He writes on the economics of AI at scale for practitioners. What works, what fails, and where the budget actually goes.