Agent Quality Scorecard: A 2026 Probe Found 28 Metrics, One Threshold, and Zero Weights

An agent quality scorecard converts AI agent performance metrics into a single release decision across five dimensions: task success, grounding, tool calls, cost, and risk. Task success, tool calls, and cost carry published weights. Grounding and risk act as vetoes. Any dimension below its floor blocks the release, whatever the weighted score says.

Essential Insights

• An agent quality scorecard differs from a metrics dashboard in one respect that matters: the scorecard names which number can stop a release.
• Task success, grounding, tool calls, cost, and risk cover the five failure families that the 2026 answer pool agrees on, in five different vocabularies.
• Weights are the missing artifact in this category, not metrics; the pool publishes definitions in volume and authority almost never.
• Grounding and risk behave as gates rather than weighted line items, because a failure in either one cannot be offset by a fast, cheap, well-formed answer.
• Cost per successful task is the denominator that makes agent economics legible, since cost per run gets cheaper every time the agent gives up early.
• Tool-call correctness is countable without a judge model: correct calls divided by total calls, read off the trace the agent already writes.
• A scorecard with no named owner and no floor is a reporting habit, and reporting habits do not stop releases.

Five Dimensions Carry the Whole Decision

An agent quality scorecard scores five dimensions and nothing else: task success, grounding, tool calls, cost, and risk. Task success asks whether the job finished end to end without a human finishing it, measured as completed tasks over attempted tasks on a fixed scenario set. Grounding asks whether the output is supported by the retrieved material and by reality, which splits into two different failures: answers unsupported by the provided context, and answers unsupported by anything at all. Tool calls ask whether the agent chose the right tool with the right arguments, counted from the trace as correct calls over total calls. Cost asks what one finished unit of work costs, tokens plus paid API calls plus retries. Risk asks what the worst plausible day looks like: the blast radius of an action the agent takes wrongly, and whether that action can be reversed.

Five dimensions is a deliberate ceiling. Our own complete measurement library for this category runs far longer, because a library exists to be exhaustive. A scorecard exists to be signed. Anything past five dimensions stops being a decision instrument and starts being a monthly report that circulates, gets skimmed, and changes nothing about who owns the workflow on the Tuesday it breaks.

What the 2026 Answer Pool Publishes, and What It Leaves Out

Metric catalogs dominate this query, and the catalogs are accurate. The best of them are better than what most teams would write for themselves: real definitions, honest math, tool-call accuracy expressed as correct calls divided by total calls, and a clean separation between closed-domain hallucinations and open-domain ones. None of that is a scorecard, because none of it says which number stops a release.

Across the three page-one guides we read end to end on 2026-08-27, Monte Carlo names five metrics, Confident AI names seven, and Galileo names sixteen: twenty-eight metrics, one published numeric pass threshold (Galileo's 0.7 action-advancement line), and zero weights. That single threshold is worth quoting, since action advancement above 0.7 signals clear progress and below 0.3 signals an agent spinning without advancing. Thresholds do exist further out in the pool: DataRobot publishes production benchmarks of 85 percent or better on goal accuracy and 95 percent or better on task adherence. One page comes close to the gate model outright, treating cost, latency and step count as budgets on the trace rather than judge scores, with token ceilings, wall-clock timeouts, and a cap on tool calls per task. Credit where it is due: that is a gate in everything but name.

We published the complete measurement library for this category in June 2026, and the scorecard is the artifact that library deliberately withheld. Naming metrics is the easy half. Ranking them is the half that decides whether a workflow changes hands.

Weights Turn Five Numbers Into One Release Decision

Weighting is the step the catalogs skip, and it is the step that turns a readout into a decision. Weight task success at 40, tool-call correctness at 25, cost per successful task at 20, and latency at 15, then compute nothing at all until grounding and risk have both cleared their gates. Those four numbers are defensible arguments, not physics, and that is the point: they are written down before the run, so nobody relitigates them in the room where the release gets decided.

Each weighted dimension also carries a floor, which is a separate number from its weight. A floor is the value below which the dimension fails on its own, no matter how well the others performed. Set the task-success floor near the published production benchmarks rather than at your current score, so the floor is a standard instead of a mirror. Set the tool-call floor high, because tool errors do not stay contained: a bad call burns tokens on work that was never going to help, then feeds its bad output into every step that follows.

The weight column is where the argument happens, which is presumably why it keeps getting left out.

A metric nobody can use to stop a release is not a metric. It is a decoration with a number on it.

Grounding and Risk Are Vetoes, Not Weighted Line Items

Grounding failures and risk failures behave differently from cost failures, which is why averaging them into a composite is a category error. Grounding and risk do not belong in the weighted average at all: they are vetoes, and a veto that can be outvoted by a good latency number is not a veto. Consider the failure the category calls a ghost action, where the transcript announces a completed booking and the trace shows the booking tool was never invoked. Fluency scores well. Latency scores well. Cost scores beautifully, because a tool that never ran costs nothing to run.

Risk earns the same treatment for a different reason. Task success is a rate, so it tolerates averaging; a reversible mistake in 4 percent of runs is a number a business can price. An irreversible action taken wrongly once is not a rate, it is an incident, and the score it produces on a weighted card is indistinguishable from a rounding error. Scoring the workflow rather than the model is the whole discipline there, and it has its own instrument in our risk assessment framework.

Practically, the veto changes the order of operations. Run the two gates first, on their own evidence: grounding sampled from traces against the retrieved context, and risk scored per action against reversibility and blast radius. Only if both clear does the weighted composite get computed at all. Reversing that order produces the most familiar artifact in agent operations, which is a green composite with an ungrounded agent underneath it.

Cost Per Successful Task Is the Only Denominator That Survives

Cost per run is the number vendors report, and it is the number that flatters a failing agent. Failures are cheap. An agent that abandons a task after two calls costs a fraction of one that grinds through fourteen completions to land the right row in the right spreadsheet, so an agent getting worse can post an improving cost curve for a full quarter. Divide by finished work instead: total spend over successfully completed tasks, with retries and abandoned attempts charged to the successes they were trying to produce.

The published economics only become readable under that denominator. Fin AI's framework prices AI resolutions between $0.50 and $1.84 against $6 to $8 for a human resolution, and pairs deflection rates of 40 to 70 percent with a warning that high deflection alongside high repeat contacts is containment rather than resolution. Containment is exactly what cost per run rewards. A deflected customer who returns twice has consumed three cheap runs and produced zero resolutions, and only the successful-task denominator makes that visible as a cost increase rather than a volume win.

Two supporting numbers belong on the same line as cost, because they explain it: steps per task and tokens per task. Both come off the trace, both spike before the invoice does, and both point at the specific span that got expensive. Return on investment sits one level up from this, sorted into its own four classes in our return measurement framework.

Scorecard Versus Dashboard: Which Artifact Does Which Job

A dashboard and a scorecard answer different questions, and running one in place of the other is the common failure in this category. A dashboard reports state continuously for whoever has the tab open. A scorecard produces one verdict, on one date, for one person with the authority to halt a rollout. The comparison below sets the scorecard against the two artifacts most teams already own.

Dashboards remain useful, and a scorecard depends on one: the trace data feeding the gates comes from continuous monitoring. The distinction is authority, not instrumentation.

Where the Scorecard Breaks

An agent quality scorecard fails in three predictable ways, and all three are worth naming before anyone signs one. First, scenario drift: the fixed scenario set that makes scores comparable across releases stops resembling live traffic within a few months, and a card scored on stale scenarios certifies an agent nobody is actually running. Refresh the set from production traces on a schedule, and treat every incident as a permanent addition to it.

Second, judge drift. Grounding and task success usually get scored by another model, and an unaudited judge quietly redefines the floor it is enforcing. Keep a small human-labeled slice, re-score it whenever the judge model changes, and fix the metric specification rather than the number when the two disagree.

Third, an unowned card. A scorecard with weights, floors, and vetoes but no named person who signs it produces exactly what the unweighted dashboard produced, one document later. The oversight side of this is a separate build with its own controls, which is why agent governance covers approval gates, exception queues, and audit trails as runtime machinery rather than policy language.

None of these three failures is exotic, and none of them is a reason to skip the card. The scorecard's job is narrow: convert five families of AI agent performance metrics into one decision, on one date, in front of one person who can say no. Everything else in this category is reporting.

Frequently Asked Questions

How to measure AI agent performance?

AI agent performance is measured across five dimensions rather than one score: task success, grounding, tool calls, cost, and risk. Task success is completed tasks over attempted tasks on a fixed scenario set; tool calls are correct calls over total calls, read from the trace; cost is total spend over successfully completed tasks. Grounding and risk are scored as pass or fail gates rather than averaged in.

What metrics are used to measure AI performance?

AI performance metrics for agents fall into families that most published catalogs share under different names: outcome metrics (task success, resolution rate), quality metrics (grounding, hallucination rate, correctness), execution metrics (tool-call correctness, step efficiency, latency), efficiency metrics (tokens and cost per task), and safety metrics (policy violations, escalation rate, blast radius). A scorecard uses one measurement per family, not all of them.

How to evaluate your AI agent?

Evaluating an AI agent starts with a fixed scenario set and a trace for every run, because most agent failures hide in the execution path rather than the final answer. Score the two gates first, grounding and risk, then compute the weighted composite from task success, tool calls, cost, and latency. Publish the weights and floors before the run so the verdict is not negotiated afterward.

How to measure the performance of an agent?

Agent performance is measured per finished unit of work, not per run. Count completed tasks against attempted tasks, count correct tool calls against total tool calls, and divide total spend by successfully completed tasks so abandoned attempts are charged to the successes they were meant to produce. Latency belongs in the same set, measured end to end rather than at first token.

What is an agent quality scorecard?

An agent quality scorecard is a fixed instrument that turns agent measurements into one release decision: five dimensions, a named measurement for each, a weight for the three that average, a floor for every dimension, and a veto for grounding and risk. The scorecard carries a date, an owner, and a verdict. A dashboard carries none of those.

How does a scorecard differ from an evaluation dashboard?

A scorecard and a dashboard differ in authority rather than in data. Both read the same traces, and both can display the same numbers. Only the scorecard assigns weights, publishes floors, names an owner, and produces a dated verdict that can withhold a release.

Who should own an agent quality scorecard?

Ownership of an agent quality scorecard belongs to the person who can halt the rollout, which in a founder-led business is usually the operations lead who owns the workflow rather than the engineer who built the agent. That person signs the card at each release. Engineering supplies the traces and the scenario set.

When does an agent quality scorecard not help?

An agent quality scorecard does not help before an agent has real traffic, because its scenario set and floors are both derived from observed work. Early builds are better served by failure-mode design and a narrow scope. The scorecard becomes the right instrument once the agent is handling volume and someone has to decide whether it keeps the workflow.