Why Pass Rate Lies: Revision Rate, Trajectories, and Coverage

Pass rate looks fine while the agent is failing in ways that matter because “pass” is usually a thin binary on the final artifact. It ignores how many revisions it took, whether the tools were right, how much of the job space you never tested, and what each successful task actually costs. You need a metric panel that can veto a deploy — not a vanity percentage.

This spoke sits under the Agentic Systems Operating Manual. Observability for agents owns the dashboard and traces; this post owns which numbers actually gate a ship. Pair with the evaluator is the product when criteria themselves are soft.

The short answer

• Task success for agents means: correct outcome, acceptable trajectory, bounded cost, and human rewrite rate you can live with.
• High pass with high revision rate means the agent is grinding to green — customers feel the latency and you feel the spend.
• Score trajectories: tool choice, arguments, step count — not only the final blob.
• Eval coverage asks what fraction of real job shapes your golden set touches.
• Gate deploys on a small panel: pass, revision, trajectory, coverage, cost per success, online/offline gap.

What task success really means for agents

A CRM note can “pass” an evaluator and still be the wrong account, written after twelve tool calls, rewritten by sales, and three times the cost of a human doing it cold. Binary pass hides that story.

Define success as a bundle:

Dimension	Question
Outcome	Did criteria pass on the artifact?
Trajectory	Were tools and args appropriate?
Efficiency	Steps and tokens within band?
Human load	Did a human rewrite or reject?
Economics	Cost per successful task in band?

If you only chart the first row, your agent is optimized for looking done.

Why 90% pass can still mean heavy human rewrites

Illustrative — not a measured fleet statistic. Imagine an offline set where 90 of 100 cases meet criteria on the final artifact. On 40 of those passes, a human still edits tone, adds a missing field, or fixes a wrong link before the note goes out. Your pass rate says “ship.” Your revision and rewrite rates say “copilot with expensive thrash.”

Sources of flattering pass:

1. Evaluator too soft — criteria miss the fields humans care about
2. Pass after N revisions — counted as success with no revision penalty
3. Golden set too friendly — only happy paths
4. Humans silently fix — online truth never reaches the metric

Track human rewrite rate and agent revision depth beside pass. When rewrite rate stays high while pass climbs, you improved the judge or the grind — not the product.

Revision rate: the metric pass rate hides

Revision rate (or revision depth) asks: how many evaluate→revise cycles ran before terminal?

Pattern	Pass rate	Revision depth	Read
Clean hit	High	Low	Healthy
Grind to green	High	High	Latent failure
Early escalate	Lower	Low	Honest control
Flail then fail	Low	High	Broken loop

Gate idea: a deploy may keep pass rate flat but must not raise p50/p95 revision depth beyond an agreed band. Grind is a quality bug with a cost costume.

How to score trajectories

Trajectory scoring grades the path, not only the destination. Minimum dimensions:

1. Tool choice — required tools used; forbidden tools never called
2. Arguments — ids and filters match the job; no invented keys
3. Step count — within band for the job type
4. Order constraints — e.g. read-before-write, verify-before-irreversible
5. No-progress events — fingerprint blocks should be zero on happy paths

Simple scoring modes that work in practice:

Mode	When
Checklist pass/fail	Pilot, clear must-use tools
Weighted deductions	Mature job types with known anti-patterns
Compare to expert trace	Small golden set with recorded human paths

You do not need a research benchmark. You need “called enrichment before CRM write” as a first-class fail even when the note text looks fine.

Eval coverage: the denominator pass rate skips

Pass rate is passes / evaluated. Coverage asks evaluated shapes / shapes that appear in production.

Without coverage, you can have 95% pass on a toy set and collapse on the first weird tenant.

Build a coverage map:

• Job types in production vs job types in golden set
• Tenant size bands (solo, mid, messy CRM)
• Known failure modes (duplicates, empty enrichments, auth errors)
• Languages / locales you actually serve
• Write vs read-only paths

Report coverage % as “share of last 30 days’ production job fingerprints that match at least one golden case family.” Exact formulas vary; the point is to stop celebrating pass on an unrepresentative slice.

Cost per successful task vs cost per run

Cost per run flatters agents that fail cheap and pass expensive — or the reverse. Finance cares about cost per successful task (and per task that ships without human rewrite, if that is your bar).

Metric	Flatters	Use for
Cost / run	Cheap failures	Capacity planning
Cost / pass	Grind that eventually passes	Unit economics of “green”
Cost / shipped without rewrite	Honest automation	Go / no-go on autonomy

Illustrative arithmetic: if average cost/run is low but only one in three runs ships without rewrite, your true cost is closer to cost_per_run / ship_rate plus human time. Chart the honest number next to pass rate or you will “save money” into a support queue.

See also cost controls for agent fleets for budgets and kill switches — different lever, same economics story.

How online and offline scores diverge

Offline golden sets are stubs, frozen tools, and known answers. Online is live schemas, live latency, and distribution shift.

Typical divergence patterns:

Pattern	Offline	Online	Likely cause
Offline high, online low	Strong	Weak	Drift, stubs too clean
Both high, rewrite high	Strong	Strong	Soft criteria
Offline low, online “fine”	Weak	Strong	Prod sampling biased to easy jobs
Spike after deploy	Drop	Drop	Real regression

Rule: never promote on offline alone. Sample online through the same evaluator (observability for agents). Alert when the online/offline gap widens past your tolerance — that gap is often the first smoke of schema drift or retrieval rot.

Pass@1 vs Pass@k — which ships?

Pass@k (success if any of k samples works) is a research comfort metric. Production agents usually get one billed trajectory per job unless you explicitly budget parallel attempts.

Metric	Meaning	Ship decision
Pass@1	First trajectory meets criteria	Default gate for autonomy
Pass@k	Best of k meets criteria	Research / model compare only
Pass@1 with revisions	Success inside revision ceiling	Allowed if depth stays in band

If you report Pass@k to executives as “the agent works,” you are selling lottery tickets as reliability. Use Pass@k for model bake-offs; ship on Pass@1 (with bounded revisions) and rewrite rate.

Which metrics gate a deploy

Minimum veto panel before widening autonomy or merging prompt/tool changes:

1. Offline pass@1 — no drop beyond agreed delta on golden set
2. Revision depth — p50/p95 inside band
3. Trajectory checklist — no new systematic tool-choice fails
4. Eval coverage — not reduced; new failure modes get cases
5. Cost per success — inside band
6. Online sample — after canary, online pass and rewrite rate hold

Any single green light is insufficient. Pass rate alone is never a gate.

Weekly metric panel ops will trust

Keep this separate from the full observability wall. One screen, business-readable:

Panel	Owner acts when…
Pass@1 (online sample)	Drops vs trailing baseline
Human rewrite / reject rate	Climbs while pass flat
p95 revision depth	Crosses band
Trajectory fail codes (top 3)	Same code repeats
Cost / success by job type	Spikes after deploy
Coverage gaps (new prod shapes)	Untested families appear

Deep traces live one click away. If the weekly meeting needs a data scientist to interpret the primary screen, the panel failed.

Illustrative: green CI, angry sales

Illustrative scenario. CI shows 92% offline pass after a prompt change. Trajectory scoring was not wired. Production: agent stops calling the verify tool, still produces notes that meet soft criteria, sales rewrites account links daily. Pass rate did not lie — it answered a weaker question than the business asked.

Fix: add trajectory rule verify_before_crm_write, measure rewrite rate, block deploy on trajectory checklist regressions even when pass is flat.

Anti-patterns

Optimizing only the judge until pass hits a target. You invented grade inflation.

Averaging all job types into one pass number. A tiny easy job hides a broken expensive one.

No ownership on rewrite rate. If sales suffers in silence, metrics stay pretty.

Shipping on Pass@k screenshots. Not how production runs.

Duplicating the observability post’s dashboard here. Traces and panels are necessary; gates are the deploy policy this spoke owns.

Connecting to the observability dashboard without duplicating it

Observability stores runs, tool spans, scores, and cost. This spoke decides thresholds and veto rules. Practical split:

• Observability: emit revision count, trajectory sub-scores, rewrite flags, cost fields on each run
• Metrics / release: compare those fields to bands; block or canary
• Weekly ritual: read the panel; open traces for top trajectory fail codes

Do not build two dashboards with the same six charts. Build one telemetry path and a release checklist that references it.

Pilot minimum

A Spurlock Studios $1,500 · 5-day agentic pilot should leave you with more than a pass percentage: evaluator criteria, a small golden set with at least one trajectory checklist, revision depth on traces, and a written deploy gate. Fancy coverage math can grow later; “pass rate alone” should already be dead as a ship criterion.

/agentic · full stack in the operating manual.

Decision list: is the agent actually working?

Ask weekly:

1. Is online pass holding without a climb in rewrite rate?
2. Is revision depth stable or creeping?
3. Are trajectory fails concentrated in one tool or job type?
4. Did coverage grow with new production shapes?
5. Is cost per success inside the band you would defend to finance?

If you cannot answer from one panel, you do not know — you are hoping.

CTA

Stop shipping on a flattering percentage.

/agentic · /contact?intent=agentic-pilot

FAQ

Pass@1 vs Pass@k — which ships?

Ship on Pass@1 with a bounded revision ceiling. Use Pass@k for model comparisons and research-style evals. Production jobs rarely get k free attempts, so Pass@k overstates reliability for autonomy decisions.

What is eval coverage?

It is how much of real production job diversity your golden set and evaluators actually touch. High pass on a narrow set is not readiness — track uncovered job shapes and add cases as they appear online.

Cost per successful task vs cost per run?

Cost per run includes cheap failures and hides grind. Cost per successful task (ideally per task shipped without human rewrite) is the unit economic signal that should sit beside pass rate on the gate panel.

How do online and offline scores diverge?

Offline sets use stubs and frozen distributions; production drifts. Promote only when online samples through the same evaluator stay within tolerance of offline — a widening gap is a release smell, not noise.

Which metrics gate a deploy?

At minimum: offline pass@1 delta, revision depth band, trajectory checklist, coverage not reduced, cost per success, and a post-canary online hold on pass and rewrite rate. Pass rate alone never ships.

How does this connect to the observability dashboard without duplicating it?

Observability emits the fields (scores, revisions, trajectory fails, cost). This spoke sets the veto thresholds and weekly panel. One telemetry path, one release checklist — not two competing dashboards.