LLM-as-Judge Reliability: Calibrate the Scorer Before You Trust the Score

You cannot trust an LLM-as-judge out of the box — it will flatter bad agent runs if you never calibrate it. Treat the judge as a noisy instrument: measure agreement with humans, kill known biases, and re-validate after every model or rubric change. A high judge score with unmeasured calibration is a vanity metric with better fonts.

This is meta-eval. How to build the evaluator stack lives in evaluators before agents and the evaluator is the product. This spoke asks whether the scorer itself is lying. Parent context: Agentic Systems Operating Manual.

The short answer

• Mechanical checks first; judges only where humans would disagree on soft criteria.
• Calibrate against a labeled set before the judge gates CI or production.
• Agent trajectories break naive “grade the final answer” judges — score tools and intermediate claims too.
• Watch position bias, verbosity bias, and same-model circular scoring.
• Re-run calibration when the worker model, judge model, or rubric changes — judge drift is real.

What LLM-as-judge means in an agent harness

In an agent eval harness, the judge is a second model call (or panel) that scores a run against a rubric and returns structured verdicts: pass/fail, criterion codes, short evidence quotes.

It is not:

• A replacement for schema validation
• Ground truth by virtue of being “smarter”
• Safe to share weights casually with the worker without measuring circularity

Typical placement:

1. Worker run completes (or hits a checkpoint)
2. Mechanical checks run (schema, allowlist, required artifacts)
3. Judge scores remaining soft criteria
4. Harness maps scores → pass / revise / escalate / fail

If step 2 is empty, you are paying a judge to do string checks. Stop.

Why agent trajectories break naive judges

Chatbot judges often see: prompt, final answer, rubric. Agent runs add tools, multi-turn state, and side effects. Failure modes unique to agents:

Failure	What a naive judge misses
Wrong tool, right-looking final text	Grades the essay, ignores the CRM write
Hallucinated tool success	Believes the worker’s narration over the tool ledger
Criterion satisfied mid-trace then undone	Scores the last message only
Policy near-miss	“Sounds careful” while an irreversible tool nearly fired

Research on agentic rubric verification (including RuVerBench-style work in 2026) keeps finding noise when judges score complex trajectories. Practical takeaway: feed the judge a structured trace digest, not a chat dump — tool names, redacted args/results, state transitions, and the final artifacts.

Do not invent a single published “accuracy %” for all judges. Your calibration numbers are the only ones that count for your rubric.

Calibration against human labels

Procedure that actually moves reliability:

1. Sample 50–100 runs covering pass, fail, and escalate (stratify by job_type).
2. Blind-label with humans using the same rubric the judge will see. Two raters when stakes are high; resolve disagreements explicitly.
3. Run the judge on the same packages; store scores + evidence.
4. Compute agreement: per-criterion accuracy / F1, overall pass-fail agreement, and confusion pairs (judge pass / human fail is the dangerous cell).
5. Tune rubric wording, evidence requirements, and which criteria stay mechanical.
6. Freeze a calibration report with judge model id, rubric version, and date.

Metric	Why it matters
Human–judge pass/fail agreement	CI gate sanity
False pass rate (judge pass, human fail)	Customer risk
False fail rate	Cost / latency from over-refusal
Per-criterion agreement	Finds broken rubric lines

Hedge, not folklore: “good enough” for a soft-launch gate is often in the ballpark of strong majority agreement on pass/fail for your risk class — but you set the threshold from blast radius, not from a blog’s lucky number. Irreversible tools demand tighter false-pass bounds than draft-only jobs.

Position bias and verbosity bias in agent traces

These show up differently than in pairwise chatbot evals.

Position bias: When the judge sees multiple candidate revisions or tool results in a list, earlier or later items can win unfairly. Shuffle or score candidates independently when you compare revisions.

Verbosity bias: Long, confident worker narrations score higher than terse correct tool use. Countermeasures:

• Require evidence quotes tied to tool ledger ids, not vibes
• Cap narrative length in the judge package
• Score “correctness of actions” separately from “quality of prose”
• Penalize unsupported claims explicitly in the rubric

Bias	Symptom in traces	Mitigation
Position	Revision A always wins when listed first	Independent scoring / shuffle
Verbosity	Wordy fails beat short passes	Evidence-first rubric
Authority	Judge trusts “I verified via CRM” without tool span	Ledger required
Leniency	Soft criteria always “mostly met”	Binary criteria + examples

If your judge prefers essays, your agent will learn to write essays instead of calling tools correctly.

Same model as worker — ever OK?

Sometimes, for low-stakes draft scoring in staging. Rarely for production gates on irreversible work.

Risks:

• Shared blind spots (both miss the same policy hole)
• Style favoritism (worker prose matches judge priors)
• Correlated drift when the provider updates the family

Setup	Use when
Same model family, same pin	Cheap staging smoke only
Same family, different pin / size	Acceptable if calibrated; still watch circularity
Different vendor for judge	Prefer for production gates when cost allows
Panel (2 judges + tie-break rule)	High blast radius criteria

Capability language beats fashion: pick a judge that follows rubrics and returns structured JSON reliably. Pin the id. Re-calibrate on change.

Detecting judge drift

Judge drift is a silent production bug: worker prompts unchanged, online “pass rate” climbs or collapses, humans still rewrite.

Triggers that force a re-calibration run:

• Judge model pin or provider snapshot changed
• Rubric version bumped (even “clarifications”)
• Worker model upgraded (distribution of traces changes)
• New tool or side-effect class added
• Human override rate diverges from judge pass rate for two weeks

Drift checks:

1. Hold out a frozen gold slice (never used for prompt tuning).
2. Weekly or on deploy: score the slice; alert if agreement or false-pass rate moves past your band.
3. Sample online disagreements (human reject after judge pass) into the next calibration set.

Judge spans belong on the trace beside tool calls — same run id, same weekly ritual as the rest of the control plane.

Mechanical checks that should replace the judge

Move these out of the LLM judge entirely:

Check	Why mechanical
JSON / schema validity	Binary, cheap
Required fields present	Binary
Tool allowlist / deny list	Policy, not taste
Max turns / budget exceeded	Harness facts
Forbidden strings / PII patterns	Regex or classifiers
Idempotency key present on writes	Ledger fact

Judges earn their tokens on: tone, completeness vs a messy brief, “did the research address the question,” soft brand constraints. If a criterion can be a unit test, make it a unit test.

Failure mode: correlated easy-case accuracy

What breaks: your calibration set is 80% obvious passes. Judge–human agreement looks excellent. Production is the hard 20%. The judge rubber-stamps fluent wrongness.

What it costs: CI stays green while revision rate stays ugly — the pass-rate lie with a judge costume.

What you do instead:

1. Stratify the labeled set by difficulty and failure code.
2. Track agreement on the hard stratum separately.
3. Keep a rising share of production disagreements in the set.
4. Never celebrate aggregate agreement alone.

Easy cases are where judges look smart. Hard cases are why you hired them.

Rubric design that survives agents

Rules of thumb for judge-ready rubrics:

1. One criterion, one failure code.
2. Each criterion names observable evidence (artifact field, tool result, quote).
3. Include 2–3 positive and negative exemplars per soft criterion.
4. Separate “process” criteria (allowed tools, no speculative writes) from “outcome” criteria (user gets value).
5. Version the rubric (rubric_id, semver). Store it on every judge span.

Bad criterion: “Be helpful and accurate.”
Better: “Every numeric claim in the customer email appears in tool:billing.get result or is marked uncertain.”

CI gating without false comfort

Suggested promotion ladder:

Gate	Judge role
PR / prompt change	Mechanical + judge on golden set; block on false-pass regressions vs baseline
Staging soak	Online sample; compare human spot-checks
Soft-launch	Judge advisory or dual-run; humans still own irreversible tools
Autonomy expand	Judge gate only after calibration report signed off

Agreement thresholds are a product decision. Document them next to blast radius. Do not copy a research paper’s headline number into your runbook without re-measuring on your traces.

What this post does not replace

Spoke	Owns
Evaluators before agents	Build order: criteria → mechanical → judge → online
The evaluator is the product	Why eval quality is the product surface
This post	Meta-eval: is the judge calibrated and stable?

If you skip the first two and only add a judge prompt, you have cosplay.

Pilot slice: calibration in five days

A Spurlock Studios pilot can include a thin meta-eval pass when the job already has soft criteria:

Day	Judge work
1	Split mechanical vs judge criteria
2	Label 30–50 runs (or dense fixtures)
3	First judge pass + confusion matrix
4	Rubric surgery; kill verbosity loopholes
5	Freeze rubric_id + pin; wire judge span to traces

You will not finish academic-grade inter-annotator studies in five days. You will know whether the judge is roughly usable or actively dangerous. Book via /agentic.

Anti-patterns

“The flagship model is the judge, so we’re fine.” Capability helps; calibration decides.

Judge sees full chain-of-thought and grades style. Prefer actions + artifacts; CoT as optional debug, not scoring fuel, unless you measured that it helps.

One giant “quality 1–5” score. Un-actionable. Prefer criterion codes ops can fix.

Recalibrating never. Then your dashboard is a fiction that ages.

Worked example: support draft agent

Criterion	Judge or mechanical?
Contains order id from ticket	Mechanical
No refund promise unless tool says eligible	Mechanical on tool + regex
Tone matches brand examples	Judge
Answers all explicit customer questions	Judge with checklist from ticket

Illustrative pattern (not a universal stat): judge passes “tone” on long drafts; humans fail short correct ones. Fix: verbosity penalty + exemplar shorts; require factual claims to cite tool:orders.get. That bias fix often beats swapping judge vendors.

Evidence package minimum: job goal, rubric version, final artifacts, tool ledger digest, harness terminal reason. No secrets, no giant RAG dumps. No ledger → you are grading creative writing.

Panel judges: only when blast radius is high and single-judge false-pass stays above band after rubric work. Independent scores, predefined tie-break; skip multi-agent debate theater as the SMB default.

FAQ

When should mechanical checks replace a judge entirely?

Whenever the criterion is binary and observable without taste: schemas, allowlists, budgets, required ids, forbidden actions. Judges are for soft criteria. If your entire rubric is mechanical, delete the judge call and celebrate the latency win.

Should the judge see the worker’s chain of thought?

Default no for scoring. CoT invites style grading and leaked rationalizations. Prefer tool ledgers and artifacts. If you experiment with CoT-in-the-judge-package, A/B it on your labeled set — keep it only if false-pass rate improves.

Same model as worker — ever OK?

For low-stakes staging or draft-only jobs after calibration, sometimes. For production gates on irreversible tools, prefer a different model family or a panel, and always measure circular agreement on hard cases.

What agreement rate with humans is “good enough” to gate CI?

Whatever bound matches your blast radius — documented, measured on a stratified labeled set, with special attention to false passes. There is no universal published percentage that absolves you from measuring on your rubric and traces.

How do position and verbosity bias show up in agent traces?

Position bias skews revision tournaments; verbosity bias rewards long narrations over correct short tool use. Mitigate with independent scoring, evidence-first rubrics, and ledger-linked claims.

How does this relate to evaluators-before-agents without replacing it?

Evaluators before agents tells you to build the eval stack before autonomy. This post assumes that stack exists and asks whether the LLM judge component is calibrated. You need both: a real evaluator, and a scorer you have meta-evaluated.

CTA

Calibrate the scorer before you trust the score.

/agentic · /contact?intent=agentic-pilot