Spurlock Studios
Contact
Tag

evals

Posts tagged evals.

Why Pass Rate Lies: Revision Rate, Trajectories, and Coverage AI Agents

Why Pass Rate Lies: Revision Rate, Trajectories, and Coverage

Pass rate flatters bad agents. Gate deploys on revision rate, trajectory scores, eval coverage, and cost per successful task—not a single green percentage.

11 MIN
Golden Sets from Production Failures: Turn Bad Runs into Regression Fuel AI Agents

Golden Sets from Production Failures: Turn Bad Runs into Regression Fuel

Turn a bad production agent run into a regression test by harvesting the trace into a stubbed fixture with a clear pass criterion.

11 MIN
Pin the Model, Gate the Upgrade: Catch Agent Drift Before Customers Do AI Agents

Pin the Model, Gate the Upgrade: Catch Agent Drift Before Customers Do

Yes—pin production agents to explicit model IDs. Floating aliases change behavior with no deploy. Upgrade only through a golden-set gate you actually run.

11 MIN
LLM-as-Judge Reliability: Calibrate the Scorer Before You Trust the Score AI Agents

LLM-as-Judge Reliability: Calibrate the Scorer Before You Trust the Score

An LLM judge is a noisy instrument, not ground truth. Calibrate against human labels, kill position and verbosity bias, re-check after model or rubric changes.

11 MIN