The Evaluator Is the Product

The single largest accuracy jump I have measured in an agentic system did not come from a model upgrade. It came from deleting a paragraph of a prompt and adding a second agent whose only job was to disagree.

The self-grading problem

A model asked to check its own output is not checking anything. It is generating a continuation of a context that already contains the assertion that the work is done. The self-assessment is conditioned on the work, which is exactly the correlation you were trying to break.

In practice this shows up as agents that confidently report success on tasks they did not complete. The tool call errored, the model summarised the error as a minor issue, and the run closed green.

Raw single-pass accuracy on the internal benchmark I use sat around 72%. Most of the misses were not wrong answers — they were unnoticed failures.

Separating the roles

The fix is structural. The worker produces output. A separate evaluator, with its own context containing only the acceptance criteria and the artifact, returns a verdict:

{
  "verdict": "fail",
  "criterion": "all tests pass",
  "evidence": "3 failing in auth.test.ts",
  "next": "fix the null guard in verifyToken"
}

Three things make this work:

The evaluator never sees the worker’s reasoning. It sees the criteria and the artifact. Giving it the transcript reintroduces the correlation you just paid to remove.

Criteria are mechanical wherever possible. “Tests pass” beats “code is good.” “Returns valid JSON matching this schema” beats “output is well-formed.” Anything you can assert in code should be asserted in code, and the model should only judge what genuinely needs judgement.

Failure returns evidence, not vibes. The worker cannot act on “this is not right.” It can act on “three tests fail, here is the output.”

With that loop in place and a ceiling of three correction attempts, the same benchmark runs at 99.4%. Same models. The difference is entirely architectural.

The retry ceiling matters

Unbounded correction loops are how you spend $400 discovering that a task is impossible. Three attempts, then stop and escalate to a human with the full trace. An agent that knows how to give up is more useful than one that does not, because the failure arrives while someone can still do something about it.

What this means for how you build

Most teams shipping agents are optimising the wrong surface. They iterate on the worker prompt, upgrade the model, add tools. Meanwhile there is no independent judgement anywhere in the system, so nobody can tell whether any of it helped.

Build the evaluator first. It is the only component that tells you whether the rest of the system works, and it is the one that turns a demo into something you would put in front of a customer.