Why Agent Demos Die in Production: Control Gaps, Not Model IQ

Your AI agent works in the demo but fails in production because the demo optimized for a staged happy path, not a control loop. The model did not get dumber overnight. Production introduced real schemas, real tenants, real tool failures, and real side effects — and your harness never owned those constraints.

This spoke sits under the Agentic Systems Operating Manual. Pair it with when not to build an agent and agent pilot scope before you widen autonomy.

The short answer

• Demos prove “can it look smart once?” Production asks “can it stay correct under drift, load, and bad tools?”
• Most “hallucinations” in launch week are system bugs: schema drift, stale tool results, authorization bleed, cascade after one bad tool call.
• Readiness is a checklist on the harness — evaluators, sandboxes, reason codes, kill switches — not a higher model tier.
• Soft-launch without a reliability audit is how you buy expensive chaos.
• Fix the control loop first; then argue about prompts.

What a demo optimizes for that production doesn’t

A demo is a theater set. Inputs are curated. Tools return clean JSON. Credentials are a single sandbox tenant. Latency is low. Nobody else is writing to the CRM while the agent runs. The audience watches one path succeed.

Production is adversarial by accident:

Demo assumption	Production reality
Fixed tool schemas	Vendors ship breaking field renames
One tenant, one role	Multi-tenant auth with bleed risk
Tools always succeed	Partial failures, timeouts, empty results
Single operator watching	Overnight runs, no human in the room
“Looks right” is enough	Evaluator criteria or customer complaint

If your success metric was applause, you measured the wrong thing.

Control gaps, not model IQ

When the demo dies, the instinct is to swap models or rewrite the system prompt. That treats intelligence as the bottleneck. For business agents, the bottleneck is almost always the control loop: plan → act → evaluate → revise → terminate, with deterministic guards around non-deterministic steps.

Name the gap before you touch the model:

1. No evaluator — terminal success is “HTTP 200” or “agent said done.”
2. No side-effect classes — write tools run with the same trust as read tools.
3. No schema contracts — tool args/results are free-form blobs.
4. No tenant binding — run context does not pin authorization.
5. No cascade brake — one bad tool result feeds the next plan as truth.

Upgrade the harness. Then, if quality is still soft, change the model. Blame order matters.

Failure modes that get mislabeled as “hallucination”

Treat these as system bugs until proven otherwise.

Schema drift

The tool once returned customer.email. Three months later the API returns contact.primaryEmail. The agent invents an email field that “should” exist. That is not a creative model — that is an unversioned contract.

What breaks: CRM writes with null emails, silent skips, or fabricated values.

What you do: Pin tool schemas, fail closed on unknown shapes, version adapters, and alert on parse-error rate spikes.

Stale tool results

A cache or previous span result is reused after the underlying record changed. The agent plans from yesterday’s pipeline stage and “confidently” books the wrong follow-up.

What breaks: Wrong next actions that look internally consistent.

What you do: TTL on tool results, etags or updated_at checks before writes, and span metadata that marks result_stale=true.

Authorization bleed across tenants

Demo used one API key. Production shares a worker pool. A run for Tenant A accidentally carries Tenant B’s token, or a tool accepts an id without checking ownership. The model did not “decide” to leak — the harness never enforced tenancy.

What breaks: Cross-customer reads or writes. This is an incident, not a quality ticket.

What you do: Bind tenant_id to every tool call at the harness layer; refuse tools that ignore it; test bleed deliberately (see checklist below).

Cascade failure after one bad tool result

First tool returns an empty list or a wrong match. The agent treats that as ground truth, invents a narrative, and writes it downstream. Later steps look like hallucination; the root cause was trusting a bad observation without a verify step.

What breaks: Plausible wrong CRM notes, tickets, or emails.

What you do: Require verification tools for high-stakes entities; evaluator criteria that reject “asserted without evidence”; escalate when confidence criteria fail — not when the model feels unsure.

Genuine model error exists. Lead with harness bugs first; you will be right more often.

Illustrative walkthrough: demo green, prod red

Illustrative — not a client result. Staging demo: agent looks up a lead, enriches firmographics, writes a CRM note. Tools are stubbed to always return the same Acme Corp payload. Soft-launch: real CRM has duplicate company names; enrichment returns two candidates; the agent picks the wrong one and writes a note on the wrong account.

Misdiagnosis: “the model hallucinated the company.”

Actual failure mode: no disambiguation gate, no evaluator check that crm_account_id matched the enrichment candidate ids, no escalate path for multi-match.

The fix is a control: if candidates.length != 1 → escalate, plus a golden case for duplicate company names. The model upgrade is optional.

How to run a reliability audit before soft-launch

Run this before the agent can write in production. Timebox it; do not wait for a perfect platform.

• Inventory every tool: read / write / irreversible; name the side-effect class
• Pin and version each tool schema; record adapters with dates
• Bind tenant + role to every tool invocation in the harness
• Require evaluator criteria for the job type (evaluators before agents)
• Define terminal reason codes (eval_pass, tool_auth_error, policy_violation, …)
• Kill switch that freezes writes without redeploying prompts
• Golden set with at least one case per known failure mode above
• Staging soak with real schemas (not stubs) for 48 hours
• On-call owner for freeze-writes decisions

If any checkbox is empty, you are still in demo mode with a production URL.

Measuring demo→prod readiness

Do not use “demo succeeded” as a gate. Use a thin readiness scorecard:

Signal	Demo-only smell	Ready signal
Tool fidelity	Stubs / fixtures	Live schemas + recorded adapters
Auth	Single sandbox key	Tenant-bound tokens under test
Eval	Human nods	Automated criteria + escalate
Failure drills	None	Forced bad tool + auth bleed tests
Observability	Chat console	Run ids, tool spans, reason codes
Cost / budget	Unlimited	Cap + kill switch wired

Ship when the right column is true for the job types you are soft-launching — not when the slide deck looks clean.

Edge-case inputs collapse agents for boring reasons

Demos avoid messy inputs. Production receives empty strings, HTML in “plain text” fields, dual-language names, and ids that look valid but point at archived records. Agents without input validation treat garbage as narrative fuel.

Procedure for hardening intake:

1. Define a typed intake schema per job type.
2. Reject or escalate on schema fail before any model call.
3. Normalize known dirty fields (strip HTML, trim, canonicalize phones).
4. Add golden cases from the last ten production complaints.

Bravery at intake is not a product strategy.

Soft-launch sequence that respects the gap

1. Shadow mode — agent proposes; humans execute. Score proposals offline.
2. Write with human approve — irreversible tools behind a button.
3. Narrow autonomy — one job type, one tenant cohort, hard budget.
4. Widen only after online scores hold for a defined window.

Skipping steps compresses the demo→prod gap into a single outage. For scope discipline, see agent pilot scope.

Hold each stage until you can answer: what failed, which reason code, who owns the fix. If shadow mode only produces vibes, you are still demoing.

What belongs in the five-day pilot

A Spurlock Studios $1,500 · 5-day agentic pilot is not a longer demo. Its job is to close the control gap on one real job: evaluator criteria, tool contracts, tenant binding, traces with reason codes, and a kill switch. You leave with a reliability audit trail, not applause.

Widen autonomy only after those pieces exist. Context lives in the operating manual.

Anti-patterns that keep the gap open

“We’ll add evals after launch.” Then launch is the eval — paid for by customers.

Stubbing tools forever. Schema drift never appears until it hurts.

Treating every wrong write as a prompt bug. You will rewrite prompts while the auth bug remains.

Measuring only success demos. Sample failures; force them in staging.

Confusing model upgrade with harness upgrade. Different levers, different costs.

Sharing one long-lived API key across tenants “until SSO is ready.” Authorization bleed is not a backlog item once writes are live — it is an incident waiting for a run id.

Decision list: model or harness first?

Ask in order:

1. Did a tool return an unexpected shape or error? → harness / adapter
2. Did the run cross tenant boundaries? → harness / auth (incident)
3. Did a bad observation cascade into a write? → verify gates + evaluator
4. Did criteria fail but the agent still terminated “success”? → evaluator authority
5. Only then: did the model choose a wrong plan under correct observations? → prompt / model

If you start at step 5 every time, you will never close the demo→prod gap.

Forced failure drills (staging only)

Before soft-launch, break the agent on purpose:

Drill	Inject	Expect
Schema rename	Adapter returns new field names	Fail closed + alert, no invented fields
Empty enrichment	Tool returns []	Escalate or verify — no CRM write from fiction
Wrong tenant token	Harness omits/binds bad tenant_id	Hard refuse + tool_auth_error
Mid-run 500	Write tool errors once	Bounded retry or escalate; no silent success

If a drill does not produce the expected reason code, you found a control gap while the blast radius is still staging.

CTA

Close the control loop before you scale the demo.

/agentic · /contact?intent=agentic-pilot

FAQ

Why do edge-case inputs collapse agents?

Because demos never train the harness on dirty intake. Empty fields, HTML junk, and ambiguous ids become model narrative instead of schema rejects. Validate and escalate before the first tool call; add those cases to the golden set.

How does schema drift break tools months later?

APIs rename fields and change nullability without your prompt noticing. The agent fills gaps with invented structure that “should” exist. Version adapters, fail closed on unknown shapes, and alert when parse errors spike.

What’s cascade failure after a bad tool result?

One wrong or empty tool observation becomes “truth” for later planning, so downstream writes look like hallucination. Require verification for high-stakes entities and evaluator criteria that reject unsupported assertions.

How do I test authorization bleed across tenants?

In staging, run two tenants with distinct data and deliberately swap or omit tenant_id on tool calls. The harness must refuse. Add automated cases that attempt cross-tenant reads and writes and expect hard failure plus a reason code.

Should I blame the model or the harness first?

Harness first: schemas, auth binding, evaluators, cascade brakes, kill switches. Genuine model error is real, but launch-week failures are usually control gaps mislabeled as intelligence failures.

What’s the five-day pilot’s job in closing this gap?

Install the minimum control loop on one job — criteria, contracts, tenant binding, traces, kill switch — against live schemas. The pilot proves production readiness machinery, not a prettier demo path.