“It looked good in the playground” is not a release criterion. Neither is “the PM liked the demo.” Production AI fails in boring ways: a prompt tweak that silently drops a required field, a model swap that sounds smarter but skips citations, a retrieval change that passes the eyeball test and fails on edge-case tickets from last quarter.
You need a golden set — 30–100 real prompts with expected behaviours — and you need to run it on every prompt or model change. Not quarterly. Not “before the big launch.” Every change.
Minimum viable evals
- Exact checks for structured outputs: JSON schema, tool names, enum values. These are cheap and they catch the failures that matter in production.
- LLM-as-judge only where humans calibrated it — and spot-check the judge monthly. An uncalibrated judge is vibes with extra steps.
- Regression gates in CI: fail the build if quality drops beyond a threshold you agreed on in advance.
- Track cost and latency beside quality. A “better” model that 5× the bill can still fail the product.
That is the floor. Most teams stop there and wonder why quality still drifts. The ceiling is a living eval set sourced from production failures — but you cannot get there without the floor first.
What we wire into Forge builds
Every Forge engagement ships with the same eval skeleton: a folder of test cases, a runner script, and a CI job that blocks merge on regression. Prompts live in versioned markdown; evals reference prompt version and model ID so a failure tells you what changed, not just that something broke.
We split checks into three tiers. Tier one: deterministic — schema, regex, exact string for IDs and error codes. Tier two: retrieval — did the answer cite the right document chunk? Tier three: subjective — tone, completeness, “would a support lead accept this?” Tier three uses LLM-as-judge sparingly and always with human-labelled examples in the calibration set.
Cost and latency ride alongside quality in the same report. If evals only measure “sounds good,” you will ship a model upgrade that destroys margin and nobody will notice until finance asks.
Building the golden set without an ML platform org
You do not need a labelling team. You need three people and one afternoon:
- Pull the last 50 real user questions from support logs, Slack, or ticket exports. Redact PII.
- Write the expected behaviour in plain English — not the exact output, the behaviour. “Must cite policy doc X.” “Must refuse if account ID missing.”
- Run the set manually once. Fix the obvious failures before you automate anything.
Refresh monthly from new failures. An eval set that never changes is a snapshot of problems you solved six months ago.
When vibes are enough — briefly
Internal-only tools with no customer-facing output and a human in the loop on every action can survive on spot checks longer than anything customer-facing. Even then, we still run schema evals on tool calls. The model can hallucinate a parameter; your billing API does not care that it sounded confident.
Everything else — Embed workflows, Edge-facing agents, anything that sends email or moves money — gets the full harness before it goes live. Vibes are for exploration. Evals are for shipping.