They had a written bar that every AI change was required to clear before shipping. They had no way to tell whether a change cleared it. For eleven months, the gate had approved everything.
The VP of Engineering booked the call because releases had become frightening. Their in-product drafting assistant was live with about forty thousand monthly users, and roughly once a quarter a change would land that made outputs noticeably worse. They always found out the same way: a customer told them.
Their theory was that they needed more QA. They were preparing to hire two contract testers.
The pre-audit told a different story. They scored well on governance: a defined release bar, a documented rollback, a kill switch someone in support could actually pull, and zero on evaluation. Every control they had built was a control over a process that measured nothing.
Their release checklist contained the line "model output quality verified." In practice, verification meant an engineer opening the staging environment, trying four or five prompts they had in their head, and deciding it seemed fine.
That is not a weak test. It is an absence of a test wearing the costume of one, and it is worse than having no checklist, because the checklist created the belief that quality had been checked. Nobody escalated a concern, because on paper the box had been ticked.
Two contract testers would have made this failure mode more expensive and no less likely. You cannot staff your way out of not having a definition of correct.
The prior quarter's regression illustrated it exactly. Someone had changed a prompt to reduce verbosity. It worked. It also caused the assistant to drop the caveat sentence on legally-sensitive drafts, roughly one time in nine. No manual spot check of five prompts would ever have surfaced a one-in-nine failure on a narrow slice of traffic, and none did. A customer's legal team found it.
Synthetic test cases test the cases you thought of, which are the cases you already handle. I built the evaluation set from production logs instead:
The instinct is to ask a model to score outputs one to ten. That produces a number that moves and cannot be acted on. I split the criteria by what kind of check each one needs:
| Criterion | Grader | Why |
|---|---|---|
| Required caveat present on sensitive drafts | Deterministic | It is a matching problem. A model is slower, costlier and less reliable than a rule. |
| Output schema and length bounds | Deterministic | Same. |
| No customer PII echoed into the draft | Deterministic | Never delegate a compliance check to a probabilistic judge. |
| Factual consistency with source document | LLM judge | Genuinely requires reading comprehension. |
| Tone appropriate to document type | LLM judge | Irreducibly subjective; a rubric plus examples does better than a rule. |
Roughly two-thirds of what they cared about turned out to be deterministically checkable. That is typical, and it is good news: those checks are fast, free and never drift.
I calibrated the LLM judge against the human labels before trusting it, and reported the agreement rate alongside every result. A judge that agrees with your humans 71% of the time is not a quality gate. It is a random number generator with good manners. Ours landed at 89% after two rubric revisions, and I published that figure on the dashboard so nobody forgot it was a measurement with error bars.
Their first instinct was to grade with the same model family that generated the output. This is the most common mistake in LLM evaluation and the hardest to detect, because a model and its judge share failure modes. If the generator finds a particular hallucination plausible, its sibling judge tends to find it plausible too. The suite goes green. The blind spot is invisible precisely where it matters, and every result it produces is quietly optimistic.
I replaced the single judge with a three-model council drawn from different vendors, each scoring independently against the same rubric, with no visibility of the others' verdicts:
# per subjective criterion judges: 3 models, 3 different families verdict: majority disagreement: routed to a human, always # disagreement is the product, not the noise
The council does three things a single judge cannot:
Cost was the objection, and it evaporated on contact: three judges over 400 examples runs about six dollars. One shipped regression costs more than a decade of that.
Councils reduce human review; they do not remove it, and a design that claims to should be treated with suspicion. What changes is where the human is spent. Before, an engineer glanced at five prompts and certified the whole release. After, humans see only:
Every one of those human decisions becomes a permanent label. The loop is closed: the set gets better because people are reviewing the hard cases, and it gets better fastest at exactly the edges where the automated graders are weakest. A review process that does not write its verdicts back into the evaluation set is not human-in-the-loop. It is just humans, in the way.
A single pass/fail bar is either so low it never fires or so high it blocks legitimate work. I set two:
# blocks the merge absolute_floor: caveat_present_on_sensitive: 100% # compliance — no tolerance pii_leak: 0 # compliance — no tolerance schema_valid: 99.5% regression_tolerance: # vs current production, per criterion factual_consistency: -2pp # a 2-point drop fails the build tone_appropriate: -3pp
The absolute floor encodes things that must always be true. The regression tolerance catches the far more common failure: a change that is not broken, merely slightly worse, shipped fifteen times in a row until the product is materially degraded and no single release is to blame.
The suite runs on every pull request touching a prompt, a model version, a retrieval config or a post-processing step, and it fails the build. An evaluation dashboard nobody is required to read is a dashboard nobody reads. This was the shortest conversation of the engagement and the one that mattered most.
A routine model version upgrade: the vendor's newer, better, cheaper model. Every manual check passed. It looked like an obvious win.
The suite failed the build on one criterion:
| Criterion | Production | Candidate | Δ |
|---|---|---|---|
| Factual consistency | 94.1% | 95.8% | +1.7pp |
| Tone appropriate | 91.0% | 92.4% | +1.4pp |
| Schema valid | 99.9% | 99.9% | — |
| Caveat present (sensitive) | 100% | 96.3% | FAIL |
The newer model was better at almost everything and had quietly become more willing to omit a boilerplate legal sentence it evidently judged redundant. On the old process this ships. It is better on every number anyone was looking at, and the failure surfaces months later through a customer's legal department.
They kept the upgrade. It took two days to add the caveat to the post-processing step rather than relying on the model to remember it, which is where a compliance requirement should have lived from the start.
| Phase | Elapsed | What happened |
|---|---|---|
| Golden set | 3 weeks | Sampling, labelling, rubric arguments. The slow part, and not compressible. |
| Graders | 2 weeks | Deterministic checks first, then judge calibration. |
| Thresholds & CI | 1 week | Wiring, failing builds, tuning the noise down. |
| Handover | 1 week | Their engineers adding criteria unsupervised. |
Ongoing cost is about forty dollars a month in judge inference and roughly a day a quarter refreshing the golden set as traffic shifts. They did not hire the two contract testers.
A launch gate with no measurement behind it is not a weak control. It is a mechanism for manufacturing confidence, and it will faithfully record that it approved the change that hurt you.
The Pilot-to-Production Audit answers that in two to three weeks, per pilot, in writing. The triage call before it is thirty minutes and free, and plenty of callers leave it knowing they don't need an audit.
Book a triage call