A support-triage classifier we had shipped quietly got worse for about six weeks. Nobody noticed. There was no error, no alert, no exception in any log — tickets were still being categorised, the API still returned 200, the dashboard still showed throughput.
What we eventually found was that someone had appended two sentences to the system prompt during an unrelated hotfix to handle one awkward edge case. Accuracy on everything else dropped roughly eleven points. We discovered it because a support lead mentioned, offhand, that billing tickets seemed to be landing in the wrong queue "recently."
Six weeks. Because there was nothing in the system capable of telling us.
That is the gap that evals and LLM observability fill, and it is the least glamorous and most load-bearing part of shipping AI features. Traditional monitoring answers "did it respond?" With a language model, it always responds. The question is whether the response was any good, and that needs different machinery.
Start With Twenty Examples in a JSON File
The single biggest reason teams have no evals is that they imagine an eval framework. They picture infrastructure, a scoring service, a dashboard, and then never start.
The version that works is embarrassingly simple:
[
{
"id": "billing-refund-01",
"input": "I was charged twice for March and support hasn't replied in 4 days",
"expected": { "category": "billing", "urgency": 4, "needs_human": true }
},
{
"id": "ambiguous-02",
"input": "the export is broken and also can you change my plan",
"expected": { "category": "bug", "needs_human": true },
"note": "two issues; classify the one the user is angriest about"
}
]
Thirty of those and a script that runs them, prints a score, and diffs against the last run. That is a full afternoon's work, it needs no new services, and it would have caught our eleven-point drop the day it happened.
The cases that earn their place are the awkward ones. A file full of obvious inputs tells you nothing you did not already know. Every production bug you fix should become a case — that is how the set grows into something valuable rather than something you assembled once.
Grading Without Fooling Yourself
How you score depends on what the output is, and mixing these up is where eval efforts go wrong.
Exact or structural match for anything with a right answer — a category, an extracted field, a boolean. This is most production AI, and it is the easy case. Use it wherever you can.
Assertions for open-ended output where full grading is overkill but specific properties must hold. Does the answer cite a source? Is it under 100 words? Does it avoid naming a competitor? Did it refuse when it should have refused? Cheap, deterministic, and catches a surprising amount.
A model as judge for genuinely subjective quality. This works, with caveats people ignore. Give the judge a rubric rather than "rate this 1-10" — scores without criteria are noise. Ask for pairwise comparison ("is A or B better for this rubric?") rather than absolute scoring, which is far more stable. And use a different model as the judge than the one being judged, because models rate their own output generously.
Whatever you use, spot-check the grader against your own judgement before trusting it. A judge that disagrees with you is worse than no eval, because it produces confident numbers.
Run It Where Changes Happen
The eval set is worthless if it runs when someone remembers. Ours runs on any pull request that touches a prompt file, a model config, or the retrieval code, and it posts the delta as a PR comment:
eval: support-triage 42 cases
accuracy 0.93 → 0.81 ✗ -12pts
needs_human 0.88 → 0.86 -2pts
avg tokens 840 → 1210 +44%
cost / 1k $2.1 → $3.0
regressions: billing-refund-01, ambiguous-02, +3 more
That is what would have stopped the hotfix. Not a policy, not a review checklist — a number in front of the person merging, at the moment they merge.
Tokens and cost belong in the same output deliberately. A change that gains two points of accuracy while doubling spend is a decision someone should make consciously, and it is invisible unless both numbers sit side by side.
What to Log in Production
Evals cover what you thought to test. Observability covers everything else, and the two answer different questions.
For every call I want: the full assembled prompt, the raw response, model and parameters, token counts split by input/output/cached, latency, cost, and the IDs that let me trace it back — user, tenant, feature, request. Plus, if the feature has a review step, whether a human accepted or edited the output.
That last field is the most undervalued signal in AI products. Human edit rate is a free, continuously-updating quality metric. If reviewers accepted 85% of drafts last month and 60% this month, something changed, and you know it without running a single eval.
The alerts I actually keep, having tried and deleted many:
Schema validation failure rate. The model returning malformed output is the clearest signal something shifted.
Refusal or fallback rate. A spike means inputs changed or a guardrail is over-triggering.
Average input tokens. Creeping context is both a cost and a quality problem, and it always creeps.
Retrieval hit rate, on RAG features — how often anything scored above threshold. Falling means the corpus and the questions have drifted apart.
p95 latency. Ordinary, still matters.
Note that none of these need a human to grade anything. They are all mechanical, they run continuously, and they catch most real incidents.
Tracing Agents
For multi-step systems, per-call logging is not enough — you need the run as a tree. Which step called which tool, with what arguments, what came back, how the state changed, where the token budget went.
OpenTelemetry works for this and the ecosystem has converged on conventions for it, which means you can use your existing tracing backend rather than adopting a separate AI-specific product. When a user reports "the agent did something strange," you open one trace and read the actual sequence instead of reconstructing it from log lines.
The Part That Is Not Technical
Every team that gets this right does one thing culturally: prompts live in version control and change through pull requests. Not in a database field, not in an admin UI, not pasted into a config at 2am.
The moment a prompt is a code artefact, it inherits review, history, and the eval gate. Our six-week regression was possible because the prompt was editable in a way that bypassed all three. Fixing that was one afternoon of moving strings into files, and it is the change I would make first on any AI codebase I inherited.
