Agreement Is Not Accuracy: The Evaluation Blind Spot
Escalation rate, override rate, and disagreement rate all measure the system against the humans checking it. A change that is slow or small escapes all…
The evaluation loop is the fifth component of the guardrail layer: the escalation rate, the override rate on cases the system resolved on its own, and the disagreement rate between the system and the humans reviewing it. Those are the right instruments. All three compare the system to its reviewers, so all three measure one quantity, and the quantity is not accuracy. It is agreement. They come apart in a predictable set of cases, and the dashboard stays green through every one. Model drift is the failure the loop was built for, and the loop is good at it. When classification quality degrades, reviewers see outputs they would not have produced themselves, and they correct them. The override rate climbs. The disagreement rate climbs with it. The correction is a reflex, and the reflex is what the instrument reads. All three numbers compare the system to the people checking it, which makes the reviewer the reference standard. That holds only while the reviewer is independent of the thing being measured. A person who opens a case, reads the extracted data and the confidence score, and then decides has been handed an answer before forming one. Doing that every working day is training. The loop also feeds every human judgment on an escalated case back into the scoring model, which is what keeps the escalation rate falling. Each side is trained on the other. That expands coverage. It also disqualifies their agreement as evidence that either one is right. The reference is not independent. Rule drift runs the opposite way. The model does not change. The business does. A category the risk committee treated as acceptable in March gets tightened in July. A counterparty stops accepting a document type that satisfied the requirement all of last year. Neither reaches the build team as a change request, because to the people making the change it is not a system change. It is how the work is done now. Reviewers absorb a change like that informally, from a huddle, from one person telling the next. They absorb it unevenly, so the team runs several versions of the rule at once. Their own standard moves, incompletely, and their sense of the correct answer was already shaped by months of reading the system's output. Both end up off the actual rule, in the same direction, and the gap between them stays small. Agreement is highest exactly when both are wrong together. The number built to detect the problem is now the evidence that no problem exists. What decides whether an instrument sees a change is not which side moved. It is whether the change is large and fast relative to the noise the reviewers already generate. Rule drift escapes by being slow. A vendor model update escapes by being small, and it is a model change. The vendor ships on its own schedule, so behavior moves with no deployment and no commit to diff. Assume, for the arithmetic, that classification quality drops four points. Nothing errors. No threshold is crossed, because the threshold reads a confidence score, and a confidence score does not necessarily fall when accuracy does. The override rate moves by less than it moves between an ordinary Monday and a holiday Friday, and nobody runs a significance test on an operational metric. The first signal comes from a customer, weeks later. What catches both is a frozen evaluation set. A few hundred cases with known-correct outcomes, held outside the production path, re-runnable on demand and re-run on a schedule. Known-correct does not mean a reviewer agreed at the time. It means the outcome was verified afterward, downstream, where the consequence landed. In the onboarding system that corpus already existed. The 14 months of historical adjudication data that set the confidence threshold is a labeled corpus, and it is the seed for the suite that tests the system built on it. From there the set only grows. Every case the system got wrong is added permanently, with the verified outcome attached, on the day the error is found. A sample tells you how the system performs on representative cases. A suite that has absorbed every past failure tells you whether a change brought back a mistake the business already paid to learn. The piece that catches a moved rule is the piece that gets skipped. Re-derive the labels periodically from downstream outcomes rather than from reviewer agreement. Pull the cases the system decided two quarters ago and look at what happened afterward. The approval that was reversed. The account that came back. The file the counterparty rejected. Where the verified outcome disagrees with the stored label, two things are possible. Either the decision was wrong when it was made, in which case it is an error and belongs in the suite permanently. Or the rule moved after the label was written, and a slice of the suite is now enforcing a standard the business has abandoned. Shape tells them apart. Errors scatter. A rule change clusters in one category or one document type. When they cluster, re-label the slice and date the change, because a suite that fails for a correct reason is a suite somebody mutes. Re-derivation only works where the consequence became visible, and it is visible almost entirely on the cases the system approved. A rejection generates nothing. The applicant who was declined never comes back to prove they would have been a good customer, so the false negatives never enter the corpus and the suite gets steadily better at measuring one half of the error surface. Pull a fixed share of rejections for manual adjudication on a schedule and treat the cost as the price of the instrument. In our engagements the evaluation loop gets specified during the build, while the people who wrote the thresholds are still in the room, and it is the component most likely to be orphaned at the handover. The evaluation loop is the one component with no daily operator. The escalations have a queue, the audit trail has an auditor, and the kill path has a named owner. The evaluation suite has whoever remembers it exists. Six months later it is a folder. The system runs, and nobody in the building can say whether it is still right. So the harness ships as a client-owned artifact on the same terms as the rest of the system: in their repository, runnable on one command, producing a single page that names the cases that changed answer since the last run. Run it before every model version change, after every business rule change, and on a fixed schedule when nothing has changed at all. The schedule is the only one of the three that catches a change nobody announced. A team that cannot re-run the suite is holding a system it can operate and cannot verify. We specify the evaluation harness in the same weeks as the confidence thresholds, because the labeled corpus that sets a threshold is the corpus that tests it later. The suite ships into the client's repository with an owner and a cadence attached, and the team runs it once before we leave. What that buys is narrow: an answer, next quarter, to a question that otherwise has none.