The Baseline Disappears at Launch
The counterfactual disappears the moment a system goes live. The old process stops existing, and every claim about what it used to cost becomes a claim…
The counterfactual disappears the moment the system goes live. Before the build the process runs in the open, with a time per case, an error rate, and a cost anyone could count. Afterward it is gone, and what is left is memory. Memory is not neutral. The operator remembers a careful team under load. The sponsor remembers a process that was slow and error-prone. Both are sincere, and neither is checkable. The account that wins belongs to whoever is still in the building. An automation project produces two states of the world and instruments one of them. The state after launch records itself. Software emits timestamps and case counts as a by-product of running at all. The state before launch was run by people. People emit almost nothing. A ticket queue logged when a case opened and closed, not what happened in between. The question arrives twelve months in, from somebody new. A finance lead inherits the operating budget, finds a line for monitoring and model tuning, and asks what it returned. The available answer is throughput. The onboarding system clears routine cases and routes exceptions. That describes its behavior, not its financial return. A return is a comparison, and the other half had to be recorded before the system existed. Start with the distribution. Average handling time is the number an operations team already has, and it is the least useful, because the average case was never the expensive case. The cost lives in the tail: the submission that arrives incomplete, the one that waits on an absent reviewer. Two processes can report the same mean and be different businesses: a four-hour median with a ninetieth percentile of nine days, against two days every time. Those numbers are illustrative. Nobody reconstructs a distribution from memory. Error rate is second, and it has to be measured against a defined ground truth rather than against the absence of complaints. A complaint rate is not an error rate. Defining it is dull and finite: pull a sample of closed cases, have a senior reviewer adjudicate them blind, and count the disagreements. Recalled a year later, an error rate is the sentence that mistakes were rare. Cost per case is third, and the visible labor line is not it. The multiplier that turns visible labor into true cost is settled elsewhere. The multiplier is not the hard part. The inputs are. Every one of them exists only while the manual process is still running. The fourth item is not a measurement. It is a decision to keep a slice of volume running the old way after launch. A holdout is the only instrument here that survives contamination. Every other number was recorded under different conditions: a different quarter, a different volume, different reviewers. The holdout runs at the same time, on the same inputs, under the same seasonality. Nothing else does. The holdout is also the first thing cut. A holdout keeps a manual process alive past the date it was supposed to end. That cost is smaller than a year spent arguing about a number nobody recorded. Everything then depends on the routing rule, mechanical and fixed before anyone reads the case: the last digit of the submission identifier, assigned at intake. The first exception granted ends the study, because the manual arm then holds the cases somebody decided were unusual. The more common failure is not skipping the baseline. The failure is capturing it in the gap between approval and launch, when the team already knows the system is coming. Reviewers told their process is being replaced stop investing in it, and the numbers then describe a process in decline. That flatters the system. The window closes when the build is approved, not when it starts. The baseline gets taken in the opening weeks of an engagement, alongside the access work and before any code is written, because it is the only window in which the old process still exists to be measured. The distribution, the ground truth sample, and the cost per case go into a file the client keeps, and we write the method next to each number. A stranger has to be able to check the arithmetic.