Overconfidence and Calibrating Yourself

Why this matters

Ask an experienced tech how long a water heater swap takes and you get a confident answer, often a confident range. Ask them to give a range they are 90 percent sure of, then check it against twenty real jobs, and the range usually holds about half the time. That gap is overconfidence: not a center that is too low, but a range that is too narrow for how much the work actually varies. It is the reason a schedule that looks full and comfortable at 7 a.m. has two customers calling about a late tech by 3 p.m. The fix is a short exercise on your own jobs, and it changes how the day gets booked.

What the research found, in one paragraph

The classic demonstration is Marc Alpert and Howard Raiffa's work, written up in 1969 and published in 1982: people asked for ranges they were 98 percent sure contained a true answer missed it something like four times in ten. Later work with managers, including J. Edward Russo and Paul Schoemaker's, kept finding 90 percent ranges that captured the answer roughly half the time. Expertise does not cure it. Experts give tighter ranges, so they miss about as often. The useful part is that the bias is measurable and trainable: people who score their own ranges and widen them get close to their stated confidence within a few rounds.

This is a different problem from estimating too low. A tech whose average guess is dead on can still give ranges far too tight. The library's card on optimism bias in your own estimates covers the center; this one covers the width.

The procedure

Each step carries its reason and what breaks if you skip it. The example running through the steps is one plumbing service tech, Marco, and his shop's service manager.

1. Pick one job type you book often, and the next 20 of them. Calibration needs volume on comparable work, and 20 is enough that one miss is five percentage points. Mixing job types muddies the scoring, because a range that is fine for drain clears may be hopeless for diagnostics. Skip it and you score five each of four job types and learn nothing about any of them. Marco chose his own residential service calls: water heaters, fixture swaps, drain clears and disposal work, all one tech, all one kind of house.

2. Before each job, write three numbers: your best guess, and a low and a high you are 90 percent sure the actual time will fall between. Write them before you leave for the job, not after you see it. "90 percent sure" means that out of ten jobs like this, you would expect one to land outside the range. The best guess is what you would book; the range is your honest uncertainty around it. Skip writing them down and memory edits the range afterwards. Everybody remembers having expected the long one.

3. Record the actual on-site time from the job record. Use the time from arrival to departure as the job record shows it, and use that same basis for every job. Drive time is a separate variable; mixing it in makes a slow commute look like a bad estimate. Skip the fixed basis and half your misses turn out to be traffic.

Marco's log after the first round, an excerpt:

JOB  TYPE            GUESS  LOW-HIGH    ACTUAL  RESULT
114  WH swap 40g     3.0    2.5-3.5     4.2     over
117  kitchen faucet  1.0    0.75-1.25   1.1     hit
121  main drain      1.5    1.0-2.0     2.9     over
126  disposal        1.0    0.75-1.25   0.6     under
129  WH swap 50g     3.0    2.5-3.5     3.3     hit
     (all times in hours on site)
ROUND 1: 20 jobs, 11 hits, 7 over, 2 under

4. Score the round: count how many actuals landed inside the range. A well-calibrated 90 percent range catches 18 of 20. Marco caught 11 of 20, 55 percent. Seven jobs ran past his high and two finished under his low. Skip the count and you are back to remembering that most jobs went fine, which they did. Most jobs going fine is exactly what a too-narrow range looks like.

5. Find which end is failing and by how much. The misses are not symmetric, and that tells you where to widen. Marco's seven over-high misses had highs averaging 2.6 hours and actuals averaging 3.6 hours: the actuals ran a full hour, about 38 percent, past the top of his range. His two under-low misses were short by about 0.2 hours each. So his lows were nearly right and his highs were far too short. Then the habit that does the most work: for each new range, before writing the high, ask what would have to happen for the job to run past it. If two ordinary things come to mind (a seized shutoff, a flue that needs rework, a drain line with a belly), the high is not a 90 percent high. Raise it until the things that would beat it are unusual. Skip the diagnosis and people widen both ends evenly, which wastes width at the bottom and still misses at the top.

6. Run a second round of 20 with the widened ranges. Marco's second round caught 17 of 20, 85 percent: two over, one under. That is within one job of the 18 a calibrated range expects, and with one job worth five points, a round of 20 cannot tell 85 from 90. Keep running rounds until two in a row land between 16 and 19 hits. If you start landing 20 of 20 every round, your ranges are now too wide to schedule with, which is underconfidence, and you tighten the high a little. Skip the second round and you never learn whether the widening overshot.

7. Schedule with the range, not just the guess. This is where the exercise pays. The shop had been booking the sum of the best guesses into the day and calling the day full when they reached it. With calibrated ranges, the service manager books the guesses but plans the slack against the highs, using one rule.

The scheduling rule, worked through

The rule: the slack left in a day must be at least the largest single gap between a job's best guess and its high, and the job with the widest range goes last. It is a starting point; tune it to your work. It does not require every job to hit its high at once, which almost never happens. It requires the day to survive the one job most likely to blow up.

Marco's Thursday, with calibrated ranges, in hours on site:

His day runs 9.0 hours, with about 1.0 hour of driving between three stops. The best guesses add to 3.0 plus 1.5 plus 1.5, which is 6.0 hours, plus 1.0 of driving, 7.0. That leaves 9.0 minus 7.0, 2.0 hours of slack. The largest gap is the drain clear's 2.0 hours. Slack of 2.0 against a largest gap of 2.0 meets the rule at its floor, so the day holds three jobs.

The drain clear has the widest range, 2.5 hours from low to high, so it goes last. If it runs to its high, it pushes the end of Marco's day, not a customer waiting for him.

The range also sets the arrival window the office promises. The water heater starts at 8:00 and runs 2.5 to 4.5 hours, so it finishes between 10:30 and 12:30. With about 20 minutes to the second house, Marco arrives between 10:50 and 12:50. The office used to promise that customer "around 11", the best guess plus the drive, and Marco missed it whenever the water heater ran long. Now it promises an 11-to-1 window and calls ahead when the first job is past its best guess at 11:00. The customer gets a window the tech can actually keep, and the call ahead lands before they start wondering.

Under the old method the office would have seen 2.0 open hours and squeezed in a fourth job, a 1.5-hour fixture swap, leaving at most 0.5 hours of slack, before the extra drive, against a 2.0-hour gap. That is the day that produces the 3 p.m. phone calls. The rule told the office not to add it, or to add it only as a "call ahead, may move" slot the customer agrees to in advance.

Note what the high column adds to: 4.5 plus 2.5 plus 3.5 is 10.5 hours, plus driving, 11.5, which does not fit in a 9.0-hour day. That is the point. A day booked against the sum of the highs books far less work than the crew can actually do. The rule sits between the two extremes on purpose.

How you know it worked

Two checks, both from records the shop already keeps:

• The hit rate holds. Re-run a round of 20 every quarter. A rate that has slid back below 16 means the ranges tightened again, which they tend to do as a tech gets comfortable.
• The late-tech calls drop. In the month before Marco's shop switched, the office logged 9 calls from customers asking where their tech was. In the second month after, it logged 3. That is the number the customer feels, and it is the reason to do this at all.

If the late-tech calls do not fall while the hit rate looks good, the problem is not the ranges. Look at drive time or at how the office is filling cancellations.

What changes the method

• Job types with a fat tail. Diagnostics, old-house remodel work and anything behind a wall have a long right side: most finish near the guess and a few run triple. Their calibrated ranges will stay wide, and that is correct. Do not narrow a range to make the schedule look tidier; book those jobs last, or book them alone.
• New techs. A tech in their first year has no basis for a range yet. Use the shop's own history for the job type (the spread of the last 20 actuals) until their own rounds are calibrated.
• Yes-or-no calls rather than durations. "I'm 80 percent sure it's the control board" is the same skill on a different kind of question. The library has a card on calibrating diagnostic instinct over time that covers that version.

References

• Alpert, M. and Raiffa, H. (1982), "A progress report on the training of probability assessors," in Kahneman, Slovic and Tversky (eds.), Judgment Under Uncertainty: Heuristics and Biases
• Russo, J. E. and Schoemaker, P. J. H., Decision Traps (1989), on managers' confidence ranges
• See related: How to Avoid Optimism Bias in Your Own Estimates
• See related: Calibrating Your Own Diagnostic Instinct Over Time
• See related: The Planning Fallacy in Shop Projects