Hindsight Bias in the Callback Review

Why this matters

A callback review is the shop's best chance to learn something, and the phrase that kills it is "he should have known." After the outcome is in, the cause looks obvious, the cue looks loud, and the tech looks careless. Sometimes that is right. Often the cue was quiet, the check that would have caught it was not on anyone's list, or nothing available at the time would have caught it at all. A review that cannot tell those apart punishes techs for bad luck, misses process gaps, and teaches the crew to hide callbacks instead of reporting them.

What hindsight does to a review

Hindsight bias is the tendency, once you know an outcome, to believe you could have predicted it. Baruch Fischhoff demonstrated it in 1975 and called it creeping determinism: people told how a historical event turned out rated that outcome as far more likely, in hindsight, than people who were not told, and they did not realize their judgment had shifted.

Three specific distortions show up in a callback review:

• Cues get louder. The note that the old tank had no expansion tank was one line among forty on the original visit. In the review, knowing the result, it reads as a warning sign.
• Checks get cheaper. "It would have taken two minutes" is said about checks that were not on the checklist and, as often as not, would not have caught the problem.
• Alternatives disappear. The review forgets the twenty other jobs that week where the same cue meant nothing.

A sibling problem is outcome bias, judging the decision by how it turned out. See related: Outcome Bias: Judging a Decision by How It Turned Out. Hindsight is narrower: it corrupts your memory of what could have been known.

The case: a water heater callback

A plumbing shop replaces a 50-gallon gas water heater like for like. Four days later the customer calls: water on the garage floor, dripping from the temperature-and-pressure relief line. The T&P valve (temperature-and-pressure relief valve) is the safety device that opens when tank pressure or temperature goes too high.

A note on the hazard, because the review is about a live one: the T&P valve was doing its job. It is never plugged, capped or replaced to stop a drip, and anyone near the discharge line is kept clear of it because it releases scalding water. The callback tech's job was to find out why pressure was reaching the valve's setting.

He found a pressure-reducing valve (PRV) behind a removable panel near the water meter. A PRV, like a check valve or a backflow preventer, can make the house a closed system: water heated in the tank expands and has nowhere to go back to the street, so pressure climbs every heating cycle until something relieves it. The fix is a thermal expansion tank, which the plumbing code your jurisdiction has adopted generally requires where a closed system exists (both the International Plumbing Code and the Uniform Plumbing Code carry a thermal expansion control section).

The owner's first reaction at the Monday review: "He should have known there was a PRV. That's basic."

Freeze the timeline before discussing the outcome

The first move is to rebuild what the installing tech knew at each moment, from the record only: the work order, the photos, the ticket notes, the GPS. Not from what anyone now remembers thinking.

DAY 1 09:10  Work order: replace 50 gal gas, like
             for like. No mention of PRV.
DAY 1 09:40  On site. Photos: old tank, no expansion
             tank. Meter area not photographed.
DAY 1 10:05  Static pressure at hose bib: 62 psi
             (in the ticket notes)
DAY 1 11:30  New tank fired, no leaks, venting and
             gas checks recorded, photos taken.
DAY 1 11:45  Ticket closed.
DAY 4 07:20  Customer: water at T&P discharge.
DAY 4 10:00  Callback tech opens meter panel,
             finds PRV. Leaves max-pressure gauge.
DAY 5 08:00  Gauge needle reads about 150 psi.

Now ask each "should have known" against this record.

Was the cue present? Partly. The old tank had no expansion tank, which is visible in the photos. But the PRV was behind a panel, and nothing on the work order or the old installation pointed to it.

Was it a cue at the time? This is the question hindsight erases. The shop pulled its last 30 water heater replacements: 11 had no expansion tank on the old install. On those 11, the callback tech's check would have found a closed system on some and an open system on others; the shop's records did not say which, because nobody looked. "No expansion tank" alone was a common condition, not a red flag, until the shop knew to treat it as one.

Would the obvious check have caught it? The installing tech did take a static pressure reading: 62 psi, which is normal. A single static reading taken before the new tank has run a heating cycle does not show thermal expansion, because there is nothing expanding yet. The problem only appears as pressure climbing during and after heating, which is why the callback tech left a gauge with a maximum-reading needle overnight. So the check the owner had in mind, "just test the pressure," was done, and passed, and could not have caught this.

What would have caught it? Looking for a PRV, check valve or backflow device at the service, every time, on every water heater job. That is a two-minute look. It was not on the install checklist.

Sort the miss into its class

Once the timeline is frozen, every callback lands in one of three classes, and each gets a different fix.

The water heater lands in the second class. The tech followed the shop's process, and the process did not ask him to look for a closed system. The fix was one line added to the water heater install checklist ("locate PRV, check valve or backflow device at the service; if present, confirm thermal expansion control") and one field on the work order asking the customer whether they know of a pressure regulator. The tech got told what was found and why the checklist changed. He did not get a write-up.

Note what blaming him would have cost. The other four plumbers would have heard that following the checklist does not protect you, and the 11-in-30 jobs with no expansion tank would have kept going out the door unchecked under every other tech.

What it looked like across a dozen callbacks

The shop then re-ran its last 12 callback reviews through the same method. In the original reviews, the notes said some version of "should have caught it" on all 12. Re-sorted on the frozen record:

• 3 were knowable and missed: a loose union the tech's own photo showed, a skipped leak check that was on the list, a wrong part number read off the old unit.
• 6 were process gaps: four of them traced to the same work order template that never asked about access or prior repairs.
• 3 were not reasonably knowable: two component failures inside a week of install, and one customer-caused change after the visit.

That adds to 3 + 6 + 3 = 12. Half the callbacks the shop had been blaming on techs were really two template fixes away from not happening.

Techniques that keep the review honest

Present the case blind to a second tech. Give another tech the work order, photos and notes as of the moment of decision, without the outcome, and ask what they would have done. If a competent tech reading the same record does the same thing, the record is the problem.

Argue the other outcome. Fischhoff and later researchers found that asking people to explain how a different outcome could have happened reduces hindsight bias more than warning them about it. In the review, someone states how this same job, with the same cues, could have been fine. If that story is easy to tell, the cue was weak.

Count the cue's track record. Pull how often the cue appeared without trouble. A cue that shows up on a third of jobs and precedes a problem on one is not something a tech "should have known."

Price the check honestly. "Two minutes" multiplied by every job where it would have been required is the real cost. The PRV look is worth it on every water heater. A full overnight pressure log on every one is not.

When "should have known" is fair

Sometimes it is, and the method should say so plainly rather than excuse everything. It is fair when the cue was in the tech's own photos or notes, the check was on the checklist or is a step every competent tech in the trade does, and it was skipped. It is especially fair when the same tech has the same class of miss more than once. A review that never lands a miss on a person is as broken as one that always does; techs can tell, and they stop trusting either.

Running it so people report callbacks

The quiet cost of hindsight-driven reviews is that techs stop flagging their own callbacks and start fixing them on the side, off the clock, which means the shop never sees the pattern. Two rules help. First, the frozen timeline is built before anyone discusses blame, and from the record, so the tech can see it is the record on trial. Second, the class is decided by the table above and written into the review note, so "process gap" is a finding the shop owns rather than a favor.

References

• Fischhoff, B., "Hindsight is not equal to foresight: The effect of outcome knowledge on judgment under uncertainty," Journal of Experimental Psychology: Human Perception and Performance (1975)
• International Plumbing Code and Uniform Plumbing Code, thermal expansion control sections, in the edition your jurisdiction has adopted
• See related: Outcome Bias: Judging a Decision by How It Turned Out
• See related: The Fundamental Attribution Error With Customers and Crew