Survivorship Bias in Industry Advice

Why this matters

Almost everything an owner learns about running a shop outside his own walls comes from someone who made it: the speaker on the conference stage, the podcast guest, the coach with a page of testimonials, the owner in the online group posting a record month. The shops that did exactly the same thing and failed are not on stage, not on the podcast and not posting. That gap is not a small correction. It can turn a tactic that works spectacularly for two shops in seven into something that sounds like a law of the trade, and an owner who bets the year on it is betting on odds nobody showed him.

What survivorship bias is

Survivorship bias is drawing conclusions from the cases that passed through some filter while ignoring the ones the filter removed. The classic illustration comes from Abraham Wald, a statistician working for the US military in the Second World War. Engineers looked at bombers returning from missions, saw where the bullet holes clustered, and proposed adding armor there. Wald pointed out that the planes being examined were the ones that came back. The places with no holes on the survivors were where a hit brought a plane down. The armor belonged where the returning planes were clean.

The shop version: the advice you hear has been through a filter that selects on outcome. You are examining the planes that came home.

Where the filter sits

Each common source of industry advice removes the failures in its own way:

• Conference talks. Organizers book the shops with the best story. Nobody books "we tried it and closed."
• Podcasts and video channels. A guest is invited because something worked, and the host has no reason to find the eleven who tried the same thing.
• Coaching testimonials. Clients who doubled their revenue write testimonials. Clients who quit after four months do not, and the coach is under no obligation to tell you how many there were.
• Online owner groups. People post wins and ask for help on problems. They rarely come back to report that the tactic they announced six months ago quietly failed.
• Supplier case studies. A manufacturer profiles its best dealer, not its median one.
• Trade magazine profiles. "How this shop grew to 40 trucks" is a story. "How 30 shops stayed at 6" is not.

None of these sources is lying. Each one is honestly reporting a real result. The bias is in what never reaches you.

The cohort you never see

Take one tactic, say going all-in on a maintenance membership program with a dedicated sales push, and imagine every shop that tried it in the same year:

40 shops launch an aggressive membership push
  |
  +-- 6  grew strongly        -> speak, post, coach
  |
  +-- 14 roughly the same     -> say nothing
  |
  +-- 9  dropped it quietly   -> say nothing
  |
  +-- 11 sold or closed       -> cannot say anything
            (for any reason, not
             only the program)

What reaches you: 6 stories, 0 denominators

These counts are illustrative. The point is the shape: the only branch that produces stories is the top one, so from where you sit the success rate looks like 6 of 6. And notice the bottom branch: some of those shops closed for reasons unrelated to memberships, which is exactly why you cannot read the survivors as proof the program saved anyone either. Without the full cohort you can learn neither the odds nor the cause.

What a survivor cannot tell you

This is the part worth memorizing, because it is what to listen for in every talk.

The denominator. How many tried it. A speaker usually does not know, and a coach who does know rarely volunteers it.

Whether the tactic caused the result. The shop that doubled its average ticket with a tiered price menu may also have hired a strong service manager that year, raised prices across the board, or caught a hot summer. The speaker credits the thing the talk is about.

The conditions. Market size, competition, a spouse with a steady income that covered a lean year, a family member who handed over a customer list, cheap capital. Conditions are invisible on stage and they decide whether the tactic transfers.

What it cost the ones it hurt. A tiered menu that raises average ticket can also drop close rate. The survivor saw the first effect. The shop that lost a third of its replacement quotes saw the second, and is not in the room.

Worked example: the doubled average ticket

An owner of a seven-truck plumbing shop hears a speaker say: "We moved to a three-tier option menu on every repair and doubled our average ticket in twelve months." The owner wants to roll it out across the board the following Monday.

Step 1: find the denominator. He posts in two owner groups and calls through his trade association contacts, asking a narrow question: did you try a tiered option menu, and what happened to average ticket and close rate after a year? Twelve owners answer. Seven tried it. Of those seven:

• 2 report big gains in average ticket, in the range the speaker described
• 3 report modest gains, average ticket up about 10 to 15 percent
• 1 saw no real change
• 1 dropped it after close rate on replacement quotes fell noticeably

So the speaker's result is 2 of 7, and a gain of some size showed up in 5 of 7. That is a decent tactic, not a doubling machine. These seven are still a filtered sample (the people who answer a group post are not random), but they include failures, which the talk did not.

Step 2: check the conditions of the big winners. Both owners with large gains had raised their base prices in the same year and both already ran membership programs, so their customers were used to seeing options. This owner has no membership base and has not raised prices in two years. His situation looks more like the modest-gain group.

Step 3: ask the failure what happened. He calls the owner who dropped it. Her techs presented three options on small repairs where the customer wanted one price, and customers read it as an upsell. She thinks it works on replacement decisions and not on a dripping faucet.

Step 4: size the bet to the realistic case. His expectation moves from "double" to "probably 10 to 15 percent on the jobs where it fits, with a real chance of hurting close rate if used everywhere." So instead of rolling it out on every call, he runs it for one quarter on replacement and major-repair quotes only, tracking average ticket and close rate against the prior quarter.

The decision he makes is smaller, cheaper and survivable if it fails. That is the whole return on asking the denominator question.

The questions that find the denominator

A coach or speaker who answers the last two directly is worth more than one with a longer testimonial page. One who cannot or will not answer them has told you something too.

Survivorship inside your own shop

The same filter runs on your own data, and it is harder to see because the data feels complete.

Your best techs' habits. You study what your three top techs do and want to train everyone on it. But the techs who did the same things and burned out or quit are gone from the sample. If two former techs also worked twelve-hour days and left exhausted, "work like the top guys" may be advice that only survivors can take.

Customer surveys. Surveying current customers tells you what people who stayed think. The customers who left over price, a no-show or a rude call are not in the survey. A customer satisfaction score that climbs while the customer count falls is partly the filter at work: the unhappy ones are removing themselves.

Job records. Callbacks get reviewed; quiet jobs do not. If you only examine failures, you can miss that the same "risky" practice also appears on hundreds of jobs that went fine.

Exit gaps. The fix in each case is the same: go find the ones the filter removed. An exit call with a former tech, a short call to five customers who did not renew, a look at a random sample of clean jobs alongside the callbacks.

When survivor advice is still worth taking

Survivorship bias does not mean industry advice is useless. It means you discount it to fit what you know, and some advice survives the discount well:

• The downside is small and reversible. Trying a new script on one tech's calls for a month costs little even if most shops that try it see nothing.
• The mechanism is visible. "Confirming the appointment by text the day before cut our no-shows" works for a reason you can see and check in your own records. Advice whose mechanism is "it changed our culture" is harder to test.
• The failures are documented too. A speaker who says "this worked for us, and here are the two shops I know it did not work for and why" has done the denominator work for you.
• It matches your conditions. Advice from a shop your size, in a market like yours, at a stage like yours, has passed through a narrower filter than advice from a 40-truck operation in a metro of three million.

Where none of those hold, the safest move is to treat the story as a hypothesis and test it small before committing. See related: Running a Small Experiment Instead of Arguing.

References

• Mangel, M., and Samaniego, F. J. (1984), "Abraham Wald's Work on Aircraft Survivability," Journal of the American Statistical Association
• Denrell, J. (2003), "Vicarious Learning, Undersampling of Failure, and the Myths of Management," Organization Science
• See related: Base Rates: The Number You Should Start From; Reading a Benchmark Claim Skeptically; Evaluating a Consultant or Coach Pitch; Running a Small Experiment Instead of Arguing