Base Rates: The Number You Should Start From
Why this matters
Every big call an owner makes comes with a story about why this one is different. This candidate interviewed better than anyone. This customer loves us. This new service fits our market perfectly. The story is usually true and it is usually given far too much weight, because the most useful number for the decision is sitting somewhere else: how often things like this have worked before. Owners who skip that number staff up for bids they do not win, plan around hires who do not stay, and budget launches that take twice as long. Owners who start from it are wrong less often and less badly.
The inside view and the outside view
Daniel Kahneman and Dan Lovallo gave the two ways of forecasting their names in 1993.
The inside view looks at the case in front of you: its details, its people, your plan for it. You build the forecast from the ground up, and it feels thorough because you know the specifics.
The outside view ignores the specifics at first and asks what happened to a group of similar cases. That group is the reference class, and the proportion of it that succeeded, or the typical outcome within it, is the base rate.
The inside view is not wrong to use. It is wrong to start from. The details that make your case feel special are the same details every owner in every similar case also had, and those cases still came out at the base rate.
Base-rate neglect
Kahneman and Amos Tversky showed in 1973 how readily people drop a known proportion the moment they hear a vivid description. Told that a group was mostly lawyers and a few engineers, then handed a personality sketch that sounded like an engineer, people judged the person very likely an engineer and largely ignored how few engineers were in the group. Given no sketch at all, they used the proportions correctly. Given a sketch that said nothing useful, they still dropped the proportions.
That last finding is the one that matters for an owner. Even uninformative detail pulls you off the base rate. A candidate's firm handshake, a customer's friendliness on the walkthrough, how excited your service manager sounds about a new line: each feels like evidence, and most of it predicts very little.
Base rates you already have
Most shops have more of these than they realize. The base rate usually lives in your own records, and where it does not, in peer and published data.
| The question | Where its base rate lives |
|---|---|
| Will this new tech still be here in a year? | Your hiring history: hires in the last few years, how many reached 12 months |
| Will we win this bid? | Your bid log, split by job type and customer type |
| Will this estimate close? | Your estimate conversion rate for that job type and ticket size |
| Will this lead source pay off? | Past sources you tried, and how many earned back their cost |
| How long until a new service line breaks even? | Your past launches, plus peers who added the same line |
| Will this invoice past 90 days get paid? | Your own collections history by age bucket |
| Will this new business survive? | US Bureau of Labor Statistics Business Employment Dynamics: roughly a fifth of new businesses close within the first year and close to half within five |
That last row is not a reason to avoid starting anything. It is the honest starting point for any plan that assumes a second location or a new venture is safe because of who is running it.
Choosing the reference class
The base rate is only as good as the group it comes from, and the choice is a judgment call.
Too broad and the class includes cases that differ from yours in the things that drive the result. "All small businesses" is too broad for predicting whether your shop's new drain-cleaning line will work.
Too narrow and you have three cases, or you have quietly defined the class as cases like yours that succeeded. "Plumbing shops in my county that added drain cleaning and are doing great" is not a reference class; it is a list of survivors. See related: Survivorship Bias in Industry Advice.
The test: the cases in the class should share the features that actually cause success or failure for this kind of decision. For a hire, that is the role, the pay, the shop and the labor market, not the candidate's hometown. When two reasonable classes give different rates, use both and treat the gap as a measure of how uncertain you are.
How far to move off the base rate
Start at the base rate, then adjust for what is specific about your case, and adjust modestly. The amount you move should depend on how well your specific evidence has predicted outcomes in the past, not on how strongly it feels. Where your evidence has a track record (a candidate's verifiable work history at a comparable shop), move further. Where it does not (a great interview), barely move. Why early impressions and great months drift back toward average is its own subject. See related: Regression to the Mean and the Tech Who Had a Great Month.
Worked example: one rule, two decisions
A nine-person HVAC shop faces two calls in the same week. The owner runs both through the same rule: start at the base rate from the shop's own records, adjust only as far as the evidence has predicted before.
Decision one: the candidate who interviewed brilliantly. The owner's gut says this tech is a near-certain keeper, call it 90 percent.
The shop's records: 11 techs hired in four years, 5 still there at 12 months. That is a base rate of 5 in 11, about 45 percent.
Is the great interview evidence? The owner goes back through the 11 and marks the ones he remembers as standout interviews. He marks 6. Of those 6, 3 stayed a year, which is 50 percent. Of the other 5, 2 stayed, 40 percent. So in this shop a great interview has moved the keep rate from about 40 percent to about 50 percent. It is weak evidence.
His forecast lands near 50 percent, not 90. He still hires the tech, because a coin-flip chance at a good tech beats an empty seat. What changes is the plan around the hire: he keeps the two runner-up candidates warm with a call at 30 days, books a structured 90-day review, and does not tell a customer this tech will own their account.
Decision two: the commercial bid the customer "loves us" on. A property manager has invited the shop to bid replacement on four rooftop units across two buildings. The walkthrough went well, the manager was warm, and the owner feels it is about 80 percent. On that feeling he is considering bringing on a seasonal tech ahead of the award.
The shop's bid log: 19 commercial bids over two years, 4 won. Base rate 4 in 19, about 21 percent.
Does a warm walkthrough predict anything? He splits the 19 by whether he had felt confident afterward. Confident: 9 bids, 3 won, 33 percent. Not confident: 10 bids, 1 won, 10 percent. Here the feeling carries some real signal, more than the interview did.
His forecast lands near one in three, not 80 percent. At one in three he does not hire ahead of the award. He lines up a subcontractor who can start within two weeks of a signed contract, which costs nothing if the bid is lost.
What the two cases show together. The same rule produced a yes in one case and a wait in the other, and in both it moved the forecast a long way from the gut number: 90 to about 50, and 80 to about 33. The second forecast moved further off its base rate than the first (21 to about 33, against 45 to about 50) only because the shop's own records showed confidence predicting bids better than interviews predict tenure. That is the only legitimate reason to adjust further, and the owner could only see it because he counted.
Small samples and wide error
Eleven hires and nineteen bids are thin. A base rate from a handful of cases is a starting point with wide error, so state it that way to yourself: "about half, give or take a lot" rather than "45.5 percent." Two ways to firm it up:
- Widen the class with outside data. Ask three peer owners for their 12-month retention on new techs, or their win rate on commercial replacement bids. If theirs cluster near yours, you can lean on the number more.
- Keep counting. Every hire and every bid you log makes next year's base rate better. A shop that writes down its predictions alongside the outcome gets this for free. See related: Keeping a Decision Journal.
Where this goes wrong
The flattering class. The most common abuse is choosing the reference class after you know what answer you want. If an owner who wants a second location compares himself only to the shops in his association that opened one successfully, he has picked a class that cannot produce a failure rate.
The base rate as an excuse. "Half of new hires leave anyway" can become a reason to stop improving onboarding. The base rate describes what has happened under your current methods. If you change the method, the old rate is the thing you are trying to beat, and you measure against it.
Forgetting to come back. A base rate is a starting estimate, not a verdict. When the tech hits 90 days strong, or the property manager asks for a revised scope, that is new evidence and the forecast should move.
References
- Kahneman, D., and Tversky, A. (1973), "On the Psychology of Prediction," Psychological Review
- Kahneman, D., and Lovallo, D. (1993), "Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking," Management Science
- US Bureau of Labor Statistics, Business Employment Dynamics, establishment survival data (re-check the current release for the latest rates)
- See related: Forecasting a New Service Line From a Reference Class; The Planning Fallacy in Shop Projects; Regression to the Mean and the Tech Who Had a Great Month; Survivorship Bias in Industry Advice