Reading a Benchmark Claim Skeptically

Why this matters

"The average shop closes 55% of replacement estimates." You read a line like that at a trade show, in a vendor's newsletter or from a coach, and your own number is 34%. Within a week you have a new sales script, a new pay plan for the comfort advisor, or a quiet conviction that your techs cannot sell. Most of the time the gap is not a performance gap. It is a gap between two different formulas, two different groups of shops and two different reasons for publishing. This article is how to take a published industry number apart before you let it move anything in your shop.

This is the card for a number someone ELSE measured on other shops. Comparing yourself against your own trailing history is a different problem with different traps; see the Analytics shelf for that one.

The claim we will take apart

Every step below is run on one real-shaped claim. A four-truck HVAC shop's owner reads this in a sales-platform vendor's annual report:

  INDUSTRY PULSE 2026
  Average replacement close rate: 55%
  Top quartile: 71%
  ------------------------------------------
  * Based on 412 responses from users of our
    platform, Jan-Mar. Close rate = sold
    opportunities / qualified opportunities.
    Self-reported. Not audited.

The owner's own number, pulled the obvious way: 112 replacement estimates last year, 38 sold. 38 divided by 112 is 33.9%, call it 34%. Against 55%, that reads as a 21-point gap, and the owner is already drafting a message to the comfort advisor.

The footnote is where the claim actually lives. Most people never read it.

Step 1: Find out who could have answered

Before you look at the number, find the population: the set of shops the survey drew from. Here it is users of one sales platform. That is not "the average shop." It is the average shop that chose to pay for sales software, which is a group already selected for caring about close rate, running a structured sales process, and usually having a dedicated salesperson.

Why first: every later step depends on it. A number can be computed perfectly and still describe a group you are not in. What breaks if you skip it: you compare a four-truck shop where the owner runs replacement calls between service calls against a population that skews toward shops with a full-time comfort advisor and a lead-qualification step at the phone.

Ask, in this order: who was invited, how were they chosen, and what did they have in common before anyone answered a question? Members of a coaching group, customers of a software vendor, attendees at a conference, readers of a magazine. Each of those is a filter, and each filter is usually correlated with the thing being measured.

Step 2: Find the response rate and guess its direction

The report says 412 responses. The vendor's own marketing elsewhere says it has about 3,100 shop customers. 412 of 3,100 is 13.3%. So roughly seven in eight invited shops did not answer.

That matters less for its size than for its direction. Non-response bias is the error you get when the people who answer differ systematically from those who do not. Ask yourself who fills in a voluntary survey about close rate. Shops proud of theirs, shops that track it carefully (and tracking tends to come with managing), and shops with someone in the office whose job includes vendor surveys. Shops with a bad year, shops that do not measure, and shops that went under since signing up mostly stay silent.

So the guess is not "the true number is unknown." It is "the true number for platform users is probably lower than 55%, and we cannot say by how much." Write the direction down. A benchmark whose bias you can sign is much more useful than one you have merely called unreliable.

What breaks if you skip it: a 13% sample gets read as if it were a census. The related problem of only hearing from the shops that survived is its own card; see related: Survivorship Bias in Industry Advice.

Step 3: Pin the definition to a formula, then restate your number in theirs

"Close rate" is not one number. It is a fraction, and the survey and the shop almost never use the same top and bottom. Write theirs out exactly: sold opportunities divided by qualified opportunities. Then call the vendor or read the methodology page until you know what each word means. In this case:

• Qualified opportunity meant a written proposal presented in person to all decision makers. Phone quotes and visits where one spouse was absent did not count.
• Sold opportunity meant any sold job from the visit, including a repair the customer chose instead of the replacement.

Now re-count the owner's 112 estimates on that formula, one adjustment per line so the arithmetic is visible:

112 minus 21 leaves 91 presented estimates. 38 of 91 is 41.8%. Adding the 9 repairs sold on replacement visits gives 47, and 47 of 91 is 51.6%. Same shop, same year, same jobs. The rate moved 17.7 points without one customer behaving differently.

What breaks if you skip it: you chase a gap that is 17 or 18 points of bookkeeping. The owner who fires a salesperson over this has fired someone for using a stricter denominator than the vendor did.

Note the trap in the other direction too. Once you see how much the definition moves the number, it is tempting to keep choosing the formula that flatters you. Pick the benchmark's formula for the comparison, keep your own formula for running the shop, and never mix them in one sentence.

Step 4: Ask whether it is a mean or a median, and how wide the spread is

"Average" usually means the mean: add everyone's number and divide by the count. A median is the middle shop when you line them all up. The two diverge whenever a minority of results sits far out in one tail, and self-reported business numbers almost always have a long tail of very good claims.

Ask for both. In this case the methodology page gave a median of 49%. So the middle responding shop closes 49% on this formula, and the owner's re-based 51.6% sits slightly above it. The mean of 55% is pulled up by the top quartile at 71% and above.

Then ask for the spread. "Top quartile 71%" tells you a quarter of respondents report 71% or better. If the report will not say where the bottom quartile sits, you do not know whether 49% is a crowded middle or a thin one, and a shop 3 points from the median could be anywhere from ordinary to unusual.

What breaks if you skip it: a mean-versus-median gap gets treated as a performance gap. Between the median of 49% and the mean of 55% sit 6 points that belong to the shape of the distribution, not to anyone's sales skill.

Step 5: Match the comparison group

Even inside a well-defined survey, the shops are not all you. Check four things against the respondents, and ask the publisher for a cut if one exists:

• Size. A shop with a full-time salesperson runs a different funnel from one where the owner sells.
• Lead source. Shops buying leads from aggregators close lower than shops whose replacement calls come from their own maintenance customers, because a warm maintenance customer has already chosen you once.
• Market. Price sensitivity, competitor density and the share of homes still on original equipment all differ by region and by neighborhood.
• Period. January to March is heating replacement season in much of the country, often no-heat emergencies where the customer is not shopping around. A spring cooling quote is a different sale.

For this shop, 70% of replacement estimates came from its own maintenance agreements, which should close higher than the survey mix, not lower. That cuts the other way from steps 1 and 2: on that dimension the owner might be under-performing a fair comparison. Write that down as well. Skepticism is not the same as discounting everything until you look good.

Step 6: Ask who paid for the number and what they sell

Every published benchmark exists for a reason. A trade association wants members to feel informed. A software vendor wants prospects to feel behind. A coach wants a gap only coaching closes. None of that makes the number false, but it tells you which way an unconscious choice of formula, population or headline probably leaned.

Here the publisher sells sales software, the headline used the mean rather than the median, and the cover line read "Is your team leaving half its closes on the table?" The 6 points between the mean and the median ended up in the headline. That is not fraud. It is the predictable result of someone choosing the more interesting of two true numbers.

What breaks if you skip it: you absorb the publisher's framing along with its data, and the framing is usually that you need what the publisher sells. Evaluating the pitch that often follows a benchmark like this is its own card; see related: Evaluating a Consultant or Coach Pitch.

Step 7: Decide what the claim is allowed to change

After steps 1 to 6, decide out loud which of three things the benchmark earns:

1. A question. "Why do we lose so many one-spouse visits?" is a fair question this survey raised by accident, because removing those 21 visits moved the rate 7.9 points on its own (33.9% to 41.8%).
2. A target. Only if the definition, population and period all match, and you can restate your number in the benchmark's own formula. This one barely clears that bar on definition and fails it on population.
3. Nothing. Plenty of benchmarks earn nothing, and that is a legitimate outcome of reading one.

For this shop the claim earned the question. The owner pulled the 21 excluded estimates: 14 were one-spouse visits, and 2 of those 14 sold. 2 of 14 is 14.3%, against 41.8% on fully presented visits. The fix that came out of a sales-software benchmark was a scheduling rule at the phone, asking whether both decision makers can be present, which costs nothing and touches no pay plan.

What breaks if you skip it: the default outcome of reading a benchmark is "set it as a target." A target built on a mismatched definition either looks impossible (and demoralizes) or, once re-based, looks already met (and stops you looking for the real problem).

What changes the answer

A methodology you can actually check. A government statistical series or an audited financial benchmark with published definitions, sampling and response rates deserves far more weight than a vendor's self-reported survey. Steps 1 to 4 then take ten minutes instead of a phone call, and you can trust the comparison once you have matched the formula.

A gap that survives re-basing. If the owner had re-counted on the vendor's formula and landed at 30% against a median of 49%, the definition explains nothing and the gap is real. The method is not a way to dismiss benchmarks. It is a way to find out which ones are telling you something.

A benchmark tied to money or law. Insurance experience ratings, lender covenants and published wage surveys can bind you or set your price, whatever their flaws. Read them the same way, but you cannot opt out of them by finding a definitional gap.

How to tell you read it right

You can say the benchmark's formula as a fraction from memory. You can state your own number in that formula, with each adjustment on its own line. You know whether the headline figure is a mean or a median. You can name the direction of the main bias (respondents likely better than non-respondents, here). And you can say in one sentence what the number is allowed to change in your shop, including "nothing." If any of those five is missing, you have read the headline, not the benchmark.

References

• See related: Survivorship Bias in Industry Advice
• See related: Base Rates: The Number You Should Start From
• See related: Evaluating a Consultant or Coach Pitch
• See related: Correlation, Causation and Your Job Data
• U.S. Census Bureau and Bureau of Labor Statistics methodology pages, as examples of how a survey states its population, response rate and definitions