The Experiment That Proved the Wrong Thing

Why this matters

A shop that runs a test and gets a clean answer acts on it for years. It retrains the crew, rewrites the office script, ties a bonus to it and tells other owners at the supply house that it works. If the answer was produced by something other than the change being tested, all of that rests on a coincidence, and the first sign is usually a slump months later that nobody connects back to the test. This is one case, reconstructed from a shop's own job records, of a test that "doubled" maintenance-plan sign-ups and had actually measured the calendar and one technician.

The result everyone believed

An eight-tech HVAC and plumbing shop wanted more members on its maintenance plan (a yearly agreement covering two tune-ups and a priority-service promise). Through March and April the techs offered the plan however they liked. At the end of April the owner introduced a written script: at the end of every tune-up the tech walks the customer to the equipment, shows two specific findings on the unit itself, and offers the plan with the next visit date already filled in.

The two six-week windows, straight off the job board:

18 of 210 is 8.6 percent and 41 of 240 is 17.1 percent, so the rate very nearly doubled. The owner said so at the next meeting, made the script mandatory, and added a per-sign-up bonus.

Then came September and October. Same script, and ride-alongs confirmed the techs were still using it word for word. Window C ran 230 visits and 21 sign-ups, 9.1 percent. The owner's first explanation was that the crew had gotten lazy, and he was preparing to say so.

Reading the calendar before reading the numbers

The service manager asked for a week before anyone was spoken to. Her first move was not to cut the numbers at all. She listed everything that changed between the windows, from the schedule, the HR file and the weather log, and only then opened the job data:

WINDOW A   Mar 16 to Apr 26    old free-form offer
  techs running routes: 7 (Dana on leave)
  call mix: heating-season wrap-up
WINDOW B   Apr 27 to Jun 7     script starts Apr 27
  Dana back on her route May 4
  first hot week May 18
  two new hires riding along, no own routes
WINDOW C   Sep 7 to Oct 18     script still in use
  Dana on route, all 8 techs running
  cooling season over, mild weather

Between A and B, three things moved at once: the script, the return of the shop's strongest closer, and the season. Only one of those was what the owner believed he was testing. The term for the other two is a confound: a second change that rides along with the one you meant to test, so the result cannot be credited to either one alone. (The general habit of hunting for the third factor behind any pattern in your data has its own card; see related below. This one is about what it looked like in a single test.)

First cut: by technician

Dana had been with the shop nine years and, by the records from the fall before her leave, closed about 22 percent of tune-up visits with no script at all. So the manager split window B into Dana and everyone else.

Dana ran 96 of the 240 visits, 40 percent of them, and produced 24 of the 41 sign-ups, more than half. The other seven techs went from 8.6 percent in window A to 11.8 percent in window B: a rise of 3.2 points, not the 8.5 points the headline showed.

That alone changed the conversation. Most of the "doubling" was one person coming back from leave and doing what she had always done. The question now was whether even the 3.2-point rise in the other techs belonged to the script.

Second cut: the same weeks last year

A rise from spring to early summer can come from the season with no help from anyone. Early-summer tune-ups land on customers who have just felt the first hot week and are nervous about the system, and the findings are easier to show: a matted condenser coil in June is visible from the patio. Spring calls are wrapping up heating season, and the customer is thinking about something else.

The shop had the same weeks from last year, before the script existed. Last year's numbers for the techs other than Dana:

So with no script at all, the season by itself lifted the same group from 7.8 to 10.0 percent, 2.2 points. This year the group rose from 8.6 to 11.8, 3.2 points. The part of the rise the season does not explain is the difference between those two lifts, 3.2 minus 2.2, which is 1.0 point. Comparing the change in the tested group against the change in an untested one is called a difference-in-differences, and it is the cheapest honest comparison a shop with a year of records can run.

Two ways of reading the leftover, both from the same records:

• Against last year's same-season rate. At 10.0 percent, the other techs' 144 visits would have produced about 14.4 sign-ups. They produced 17, roughly 2.6 more.
• Against last year's seasonal lift. One point on 144 visits is about 1.4 sign-ups.

The two methods give two numbers because they make different assumptions about where this year's group started. Both say the same thing: somewhere between one and three extra sign-ups over six weeks across seven techs. On samples this size that is inside the swing you would see from chance alone.

Dana's own rate tells a similar story. Last year in the same weeks she closed 20 of 92, 21.7 percent. This year's 25.0 percent is 3.3 points higher on 96 visits, about three sign-ups, and it could be the script or it could be a rested tech in her first month back. There is no way to tell from one person.

The hypothesis that did not survive

The service manager had one more candidate before she would credit anything to the script: the two new hires riding along. A second person in the house might make the offer feel more serious. The records showed the hires were on 31 of the other techs' 144 window B visits.

• With a ride-along: 4 sign-ups on 31 visits, 12.9 percent.
• Without: 13 sign-ups on 113 visits, 11.5 percent.

That is a 1.4-point gap on 31 visits, which is one sign-up's worth either way. Ruled out, and the counts reconcile: 31 plus 113 is 144 visits, 4 plus 13 is 17 sign-ups.

Why the fall looked like a collapse

The same cuts explained the slump that had the owner ready to blame the crew.

Last fall, with no script, the non-Dana techs had closed 7 of 150 visits, 4.7 percent. This fall they closed 4.9 percent. The script had not stopped working in September. It had never been doing much, and the fall looked exactly like last fall. Nobody had gotten lazy. The owner had been comparing the strongest season with the strongest closer against the weakest season, and reading the gap as effort.

That was the near-miss that mattered most. A shop that disciplines its crew over a seasonal pattern teaches them that the numbers are a weapon, and the next time a rate dips the techs hide the bad tickets rather than talk about them.

The comparison that could answer the question

What the owner wanted to know was whether the script beat the free-form offer when everything else was held still. A before-and-after cannot do that in a business where people, season and call mix all move every month. So the following spring the shop ran both at the same time.

• Assignment by work order number. Odd-numbered tune-ups got the script, even-numbered got the tech's own approach. The office assigned numbers in the order calls were booked, so neither the tech nor the customer chose which arm a visit landed in.
• Same techs, same weeks. Every tech ran both arms, so Dana's skill sat on both sides of the comparison instead of on one.
• Fixed length, decided in advance. Eight weeks, no peeking and stopping early because the first week looked good.

The result: script, 19 sign-ups on 198 visits, 9.6 percent; free-form, 15 on 203 visits, 7.4 percent. A gap of 2.2 points, pointing the same way as last year's leftover and still small enough that eight weeks could not prove it. The design of a test like this has its own card; see related below.

What the shop changed

• Kept the script. It cost nothing, the techs liked having words to fall back on, and both independent reads pointed the same direction. Keeping a free, harmless change on weak evidence is a reasonable call.
• Dropped the bonus tied to it. The bonus had paid out mostly on Dana's return in window B, which rewarded a calendar event.
• Stopped telling people it doubled anything. The owner had repeated the number to two other shop owners. He called both.
• Added one line to how the shop reads any before-and-after. Before anyone looks at the numbers, someone writes down what else changed between the windows: who was on the schedule, the season, prices, marketing, the mix of job types. If that list has anything on it, the comparison is made inside the slice where only the tested thing changed, or against the same weeks last year, before anyone announces a result.

The last line is the one the shop has used most since. It caught a price-increase "test" the next winter that had run in the same month the shop's ads were paused, and it took about two minutes.

References

• See related: Running a Small Experiment Instead of Arguing
• See related: Correlation, Causation and Your Job Data
• See related: Regression to the Mean and the Tech Who Had a Great Month
• See related: A Number Moves: Is It Noise or a Real Signal?