Running a Small Experiment Instead of Arguing

Why this matters

Every shop has an argument that comes back every quarter. Quote the diagnostic fee on the phone or wait until the tech is on site. Offer the maintenance plan at the door or in the follow-up email. Raise the trip charge or leave it alone. Two people who both know the business hold opposite views, nobody has data, and the loudest or most senior voice wins until the next time it comes up. A small experiment ends that loop: you test the change on a slice of real calls for a fixed period, with the winning condition written down before anyone sees a result. It costs a few weeks and some discipline. It replaces an opinion with a number both sides agreed in advance to accept.

This article is about running the test. Reading a test that went wrong after the fact is its own skill. See related: The Experiment That Proved the Wrong Thing.

Step 1: Turn the argument into one sentence each side would sign

Write the disagreement as a prediction with a number in it, and get both people to agree it says what they believe. "Quoting the fee upfront scares people off" is a mood. "Quoting the fee on the phone will cut our booking rate on inbound service calls by more than 5 points" is a test.

Why first: most shop arguments are two people describing different outcomes. The office manager is worried about bookings; the owner is worried about techs driving to houses where the customer says no at the door. If you skip this step you will run a clean experiment on bookings, the owner will say "that was never my point," and the argument survives the data.

What breaks if skipped: the test answers a question nobody was asking. Write both predictions down, side by side, and you will often find you are testing two metrics, not one.

Step 2: Pick one deciding metric and write the success rule before you start

Choose one number that decides the question, and at most one guardrail number that can veto it. Then write the rule as a sentence with a threshold: "We adopt script B if its booking rate is no more than 5 points below script A's AND its rate of customers declining on arrival is lower." Both conditions, with the word AND, written down and dated.

Why before: once results exist, everyone finds the metric that supports their side. Booking rate looks flat? Look at average ticket. Ticket looks flat? Look at reviews. With enough metrics one of them will move by chance, and that is the one that gets quoted. Kahneman and Tversky's work on how people judge evidence keeps arriving at the same place: we are fluent at explaining a result after we see it, and poor at predicting it before. Writing the rule first takes that fluency out of play.

What breaks if skipped: the experiment becomes a Rorschach test. A rule set after the data is a description of the data, not a test of anything.

Step 3: Assign the slice so nothing else lines up with it

Decide which calls get the change and which do not, using a rule that has nothing to do with the calls themselves. Alternate calendar days is the workhorse in a small shop: odd dates get script A, even dates get script B. Calendar dates rotate across weekdays, so over four weeks each script lands on Mondays, Fridays and everything between.

Why this method: the tempting design is "two weeks of A, then two weeks of B." It is the worst one available, because everything that changes over a month (weather, a heat wave, a school break, an ad that started running) lines up perfectly with the script change. You will measure the season and call it the script.

What breaks if skipped, or done by person: splitting by CSR ("Dana runs A, Marco runs B") tests Dana against Marco. Splitting by caller type ("new customers get B") tests the customer types. Either result is real and neither answers the question.

Step 4: Count what you need and set the end date

Look at how many qualifying events you get a week and decide the end date from that, before starting. A test needs enough events on each side that a real difference shows up over the noise. A usable rough rule for a small shop: with about 80 events per side, gaps under about 10 points are routinely ordinary wobble, and a gap needs to approach 15 points before it stands on its own. With 40 per side, that bar is closer to 20 points.

Why the end date is fixed: if you check the numbers every day and stop the moment your side is ahead, you will eventually "win," because a coin flipped enough times runs ahead for a while. Fixing the end date in advance is what makes the final number mean something.

What breaks if skipped: the test ends when someone gets bored or gets the answer they wanted, and both are worse than not testing.

Step 5: Lock everything else for the duration

Write a short list of things that do not change until the end date, and tell everyone who touches the calls. The contamination list for a phone-script test usually reads:

• Both CSRs run both scripts, by the day rule, never by preference.
• No new ad, coupon, price change or website change starts mid-test.
• Nobody listens in on one script's days and not the other's. Coaching one script is changing it.
• The tech arriving at the door does not know which script booked the call, so how they open is not shaped by it.
• Calls are tagged A or B at intake, in the job record, not reconstructed later from memory.

Why it matters: contamination is quiet. An owner who sits beside the CSR on B days "just to hear how it goes" has made B a different script, delivered with more confidence and a boss in the room.

What breaks if skipped: you get a clean-looking result on a test that measured two or three things at once.

Step 6: Run it, log it, and do not peek

Run to the end date. Keep a one-line log of anything unusual: a phone outage, a storm day, a CSR out sick. Do not open the running totals. If someone insists, show the event count only, so you can confirm the test is on track for its size without showing which side is ahead.

The whole test fits on one card that lives where the office can see it:

EXPERIMENT CARD          started 03/03   ends 03/28
Question: does stating the diagnostic fee on the
  phone lose bookings or save wasted trips?
Office predicts: bookings fall more than 5 points
Owner predicts: fewer declines at the door
Slice: inbound service calls, odd dates = A,
  even dates = B
A: fee mentioned only when the tech arrives
B: fee stated before the slot is offered
Deciding metric: booked / qualifying calls
Guardrail: declined on arrival / booked
Rule: adopt B if booking rate is no more than
  5 points below A AND arrival declines are lower
Frozen: ads, prices, website, who answers
Log: 03/12 phones down 2 hrs (A day)

Step 7: Read against the rule, then decide

At the end date, compute the deciding metric and the guardrail, apply the rule exactly as written, and make the call. If the rule says adopt, adopt. If the result is inside the noise band, say so honestly and fall back on whether the change is cheap to reverse.

Why: the whole value of the exercise is that nobody gets to reinterpret it. If the rule turns out to have been a bad rule, write a better one and run a second test. Do not edit the first one after the fact.

Worked example: the diagnostic fee on the phone

A six-truck plumbing and HVAC shop runs the card above for four weeks. The shop gets about 40 qualifying inbound service calls a week, so four weeks gives about 160, split roughly evenly by the date rule.

The raw counts at the end date:

Applying the rule as written. Booking rate: B is 65.9 minus 64.1 = 1.8 points below A, which is inside the 5 points the rule allows. Arrival declines: B's 6.0% is below A's 16.7%. Both conditions hold, so the rule says adopt B.

Reading it honestly. With about 80 calls per side, the 1.8-point booking gap is well inside the ordinary wobble, so this test cannot tell the two booking rates apart. That cuts both ways: it does not show B costs bookings, and it could not have proven B costs fewer than 5 points either, because 5 points is smaller than the noise at this size. The decline difference is larger, 16.7 minus 6.0 = 10.7 points, but it sits on about 50 booked jobs per side, short of the 15-point bar, so it is suggestive rather than proven. The office manager's worry has not been confirmed. The owner's hope has some support.

Why adopting is still right. Script B is a two-way door: the shop can switch back next month by changing one line in the call guide. When a change is that cheap to reverse, a decision-grade result is enough, and waiting for proof-grade certainty costs more than being wrong would. See related: Reversible and Irreversible Decisions in a Small Shop.

The number that actually mattered. Neither side's predicted metric told the full story. Paid visits per call went from 54.9% to 60.3%, because a truck that drives to a house where the customer declines at the door is the most expensive outcome in the whole funnel. The shop adds that as the deciding metric for its next phone test.

What would have changed the call. If B's booking rate had come in 18 points below A, past the 15-point bar and outside the rule, the shop would have dropped B even with fewer arrival declines. If the shop had wanted certainty on a one-way door (a permanent price change printed on trucks, say), 80 calls per side would not have been enough and the test would have run three months, not four weeks.

How to tell the experiment was sound

Check these against your own card before trusting the result:

• The rule is dated before the start date, and the decision applied it word for word.
• The slice was assigned by a rule unrelated to the callers, and the counts per side came out within about 10% of each other (82 and 78 here). A lopsided split means the assignment leaked.
• The frozen list held. Read the log. A coupon that started on day nine contaminates everything after day nine.
• Every event is tagged at intake. If anyone had to guess which script a call got, the test lost that call.
• The result is stated with its noise. "B won by 1.8 points" on 80 calls a side is a false sentence. "No detectable difference in bookings, fewer arrival declines" is the true one.

References

• See related: The Experiment That Proved the Wrong Thing
• See related: Reversible and Irreversible Decisions in a Small Shop
• See related: Correlation, Causation and Your Job Data
• See related: Keeping a Decision Journal
• Kahneman, D., Thinking, Fast and Slow (2011), on hindsight and after-the-fact explanation