Correlation, Causation and Your Job Data

Why this matters

Once a shop's jobs live in software, patterns start jumping off the reports. One tech has twice the callbacks of another. Members spend far more than non-members. Jobs booked online carry bigger tickets. Every one of those is a correlation: two things that move together. Owners act on them by coaching the tech, pushing the plan harder, or spending on the website, and a large share of the time the pattern was produced by a third thing nobody looked at. Acting on the wrong cause costs twice: the fix does nothing, and the real driver keeps running.

Why two numbers move together when neither causes the other

When A and B rise and fall together, there are only a few explanations. A causes B is one of them. The others are the ones that catch shops:

• A hidden third factor drives both. Researchers call it a confounder. The tech with more callbacks also gets the harder jobs, and hard jobs produce callbacks whoever runs them.
• The arrow runs backwards. Customers whose systems are failing join the maintenance plan, then spend on repairs they were going to need anyway. Spending did not follow membership; the failing system produced both.
• Who got into the group. This is selection: the people in a group were different before the group existed. The customers who say yes to a plan were already your best customers. Comparing them to everyone else compares two different populations, not a plan against no plan.

There is a fourth explanation, chance on a small sample, and it is the first one to rule out. The Analytics shelf's noise-versus-signal card covers how large a gap has to be before a count your size can show it, so it is not re-derived here.

The shape behind the first one, which is the one owners miss most:

              job difficulty
             /              \
     drives /                \ drives
           v                  v
   assigned to Tech A     callbacks
           |                  ^
           +-- looks like ----+
               Tech A causes callbacks

The line along the bottom is what the report shows. The two lines from the top are what actually happened. Until you check the top of the triangle, the bottom line is a guess.

Worked case 1: the tech with the callbacks

A six-tech shop pulled a year of callbacks by technician. Two techs ran the same number of jobs:

The owner's read: A is sloppier, put him on a ride-along. Before doing that, the service manager asked who gets which jobs. The dispatcher sent diagnostic calls on older equipment, intermittent faults and system changeouts to A, because A is the better diagnostician. B ran most of the tune-ups and simple repairs. Split by job type:

Check the totals: A's 130 plus 70 is 200 jobs and 11 plus 1 is 12 callbacks; B's 50 plus 150 is 200 and 5 plus 3 is 8.

Inside each job type, A's rate is the lower one: 8.5 percent against 10.0 on complex work, 1.4 against 2.0 on simple. Overall, A's rate is the higher one, because 130 of his 200 jobs, 65 percent, are the kind that come back, against 50 of B's 200, 25 percent. When a comparison flips direction once you split it into groups, that is called Simpson's paradox, after the statistician Edward Simpson, who described it in 1951.

Two cautions on reading the split. The counts inside each cell are small, a single callback moves a rate by more than a point, so this table does not show A is the better tech either. What it shows is that the headline gap is explained by job mix, and coaching A would have targeted the one tech who was absorbing the shop's hardest work. The ride-along was cancelled. What the shop did instead was start reporting callbacks by job type as well as by tech, so the next comparison would start in the right place.

Worked case 2: members spend more

The same shop's report showed plan members spent 1.8 times as much per year as non-members in the year after joining. The owner wanted to discount the plan to sign more people up, on the logic that every new member would be worth 1.8 times a regular customer.

The service manager pulled the same customers' spending in the year BEFORE they joined. Those customers had already been spending 1.5 times the non-member average. They were homeowners who called for every problem, paid promptly and owned more equipment. The plan did not create that; it attracted it.

So the honest comparison is the members' spend after joining against their own spend before: 1.8 against 1.5 on the same non-member yardstick, which is 1.8 divided by 1.5, a lift of about 1.2 times, roughly 20 percent more. Not 80.

Then the arrow check. Of 120 customers who joined last year, 34 (28 percent) had a system over 15 years old, against 18 percent of non-members. An old system produces repair spend whether or not its owner is on a plan, and old-system owners are more likely to join because a tech just told them the unit is near the end. Some of the 20 percent is the plan; some is the age of the equipment arriving on the same customers.

The discount decision changed. A new member brought in by a discount will look more like an average customer than like the current members, because the discount reaches people who were not already inclined to join. Planning on 1.8 times would have priced the discount against a number the new members would never produce.

The field card: common shop patterns and the third factor to check

Each row is a pattern that shows up on ordinary service-business reports, the most likely alternative explanation, and the cut that tests it.

None of the right-hand cuts needs anything beyond the reports most field software already produces, filtered one more way.

How to check a pattern before you act on it

Four questions, asked in this order because each one is cheaper than the next:

1. Is it big enough to be real? If a single job moves the rate by a point or more, settle that first.
2. Who decides which jobs, customers or trucks end up in each group? If a person or a rule does the sorting (the dispatcher, the customer choosing to join, the route plan), assume the groups differ before you compare them, and split by whatever drove the sorting.
3. Could it run backwards? Ask what the second thing would do to the first if the causal arrow pointed the other way. Members spend more because they joined, or joined because they were about to spend?
4. Did the supposed cause come first? A price change in April cannot explain a revenue rise that started in March. Lay out the dates before the totals.

The split in question 2 is the workhorse. Compare like with like inside a group where the third factor is held still: complex jobs against complex jobs, the same months against the same months, the same customers before and after. If the pattern survives the split, it is worth acting on. If it vanishes, the split just saved you from fixing the wrong thing.

When a correlation is good enough to act on

Sometimes you cannot run the split cleanly and still have to decide. A pattern earns more trust when several things line up:

• It survives the split you could run, even if not every split.
• It has a mechanism you can describe in a sentence a tech would agree with. "The written findings sheet gives the customer something to show their spouse" is a mechanism. "Online customers are just better" is not.
• It moves in steps. If one photo gets a slightly better review and five photos get a clearly better one, a dose-response like that is harder to explain away than a single yes-or-no difference.
• The action is cheap and reversible. Then acting on a likely cause is fine, as long as you keep watching. Reversible decisions have their own card; see related below.

The strongest evidence of all comes from controlling who gets the change, so that nothing else can sort the groups. That is an experiment rather than a report, and its design has a sibling card of its own.

The failure that hides inside a correct split

A split can mislead too, when the factor you split on is itself caused by the thing you are testing. Say you want to know whether a new diagnostic checklist reduces callbacks, and you split by "time on site" to make the comparison fair. If the checklist is what makes techs spend longer on site, splitting by time on site removes the very effect you were looking for, and the checklist looks useless. The rule: split on things that were fixed BEFORE the cause acted (job type at booking, equipment age, the customer's history), never on something the cause could have changed.

References

• Simpson, E. H. (1951), "The Interpretation of Interaction in Contingency Tables," Journal of the Royal Statistical Society, the original description of the reversal in worked case 1
• See related: The Experiment That Proved the Wrong Thing
• See related: Running a Small Experiment Instead of Arguing
• See related: Reversible and Irreversible Decisions in a Small Shop
• See related: A Number Moves: Is It Noise or a Real Signal?