Scoring Options With a Weighted Decision Matrix

Why this matters

A weighted decision matrix is the tool owners reach for when two options each have real advantages: a van or a truck, one software package or another, hiring a tech or a dispatcher. Done well, it turns an argument into a short list of the one or two things the decision actually turns on. Done badly, it is a spreadsheet that launders a decision you had already made. The difference is almost entirely in two places: whether the weights were fixed before anyone saw the scores, and whether you ran the sensitivity check at the end. Most people skip the second, and it is the part that tells you whether the matrix decided anything.

The worked example below runs through every step: a four-tech HVAC shop replacing one service vehicle, choosing between a mid-roof cargo van and a pickup with a utility body.

Step 1: Write the decision down as one sentence, with the options named

"Replace truck 3 with either a mid-roof cargo van or a three-quarter-ton pickup with a utility body, to run residential service for six years."

Why this step comes first: the sentence fixes the time horizon and the job the vehicle does. A vehicle for six years of residential service scores differently from one for three years of new-construction rough-in. Skip it, and two people filling in the matrix will be scoring two different decisions.

Step 2: Separate the gates from the criteria

A gate is a must-have: an option that fails it is out, no matter how well it scores elsewhere. A criterion is something you trade off. Gates never get weights.

The shop's gates:

• Carries the standard truck-stock load with margin under the manufacturer's rated payload.
• Fits through the shop's bay door, measured at the shop.
• Can be upfitted with locking storage.

The high-roof van failed the second gate, which is why the options in Step 1 are the mid-roof van and the truck. That is a real finding, not a technicality: a high-roof van that lives outside all winter loses the weatherproof-storage advantage that made it attractive.

What breaks if you skip it: a must-have entered as a heavily weighted criterion can be outvoted. A van that does not fit the bay door but scores 5 on everything else can win the matrix and then sit in the lot.

Step 3: Choose criteria that do not overlap

List what genuinely differs between the options for this job. Then check each pair for double counting: if two criteria measure the same underlying thing, merge them, or that thing gets counted twice.

The shop started with eight and merged two pairs. "Fuel economy" and "maintenance cost" became running cost per mile. "Driver comfort" and "parking in tight neighborhoods" became maneuverability. Six remained.

Step 4: Set the weights before anyone scores, and write them down

Weights sum to 100. Set them with the scores hidden, ideally with a second person, and do not change them afterwards without writing down why.

Why before scoring: once you have seen that the van scores 5 on interior storage, an owner who prefers trucks will discover that interior storage does not matter very much. Fixing weights first is the only protection the method has against its user.

Step 5: Anchor the scale for each criterion, then score

Use 1 to 5, and say what 1, 3 and 5 mean for each criterion before scoring. Without anchors, a 4 means "I like it."

Anchors this shop used, two as examples:

• Running cost: 5 = lowest of the options considered; each step down is roughly 10 percent more per mile. The van's estimated mileage was about 18 miles per gallon against the truck's 14, so the truck burns 18 / 14 = about 1.29 times the fuel per mile, roughly 29 percent more. With maintenance similar, that put the truck two to three steps below the van; the shop scored it 3 against the van's 4, a conservative call it noted in writing.
• Interior storage: 5 = all stock indoors and lockable; 3 = some locking compartments, most stock exposed to weather; 1 = open bed.

The scores:

Step 6: Multiply and add

Each score times its weight, summed:

• Van: 25 x 4 = 100, 20 x 5 = 100, 15 x 4 = 60, 15 x 3 = 45, 15 x 4 = 60, 10 x 3 = 30. Total 100 + 100 + 60 + 45 + 60 + 30 = 395.
• Truck: 25 x 3 = 75, 20 x 2 = 40, 15 x 3 = 45, 15 x 5 = 75, 15 x 3 = 45, 10 x 4 = 40. Total 75 + 40 + 45 + 75 + 45 + 40 = 320.

Divided by 100, that is 3.95 against 3.20. The van leads by 75 points on the 100-weight scale.

Most matrices stop here. Do not, because so far you only know which option won under one set of guesses.

Step 7: Run the sensitivity check

The question is: how far would the weights have to move to flip the answer, and is that move plausible for this shop?

The fast way is to look at the score difference per criterion, van minus truck:

Moving one point of weight from a criterion where the van leads to one where the truck leads closes the gap by the difference between the two rows. Moving weight from interior storage (+3) to ladder carrying (-2) closes it by 3 - (-2) = 5 points per weight point. To close a 75-point gap that way takes 75 / 5 = 15 points: interior storage would have to fall from 20 to 5 and ladder carrying rise from 15 to 30.

Check it directly. With interior storage at 5 and ladders at 30, the van loses 15 x 5 = 75 on storage and gains 15 x 3 = 45 on ladders, landing at 395 - 75 + 45 = 365. The truck loses 15 x 2 = 30 and gains 15 x 5 = 75, landing at 320 - 30 + 75 = 365. A tie, exactly where the arithmetic said it would be.

Now judge the move, not the arithmetic. For a residential HVAC service van that carries a 6-foot stepladder and short line sets, weighting ladders at six times interior storage makes no sense. The answer is robust: no reasonable set of weights for this job flips it.

Then check the scores the same way. The storage score is the biggest single driver, so test it. If the shop had been too generous and the true gap on storage were 4 against 3 rather than 5 against 2, the van would lose 20 x 1 = 20 and the truck gain 20 x 1 = 20, closing the gap by 40 to 35. Still the van.

Step 8: Read what the check told you

There are three outcomes, and each means something different.

• Robust (this case): plausible changes to weights and scores leave the winner in place. Buy the van and stop deliberating.
• Fragile: a small, plausible change flips it. The options are close, and the matrix has done its job by telling you which criterion the decision really turns on. Stop scoring and go get better information on that one criterion: ride along in a demo vehicle for a day, or count how many jobs last quarter needed long material.
• Flips on a criterion you had not thought hard about: usually a sign the criteria list is wrong or a gate is hiding in it.

This shop did hit the fragile case on a second pass. One of its four techs runs mostly repipes and carries 10-foot pipe daily. Re-running the matrix for that role with ladder and long-material carrying at 30 and interior storage at 10 gave the truck the lead. The real finding was that this was two decisions, one per role, not one decision for the fleet.

How the matrix gets misused

Tuning the weights until the favorite wins. The tell is weights that change after scores are visible, or a criterion that appears halfway through. If you catch yourself adding "image with commercial customers" after the van won, that is the method being bent.

Precision theater. A 3.95 against a 3.88 is a tie. Scores are judgments on a 5-point scale; the totals are not precise to the hundredth. Treat any gap under about 5 percent of the leader's total as a tie, then go to Step 8's fragile path.

Using it for decisions with one dominant factor. If one gate or one criterion settles it, a matrix adds nothing but delay. The tool earns its place when three or more criteria genuinely pull in different directions.

Scoring the brochure. Every score should trace to something: a mileage estimate, a ride-along, a quote, a count of last quarter's jobs. If you cannot say where a score came from, mark it and treat it as the first thing to test in Step 7.

How to verify you got this right

Before acting, check four things on the sheet itself:

• The weights carry a date earlier than the scores, or a second person's initials.
• Every gate is listed separately and every option shown passed it.
• Each score has a one-line source.
• The sensitivity note names the criterion with the largest difference row and says how many weight points would flip the result, with your judgment on whether that move is plausible.

If any is missing, the matrix is an opinion with arithmetic attached.

References

• Edwards, W., work on multi-attribute utility and simple weighted scoring (SMART), the academic root of this method
• See related: Stress-Testing a Plan With Three Scenarios
• See related: Reversible and Irreversible Decisions in a Small Shop
• See related: Satisficing Versus Optimizing