Measuring Whether an AI Tool Actually Saved Time

Why this matters

Every AI tool is sold on the time it saves at the moment of use: the draft appears in ten seconds instead of the twenty minutes it took to write. That number is real and it is the smallest part of the ledger. The hours that decide whether the tool paid off are spread out and arrive late: reading the draft, fixing it, the callback a wrong sentence caused, the job a tech did for free because the estimate promised it. A shop that measures only the first number can feel faster every day while losing an hour a week. This card shows how to count the rest, then runs one rule against two real-shaped tools that come out on opposite sides of it.

The ledger has six lines, not one

Measure in hours of someone's paid time per week. Every line below is a different part of the same task.

Net hours saved per week = baseline minus (assisted + review + correction + rework + upkeep).

The vendor's headline, and the owner's gut feeling, is baseline minus assisted. That is the saving at the keyboard. The other four lines are why it is not the saving.

Two cautions belong right next to the formula. The lines are not all the same kind of hour. Review and correction are usually office or owner hours; rework is often a tech's hours plus a truck. Adding them as paid hours is fair for deciding whether the tool earns its place, but an owner hour freed and a tech hour lost are not worth the same to the shop, so say whose hours each line is. And rework lags. An estimate that promised the wrong thing costs nothing until the job is done. A one-week measurement will almost always flatter the tool, so the rework line needs at least four weeks of tracking, divided back to a weekly figure.

Getting each number honestly

Baseline: time it before the tool arrives, on real tasks. Pick ten typical instances, time them with a clock, done by the person who normally does them. Remembered baselines run high ("those took forever"), and that inflates every saving that follows. Timing your fastest person or your easiest jobs inflates it the other way.

Assisted and review: time them together as one stretch, then split. People under-report review because it feels like part of the job rather than a cost of the tool. It is a cost of the tool: without the tool there is nothing to review.

Correction: count how many outputs needed a fix, and time a few. The count matters as much as the minutes, because it tells you the tool's error rate on your work, and that rate drives rework.

Rework: log every downstream event tied to a tool output for four weeks. A disputed line, a call asking what something meant, work done to honor wording the shop did not intend. Log each with its hours and whose hours they were. If you only count rework somebody complained about, you count the least of it.

Upkeep: keep a running note. Prompt fixes, retraining a new hire, re-checking after the vendor changes something. It is small most weeks and large in a few.

The first two weeks with any tool are slower than steady state while people learn it. Do not measure them, or measure them and say so.

The gate

The rule this card runs: keep a tool only if net hours saved per week, averaged over at least four weeks including rework, is positive. Positive is the floor, not the target; a tool that nets ten minutes a week is not worth the risk it adds. Here is the same gate on two tools at the same eleven-person HVAC shop.

Tool A: the estimate scope writer that saved four hours and cost five

The owner writes about 12 estimates a week. Each needs a scope of work: what is included, what is not, the equipment, the conditions. Writing them by hand took about 25 minutes each.

Baseline: 12 estimates at 25 minutes is 300 minutes, 5.0 owner hours a week.

With a general-purpose chat assistant, the owner types the job details and gets a full scope back. About 5 minutes per estimate, including typing the request.

Assisted: 12 at 5 minutes is 60 minutes, 1.0 owner hour. The keyboard saving is 5.0 minus 1.0, which is 4.0 hours. That is the number the owner told everyone.

Now the rest of the ledger, tracked over four weeks:

• Review: reading a full scope properly took about 10 minutes, because the drafts were long and fluent and every inclusion had to be checked against what the shop actually does. 12 at 10 minutes is 120 minutes, 2.0 owner hours.
• Correction: 6 of the 12 each week needed a fix, about 8 minutes each. 6 at 8 is 48 minutes, 0.8 owner hours.
• Rework: in four weeks, two estimates went out with a line the review missed: "includes removal and disposal of the existing equipment". The shop does not normally include haul-away, but it was in writing, so it honored it. Each haul-away took about 3 tech hours including the dump run, so 6 tech hours over four weeks, 1.5 tech hours a week. One customer also disputed an exclusion the draft had worded loosely, about 1 owner hour over the four weeks, 0.25 owner hours a week.
• Upkeep: adjusting the request template after each miss, about 0.5 owner hours a week.

Hidden lines: 2.0 plus 0.8 plus 1.5 plus 0.25 plus 0.5 is 5.05 hours a week, 3.55 of them owner hours and 1.5 tech hours.

Total with the tool: 1.0 assisted plus 5.05 hidden is 6.05 hours. Against the 5.0-hour baseline, net is minus 1.05 hours a week. The tool saved 4.0 hours at the keyboard and cost 5.05 elsewhere. Counting owner hours alone, 1.0 plus 3.55 is 4.55 against 5.0, a saving of about 27 minutes a week, which is why the owner felt faster: the loss landed on the techs' clock, a column he was not looking at.

Gate result: fails. Negative net over four weeks. What failed it was not the drafting; it was that the output was long, the errors were buried inside fluent text, and an error that got through became a written promise. That is the expensive-to-check kind of output. See related: Using AI to Check an Estimate, Not to Build It.

Tool B: the invoice description cleaner that saved less than it claimed

The office turns about 60 tech notes a week into invoice descriptions a customer can read. By hand, about 3 minutes each.

Baseline: 60 at 3 minutes is 180 minutes, 3.0 office hours.

Assisted: with the tool, about 30 seconds each to paste the note and get the draft. 60 at 0.5 minutes is 30 minutes, 0.5 office hours. Keyboard saving: 3.0 minus 0.5, 2.5 hours.

The rest, over four weeks:

• Review: two or three sentences, checked against the note and the parts on the invoice, about 1 minute each. 60 minutes, 1.0 office hour.
• Correction: about 1 in 10 needed a fix, 6 a week at 2 minutes, 12 minutes, 0.2 office hours.
• Rework: two customer calls in four weeks asking what a line meant, about 1 office hour over the four weeks, 0.25 a week.
• Upkeep: about 0.25 office hours a week.

Hidden lines: 1.0 plus 0.2 plus 0.25 plus 0.25 is 1.7 hours. Total with the tool: 0.5 plus 1.7 is 2.2 hours against a 3.0-hour baseline. Net is plus 0.8 hours a week, all of it office hours.

Gate result: passes. The real saving is 0.8 hours, about a third of the 2.5 the keyboard suggested, which is typical. It passes because the outputs are short, each one can be checked against the note in under a minute, and the worst error is a confusing sentence rather than a promise. See related: Turning a Messy Tech Note Into a Clean Invoice Description With AI.

What the two cases show

Both tools did their drafting well. The difference is entirely in the lines after "assisted". Three properties of the task predicted which way it went, and you can read them before you buy anything:

• How long the output is. Review time scales with length, and errors hide in length.
• What an uncaught error becomes. A clumsy sentence on an invoice costs a phone call. A sentence in a scope of work becomes an obligation.
• Whose clock the rework lands on. If it lands somewhere the person measuring does not look, the tool will look like a success to the person who chose it.

What would turn Tool A around

The shop did not have to abandon the idea. It changed the task so the cheap-to-check property held. The owner wrote a fixed list of the shop's standard inclusions and exclusions, about 30 lines, and the tool was told to pick from that list and draft only the job-specific description of the equipment and site. Review dropped to about 4 minutes, because the inclusions were now the shop's own words, recognisable at a glance. The haul-away error became structurally impossible, since disposal was on the exclusions list. That version needs its own four weeks of measurement before anyone calls it a win; the point is that the fix was making the output checkable, not finding a better tool.

How to know your measurement was honest

• The baseline was timed on real tasks before the switch, on at least ten instances, by the person who normally does them.
• Review was timed separately and is not zero. A review line of zero means either nobody reads the outputs or nobody counted.
• Rework was tracked for at least four weeks and includes events nobody complained about.
• Each line says whose hours it is. If a saving on one person's clock is a loss on another's, the ledger shows both.
• The headline and the net are both written down. If you only record one, record the net.

References

• See related: Using AI to Check an Estimate, Not to Build It
• See related: Turning a Messy Tech Note Into a Clean Invoice Description With AI
• See related: The Verify-Before-You-Send Rule for AI Output
• See related: Where AI Helps a Small Shop and Where It Quietly Costs You