The Difference Between Accuracy and Resolution

Why this matters

A meter that displays four digits is not accurate to four digits, and the gap between those two facts is where a lot of field arguments live. A tech reports a value to a tenth, a second tech reads a different tenth, and both start doubting the equipment when the honest answer is that neither instrument ever claimed that tenth was real. Worse, the error compounds the moment you subtract two readings to get a difference, which is what most diagnostics actually do. Knowing how to read your instrument's own accuracy statement turns "the numbers disagree" into a calculation, and it occasionally tells you that the finding you were about to report is entirely inside your own measurement noise.

Three words that are not synonyms

Resolution is the smallest change the display can show. It is a property of the readout. A meter showing 229.5 has a resolution of 0.1 on that range. Resolution costs the manufacturer almost nothing and is therefore the number marketing leads with.

Accuracy is how far the displayed value may sit from the true value. It is a manufacturer's published claim, valid only under stated conditions, and it is almost always larger than the resolution. This is the number that decides whether your reading means anything.

Repeatability is whether the same measurement, repeated, gives the same answer. An instrument can be beautifully repeatable and consistently wrong. Repeatability makes you confident; accuracy makes you correct. A drifting instrument that has drifted evenly is the most convincing liar on the truck.

The habit worth building: when someone says a meter is "accurate to a tenth," ask whether they are quoting the accuracy spec or reading the last digit on the display. It is nearly always the second.

How the spec is actually written

Most field instruments state accuracy in the form plus or minus (a percentage of the reading, plus a number of counts).

• The percent of reading term scales with the value. It dominates at the top of a range.
• The counts term is a fixed number of least-significant digits. One count equals one step of the display's resolution on the range you are using. It dominates at the bottom of a range, which is why reading a small value on a large range is the classic way to throw away accuracy you paid for.

Both terms are quoted over a stated ambient temperature band, with a separate temperature coefficient per degree outside it, and the AC specs carry a stated frequency range and often a crest-factor limit. A meter used in a hot attic, on a distorted waveform, outside the band it was specified in, is operating outside its published claim entirely, and the manufacturer's number no longer applies.

Working artifact: an uncertainty budget for one field question

This is a filled-in worksheet for a real question, using an illustrative instrument spec. Say the meter's DC and low-frequency AC voltage spec reads plus or minus (0.5 percent of reading + 3 counts), and the readings are taken on a range whose display step is 0.1 V, so 3 counts is 0.3 V. Substitute your own instrument's published figures; the structure is the point.

The question. Two voltage readings were taken on one branch circuit under load, one at the panel and one at the equipment, to find the drop along the run. The sibling article on probe location owns why that pair is the measurement; this one asks whether the pair can support the conclusion.

Worst case assumes both errors sit at their limits in opposite directions, which is conservative and correct when you have no reason to believe otherwise. Root-sum-square is the statistical combination described in the Guide to the Expression of Uncertainty in Measurement, and it is the honest choice when the two errors are independent. Report which one you used, because the two answers differ by more than 40 percent of the smaller figure.

What that does to the conclusion. The drop is 9.1 V plus or minus 2.08 V by root-sum-square, so somewhere between 7.0 V and 11.2 V. As a percentage of the panel-end reading, that is 7.0 divided by 238.6, which is 2.93 percent, up to 11.2 divided by 238.6, which is 4.69 percent. The NEC's non-mandatory informational note under Article 210 suggests holding branch-circuit drop near 3 percent; it is advisory, not enforceable. Your measurement straddles it. Taking the worst-case combination instead widens the band to 6.2 V through 12.0 V, or 2.60 percent through 5.03 percent, and it still straddles.

So the honest finding is that this measurement cannot decide the question. The drop is unambiguously real, because 9.1 V is far outside the combined uncertainty either way. Whether it exceeds the guideline is not resolvable with these two readings, and reporting "3.81 percent, over the recommendation" states a precision the instrument never offered.

The fix: measure the small thing directly

Subtracting two large numbers to find a small one is the worst way to use an instrument, because you inherit the full percent-of-reading error from both large values. Measure the small quantity itself instead.

Put one lead at each end of the run and read the drop as a single value. Now the reading is around 9 V rather than around 230 V, so it sits on a much lower range. Say the reading comes in at 9.08 V on a range with a 0.01 V display step. The same spec gives 0.005 x 9.08 = 0.045, plus 3 counts at 0.01 which is 0.03, for a total of about 0.08 V. The drop is 9.08 plus or minus 0.08 V, which is 3.80 percent plus or minus about 0.03 percent against the same 238.6 V base. That is a finding you can put in front of a customer.

Two honest limits on this technique. It needs a lead long enough to span the run with both ends energized, which is not always physically or safely available, and the base value it is expressed against still carries its own uncertainty, which is second-order here but not zero. Where the direct measurement is not possible, take the difference and report the band rather than the point.

Before either measurement

Both readings above are on energized conductors in an open enclosure. 29 CFR 1910.333(a)(1) requires live parts to be de-energized before you work on or near them unless de-energizing is infeasible, and its note treats testing that can only be done energized as one such case. That gate ends when the reading does. Meter, leads and probe tips must be rated at or above the circuit's measurement category and voltage under IEC 61010-1, because an under-rated instrument on a line-voltage fault vents as an arc rather than blowing quietly, and a long trailing lead used for a drop measurement must be routed so it cannot fall across a busbar or into a moving part. Wear the electrical protective equipment 29 CFR 1910.335(a) requires for the exposure and keep one hand out of the enclosure. When you move from measuring to repairing, de-energize and lock out under 29 CFR 1910.333(b)(2), or 29 CFR 1926.417 on a construction site, and prove dead with the live-dead-live sequence in NFPA 70E-2021, 120.5.

What changes the numbers above

• Range selection. The counts term is fixed in display steps, so the same reading on a higher range carries a larger absolute counts error. Auto-ranging usually picks well; manual ranging on the wrong range quietly triples your uncertainty.
• Temperature. Outside the meter's specified ambient band, add the published coefficient per degree. An instrument left on a dashboard in summer and used immediately is not inside its own spec.
• Waveform. An averaging AC meter reads correctly only on a clean sine wave. On a distorted waveform, its error is not covered by the accuracy statement at all, and a true-RMS instrument has its own stated crest-factor limit beyond which the same is true.
• Calibration status. The published accuracy applies to an instrument within its calibration interval, traceable through a competent laboratory as described in ISO/IEC 17025. A meter that has never been checked has an unknown accuracy, not a published one.

The failure mode

The everyday version is a ticket that reports more digits than the instrument earned. It looks rigorous and it is the tell for the opposite. The costly version is a diagnosis built on a difference of two similar large numbers: a temperature split, a pressure difference, a voltage drop, a before-and-after comparison across visits. Every one of those subtracts away the signal and keeps the full error, and every one of those is what techs reach for first.

Catch it with one question before you report a difference: is the difference at least three times the combined uncertainty of the two readings that produced it? If it is not, you have a suspicion rather than a finding, and the correct next move is to measure the difference directly or to use a better instrument, not to argue about the tenths.

How to verify you got this right

Take your own primary meter, find its published accuracy statement, and compute the uncertainty at two values: near the top of a range and near the bottom of the next range up. Most techs are surprised by the second one. Then take the last differential measurement you reported and redo the budget above on it. If the difference you reported was smaller than three times the combined uncertainty, go back and correct the report, because someone may already be acting on it.

References

• Guide to the Expression of Uncertainty in Measurement (JCGM 100), on combining independent uncertainty components
• ISO/IEC 17025, general requirements for the competence of testing and calibration laboratories, for traceable instrument calibration
• 29 CFR 1910.333(a)(1) energized-testing gate; 29 CFR 1910.333(b)(2) and 29 CFR 1926.417 electrical lockout; 29 CFR 1910.335(a) protective equipment; NFPA 70E-2021, 120.5; IEC 61010-1 measurement categories
• NFPA 70 (National Electrical Code), Article 210 informational note on branch-circuit voltage drop (advisory)
• See related: How a Reading Changes With Where You Take It; Verify the Tool Before You Trust the Reading; The Calibration Schedule Worth Keeping