Measurement Resolution and Honest Rounding

Why this matters

A calculator will happily turn a tape reading to the sixteenth and a guess at a wall's squareness into an area of 130.0839 square feet. Every digit after the first decimal is invented. Nobody is fooled on a floor plan, but the same habit on a service report reads very differently: "airflow 1,407.4 CFM" or "power 1,135.2 W" tells the customer, and the next tech, that somebody measured to a precision nobody did. The opposite slip costs material: rounding a cut length the wrong way, or rounding an order down. This card is about writing each reading with the resolution it actually has, carrying that honestly through a calculation, and rounding in the direction the purpose demands. The difference between accuracy and resolution themselves is owned by Accuracy, Resolution and Repeatability Are Three Different Things; this card starts where that one stops, at the pencil.

A reading is a number plus its resolution

Resolution is the smallest step the instrument can show: a sixteenth of an inch on a typical tape, the last digit on a meter's display, one graduation on a gauge. It is not the same as accuracy, but it sets a floor: a reading cannot be known more finely than the step it was read in. A tape read to the nearest sixteenth puts the true length within half a step, about a thirty-second, either side of the mark, before any other error is counted.

Write readings so the resolution is visible:

READINGS WRITTEN WITH THEIR RESOLUTION

length A   12 ft 3 1/16 in   tape, to 1/16 in
length B   10 ft 7 3/8 in    tape, to 1/16 in
current    4.73 A            clamp, 0.01 A digit,
                             spec +/-(2% rdg + 3 dgt)
voltage    240.0 V           meter, 0.1 V digit,
                             spec +/-(1% rdg + 2 dgt)
pressure   0.48 in WC        manometer, 0.01 in WC

CURRENT, worst-case bound
2% x 4.73 A        = 0.095 A
3 digits x 0.01 A  = 0.030 A
bound              = +/- 0.125 A, about 2.6% of rdg

VOLTAGE, worst-case bound
1% x 240.0 V       = 2.4 V
2 digits x 0.1 V   = 0.2 V
bound              = +/- 2.6 V, about 1.1% of rdg

Two things stand out. The meter specs above are illustrative of the usual form, a percentage of the reading plus a count of the last digit; your meter's manual gives its own, and they change by range. And a 4.73 A reading carries a worst-case error of about 0.12 A, so the 3 in the second decimal is real as a display digit and not real as a fact about the current. Recording it is still right. Reporting it as meaningful is not.

Worked example: the room, the floor, the cut

A tech measures a rectangular room at 12 ft 3 1/16 in by 10 ft 7 3/8 in, both to the sixteenth. Work in inches to keep one unit throughout:

• 12 ft 3 1/16 in = 144 + 3.0625 = 147.0625 in.
• 10 ft 7 3/8 in = 120 + 7.375 = 127.375 in.
• Area = 147.0625 x 127.375 = 18,732.1 sq in.
• In square feet: 18,732.1 / 144 = 130.08 sq ft.
• Rough check: 12.25 ft x 10.6 ft is about 130. Agrees.

How exact is 130.08? Each length is good to about a thirty-second of an inch from resolution alone. A thirty-second on 147 in is 0.021 percent; on 127 in, 0.025 percent. In a product, relative errors add in the worst case, so the area is good to about 0.046 percent, or 0.06 sq ft. Resolution allows 130.1 sq ft and no more. In practice the real uncertainty is larger, because walls bow, corners are rarely square, and the tape was hooked on a baseboard; the resolution is the best case, never the whole story.

Now the same number goes three places, and each rounds differently.

Buying: round up. Flooring sold in cartons of, say, 22 sq ft (illustrative) after a 10 percent waste allowance: 130.08 x 1.10 = 143.09 sq ft, and 143.09 / 22 = 6.50 cartons. Order 7. Dropping the fraction to call it 6 (or rounding a figure like 6.4 to the nearest carton) leaves the job a carton short on the last row. Material always rounds up, and it rounds up after the waste factor is applied, not before. How large that factor should be belongs to The Materials Waste Factor Worth Measuring.

Cutting: round to the mark. A tape cannot cut 37.4375 in; it can cut 37 7/16. When a computed length is exactly on a sixteenth, use it. When it falls between two marks, round toward the side you can still fix: long for a piece you will trim or scribe to fit, short for a piece that has to drop into a fixed gap. Halving a 74 15/16 in span to center something gives 37.46875 in, which is 37 15/32: between the 7/16 and 1/2 marks. For a mark you are centering on, either is within a thirty-second of true; for a filler piece going into a slot, cut 37 7/16 and it drops in.

Recording: do not round. On the job sheet, the room is 12 ft 3 1/16 in by 10 ft 7 3/8 in, exactly as read. The area can be written as 130.1 sq ft beside it. The raw readings are the evidence; the rounded area is a conclusion drawn from them. Round the evidence and nobody can check the conclusion later.

Analog gauges: graduations, interpolation, and full scale

A dial gauge's resolution is its graduation spacing, and a careful reader can interpolate to about half a graduation, not finer. On a gauge marked every 2 psi, a needle sitting between 60 and 62 can honestly be written as 61, never as 61.3. Read it square-on: viewed from an angle, the needle appears to sit beside a different mark (parallax), which is a reading error of its own that no amount of care with the arithmetic recovers.

Analog accuracy is usually stated as a percent of full scale, not of the reading, and that changes what the number means at different points on the dial. Take a 0 to 100 psi gauge with an accuracy of plus or minus 2 percent of full scale (illustrative; the gauge's own grade or data sheet gives the real figure). That is plus or minus 2 psi anywhere on the dial. At 60 psi it is about 3 percent of the reading. At 10 psi it is 20 percent. The same gauge is a fine instrument for a 60 psi water service and a poor one for a 10 psi boiler fill, which is why the gauge range should put the expected reading in the middle third of the dial.

Carrying resolution through arithmetic

Two working rules, both approximations to a full error analysis and both good enough for the field:

• Multiplying or dividing: keep as many significant figures as the least precise input. A 1,200 CFM estimate times a 20 F rise does not produce an answer good to five significant figures, whatever the calculator shows.
• Adding or subtracting: the answer is no finer than the coarsest input's decimal place. Three pieces measured at 12 1/16 in, 30 3/16 in and "about 8 1/2 in" (read to the nearest half inch) total 50 3/4 in, but the total is only good to about a quarter inch, because the third piece is.

The second rule is the one that bites in layout: one rough measurement anywhere in a chain sets the precision of the total. Chained Measurements Versus One Datum covers what happens when many small errors add.

Worked example: power from two meter readings

The current and voltage readings above came off a resistance heater, so watts are simply volts times amps.

• Calculated: 240.0 V x 4.73 A = 1,135.2 W.
• Error bounds, both worst-case limits from the meter specs: 2.6 percent on the current, 1.1 percent on the voltage. Worst-case bounds add linearly in a product: 2.6 + 1.1 = 3.7 percent.
• 3.7 percent of 1,135 W is 42 W. The power lies somewhere between about 1,093 W and 1,177 W.
• Report: "about 1,140 W, within 4 percent." Not 1,135.2 W.

The bounds were added rather than combined some gentler way because a meter spec is a limit, not a measured spread, and two limits stacked in the same direction is a real possibility. If you had characterized each meter's actual spread, a different combination would apply; with only the spec sheet in hand, the linear sum is the honest one. On a motor or any load that is not purely resistive, volts times amps is not watts at all, and no amount of honest rounding fixes using the wrong formula.

Why extra decimals mislead

A number on a report is read as a claim. "Static pressure 0.4837 in WC" says the manometer resolves a ten-thousandth of an inch of water; the one in the van resolves a hundredth. A reader who trusts that precision compares it with next year's 0.4790 and concludes the system has changed by 0.0047 in WC, a difference smaller than the instrument's own last digit. They have just diagnosed noise.

The same applies to derived figures. An airflow worked out from a temperature rise and a rated furnace output is good to a few percent at best. Write it as about 1,400 CFM and the reader knows it is an estimate; write 1,407.4 and they will treat it as a measurement, and perhaps balance a system to it.

When not to round at all

Rounding belongs at the end of a calculation, once. Round in the middle and the error compounds: rounding 147.0625 in to 147 and 127.375 in to 127 before multiplying gives 18,669 sq in, or 129.6 sq ft, already about 0.4 sq ft off the 130.08 before anything else goes wrong. Carry every digit through the working, then round the answer to what the inputs support. Write the Calculation Down: The Field Math Habit owns that habit; this card owns how far to round once you get there.

And some numbers should never be rounded on the way into a record: serial numbers and model codes, obviously, but also as-found readings, setpoints you changed, and any value a later comparison will depend on. Round the summary line. Keep the reading.

A short check before a number leaves your hands

• Does each reading on the sheet show its unit and the resolution it was read at?
• Does the answer carry more digits than its roughest input? Trim it.
• Is this number for buying, cutting, or recording? Up, to the mark, or not at all.
• Was anything rounded in the middle of the working? Redo it from the raw readings.
• If two readings are being compared, is their difference bigger than the instrument's resolution and stated accuracy? If not, it is not a difference you can report.

References

• Instrument manuals for resolution and accuracy specifications by range
• See related: Accuracy, Resolution and Repeatability Are Three Different Things; The Difference Between Accuracy and Resolution; Write the Calculation Down: The Field Math Habit; The Materials Waste Factor Worth Measuring; Chained Measurements Versus One Datum