Floating-Point Equality and Precision

Part I · Module 4 · Chapter 4.10 · Floating-Point Equality and Precision

1. The Headline

puts "0.1 + 0.2          = [expr {0.1 + 0.2}]"
puts "0.1 + 0.2 == 0.3   = [expr {0.1 + 0.2 == 0.3}]"

0.1 + 0.2          = 0.30000000000000004
0.1 + 0.2 == 0.3   = 0

The sum is not 0.3, and the comparison is false.

This is not a Tcl quirk. Every language using standard floating-point arithmetic — C, Python, Java, JavaScript — gives the same answer.

2. Where the Extra Digits Come From

Real numbers are stored in binary, and most decimal fractions have no exact binary form. 0.1 is one of them, in the same way 1/3 has no exact decimal form.

You can look at what is actually stored:

puts "0.1 stored as: [format %.20f 0.1]"
puts "0.2 stored as: [format %.20f 0.2]"
puts "0.3 stored as: [format %.20f 0.3]"

0.1 stored as: 0.10000000000000000555
0.2 stored as: 0.20000000000000001110
0.3 stored as: 0.29999999999999998890

Three values, none of them exactly what you typed.

Adding the first two gives something very slightly above 0.3, while writing 0.3 directly gives something very slightly below it. They are different values, so == correctly reports that they differ:

puts "the gap: [expr {(0.1 + 0.2) - 0.3}]"

the gap: 5.551115123125783e-17

About five hundredths of a femto-unit. Real, and nowhere near anything you care about.

3. The Rule

Do not use == or != on values that came from real-number arithmetic.

That includes anything divided, averaged, scaled, accumulated, or measured. It does not include whole numbers, which are exact — Chapter 3.5 §8.

4. The Fix: Compare Against a Tolerance

Ask whether the two values are close enough, where you decide what close enough means:

set a [expr {0.1 + 0.2}]
set b 0.3

puts "exact equality: [expr {$a == $b}]"
puts "within 1e-9:    [expr {abs($a - $b) < 1e-9}]"

exact equality: 0
within 1e-9:    1

The pattern is always the same shape:

expr {abs($a - $b) < $tolerance}

Take the difference, take its absolute value so the order does not matter, and compare it with a limit you chose.

A reusable version:

proc close_enough {a b {tolerance 1e-9}} {
    return [expr {abs($a - $b) < $tolerance}]
}

puts [close_enough [expr {0.1 + 0.2}] 0.3]
puts [close_enough 1.0 1.5]
puts [close_enough 1.0 1.5 0.6]

1
0
1

The third call shows the point: the tolerance is a decision, not a constant. 1.0 and 1.5 are close enough if your tolerance is 0.6 and not if it is the default.

5. Not Everything Is Inexact

Over-correcting into "never trust a real number" is its own mistake:

puts "0.5 + 0.25 == 0.75   = [expr {0.5 + 0.25 == 0.75}]"
puts "1.0 / 3 * 3 == 1.0   = [expr {1.0 / 3 * 3 == 1.0}]"
puts "0.1 * 3 == 0.3       = [expr {0.1 * 3 == 0.3}]"

0.5 + 0.25 == 0.75   = 1
1.0 / 3 * 3 == 1.0   = 1
0.1 * 3 == 0.3       = 0

The first is exact because halves and quarters have exact binary forms. The second is exact because the two roundings happen to cancel. The third is not.

You cannot tell which case you are in by looking. That is precisely why the rule in §3 is "do not use ==" rather than "use it carefully" — the cases that work are accidental, and an accident that holds for your test data may not hold for the next input.

6. Choosing a Tolerance

A fixed tolerance like 1e-9 suits values near 1. It is wrong at both extremes:

set big_a 1.0e12
set big_b [expr {$big_a + 1.0}]

puts "two large values differing by 1.0:"
puts "  absolute 1e-9: [expr {abs($big_a - $big_b) < 1e-9}]"
puts "  relative 1e-9: [expr {abs($big_a - $big_b) < 1e-9 * abs($big_a)}]"

two large values differing by 1.0:
  absolute 1e-9: 0
  relative 1e-9: 1

At a trillion, a difference of 1.0 is proportionally tiny — far smaller than the accumulated error of any real calculation at that scale — but an absolute tolerance of 1e-9 calls them different.

Two approaches:

• Absolute tolerance — abs($a - $b) < 1e-9. Right when you know the scale of your values, which in scripting you usually do.
• Relative tolerance — abs($a - $b) < 1e-9 * abs($a). Right when values span many orders of magnitude.

Choose from the data. The important thing is that a tolerance is a decision you made, not a number you copied.

7. Printing Is a Separate Problem

Do not confuse displaying a value with comparing one:

set x [expr {0.1 + 0.2}]

puts "raw:         $x"
puts "format %.2f: [format %.2f $x]"
puts "format %.10f: [format %.10f $x]"

raw:         0.30000000000000004
format %.2f: 0.30
format %.10f: 0.3000000000

format rounds for display and does not change the stored value. A rounded display does not make a later == succeed — the value is unchanged.

This is also the right tool for reports and money, as Chapter 4.8 §5 showed.

8. Common Mistakes

Using == on a computed real number. §3.

Using eq instead. Worse — it compares the text, so 0.30000000000000004 and 0.3 differ as characters too.

Rounding for display and expecting the comparison to change. §7.

Copying 1e-9 without thinking about scale. §6.

Accumulating in a loop and testing for an exact total. Every addition can add error.

Concluding real arithmetic is unreliable. It is exact and deterministic. §2.

9. Exercises

Exercise 1 — see the gap. Print 0.1 + 0.2, the result of comparing it with 0.3, and the difference between them.

Exercise 2 — look inside. Use format %.20f on 0.1, 0.2, 0.3 and 0.5. Which one is exact, and why would you expect that one?

Exercise 3 — write the test. Write a close_enough procedure and use it to compare 0.1 + 0.2 with 0.3, and 1.0 with 1.000001, at two different tolerances.

Exercise 4 — find an exact case. Find two additions of decimal fractions that compare exactly equal with ==. What do the numbers have in common?

Exercise 5 — accumulate. Add 0.1 to a total ten times, then compare the result with 1.0 using == and with a tolerance. Print the total with %.20f to see what happened.

Exercise 6 — scale. Compare 1.0e12 with 1.0e12 + 1.0 using an absolute tolerance of 1e-9 and a relative one. Which answer do you want, and what does that depend on?

10. What You Can Now Do

That closes Module 4. Across ten chapters you have the whole of expr's operator language:

• arithmetic, including the two power surprises and the modulo sign rule
• the precedence table, and the parentheses habit that makes it unnecessary in your own code
• numeric comparison, and the way == changes its question based on the data
• eq and ne, which never change theirs
• logical operators, and short-circuiting as a protective device rather than an optimisation
• bitwise operators and the flags idiom, including the mask parenthesisation rule
• the conditional operator, and when if says it better
• the math functions, and which of them return whole numbers
• membership with in and ni, and that it compares text
• and now the limit of exact comparison

Modules 2 and 3 gave you how Tcl reads a command and what the values in it are. Module 4 completes the picture for expressions. Module 5 moves on to strings — the commands for searching, slicing, trimming and formatting text, which is where most real scripting work actually happens.

11. Remember

• 0.1 + 0.2 is not 0.3. Most decimal fractions have no exact binary form.
• This is deterministic, not unreliable. The arithmetic is exact about slightly different numbers.
• Do not use == or != on computed real numbers. Whole numbers are exact and fine.
• Compare against a tolerance: abs($a - $b) < $tolerance.
• Some cases are exact — halves and quarters, and some lucky cancellations. You cannot tell by looking, which is why the rule has no exceptions.
• Choose the tolerance from your data: absolute when you know the scale, relative when values span magnitudes.
• format rounds the display, not the value. It does not make == work.
• eq is not a workaround. The text differs too.