Turning Text Into a Number You Can Trust

Part I · Module 3 · Chapter 3.11 · Turning Text Into a Number You Can Trust

1. Where the Problem Comes From

Values carry text — Chapter 3.1. A value you created yourself is text you control. A value that came from outside is not.

Real data contains blanks, n/a, --, headers that slipped into the rows, numbers with units attached, and fields that are simply empty because the tool that wrote them had nothing to say. Any of those reaching expr is a problem.

2. Without a Guard

set field "n/a"
puts [expr {$field * 2}]

can't use non-numeric string as operand of "*"

The script stops. For a one-off that is acceptable — you see the error and fix the input.

For a script processing thousands of rows it is not, because one bad row kills the whole run and you learn about exactly one problem per attempt.

3. The Guard

From Chapter 3.5, with -strict because it is never optional:

foreach field {-0.043 42 n/a "" "3 4"} {
    if {[string is double -strict $field]} {
        puts [format "%-8s -> numeric, doubled = %s" $field [expr {$field * 2}]]
    } else {
        puts [format "%-8s -> NOT numeric, handled" $field]
    }
}

-0.043   -> numeric, doubled = -0.086
42       -> numeric, doubled = 84
n/a      -> NOT numeric, handled
         -> NOT numeric, handled
3 4      -> NOT numeric, handled

Five fields, three rejected, no crash. Note "3 4" was rejected too — two numbers separated by a space is not one number, and the guard caught it.

4. The Part That Actually Matters

handled in that output is doing a lot of work. What should happen to a bad field?

There are four honest answers, and choosing deliberately is the real skill:

Substitute a default. Use when a missing value genuinely means something known.

set value 0
if {[string is double -strict $field]} { set value $field }

Be careful: a default of 0 in a column of measurements will quietly pull an average down. A default is only safe when it is correct, not merely convenient.

Skip the row. Use when one bad row should not affect the rest.

if {![string is double -strict $field]} { continue }

Skipping silently is how a total ends up based on 900 rows when the file had 1000. If you skip, count what you skipped.

Stop with a clear error. Use when a bad value means your assumptions about the input are wrong.

if {![string is double -strict $field]} {
    error "not a number in column 3: $field"
}

Record it and carry on. Usually the best answer for data processing — keep going, and report what was not understood.

set total 0
set used 0
set skipped {}

foreach field {12 n/a 30 "" 8} {
    if {[string is double -strict $field]} {
        set total [expr {$total + $field}]
        incr used
    } else {
        lappend skipped $field
    }
}

puts "total $total from $used values"
puts "skipped [llength $skipped]: $skipped"

total 50 from 3 values
skipped 2: n/a {}

Now the output tells the truth: a total, how many values it came from, and what was ignored. Someone reading that can judge whether to trust it.

5. Whitespace Is Already Handled

A small relief when parsing columns:

set field "  42  "

puts "passes the guard: [string is double -strict $field]"
puts "arithmetic works: [expr {$field + 0}]"

passes the guard: 1
arithmetic works: 42

Surrounding spaces are accepted by both. You may still want string trim before printing or comparing the value as text, but you do not need it to make the arithmetic work.

6. The Failure Validation Does Not Catch

Now a second problem, and it affects values that are perfectly valid numbers.

Some commands compare text by default. If you give them numbers, they compare the characters:

puts "sorted as text:    [lsort {10 9 100}]"
puts "sorted as integer: [lsort -integer {10 9 100}]"

sorted as text:    10 100 9
sorted as integer: 9 10 100

10 100 9 is correct alphabetical order — 1 comes before 9. It is useless as a numeric sort.

No guard helps here. Every one of those values is a valid number. The mistake is not in the data, it is in not telling the command how to read it.

The same distinction appears in comparison:

puts "numeric: [expr {10 > 9}]"
puts "textual: [string compare 10 9]"

numeric: 1
textual: -1

expr says 10 is greater. string compare says 10 sorts before 9, which as text it does.

Both are right about different questions. You have to say which question you are asking — which is Chapter 3.1's rule returning one more time: the command decides what a value means, so pick the command, or the option, that reads it the way you intend.

7. A Complete Shape

Putting the module together:

set rows {"12" "n/a" "30" "" "8"}

set total 0.0
set used 0
set bad 0

foreach field $rows {
    set field [string trim $field]

    if {![string is double -strict $field]} {
        incr bad
        continue
    }

    set total [expr {$total + $field}]
    incr used
}

if {$used == 0} {
    puts "no usable values"
} else {
    puts [format "mean of %d values: %.2f  (%d unusable)" \
        $used [expr {$total / $used}] $bad]
}

mean of 3 values: 16.67  (2 unusable)

Six things from this module are in those lines: text arrives untyped (3.1), the variable is set and reused (3.2), the guard uses -strict (3.5), the expression is braced (3.8), incr counts (3.9), and $total starts at 0.0 so the mean is a real number rather than a whole one (3.10).

That last detail is the one to notice, and this example is built so you can see it. All three usable values are whole numbers, so if $total had started at 0 it would have stayed a whole number, and $total / $used would have been whole-number division:

starting at 0.0  ->  mean of 3 values: 16.67
starting at 0    ->  mean of 3 values: 16.00

16.00 is a plausible wrong answer printed with two decimal places, and nothing errors. Had the data contained a decimal anywhere, $total would have become a real number on its own and the bug would have hidden until the day it did not.

8. Common Mistakes

No guard at all. One bad row stops everything. §2.

A guard with no decision. Validating and then skipping silently is still data loss. §4.

A default that is not correct. 0 in a column of measurements changes the answer. §4.

Forgetting -strict. The empty field then passes. Chapter 3.5 §4.

Sorting or comparing numbers as text. §6. Validation does not catch this.

Starting an accumulator at 0 when you want a real result. §7. Chapter 3.10.

9. Exercises

Exercise 1 — guard a mixed list. Take the values 18, n/a, -4.5, "", 0, 2 3 and print, for each, either the value doubled or a message saying it was not numeric. Which two surprised you?

Exercise 2 — choose a policy. For each situation, say which of the four options in §4 you would pick, and why:

• a temperature column where a blank means "sensor offline"
• a count column where a blank means zero
• a configuration file where a blank means someone edited it wrongly
• a 2-million-row log where a handful of lines are truncated

Exercise 3 — make the silent loss visible. Take the §4 recording example and remove the skipped tracking, leaving only the total. Run both versions on the same data. What can a reader of the second version not know?

Exercise 4 — the default that lies. Compute the mean of 10, 20, n/a and 30 twice: once treating n/a as 0, and once skipping it. Both are defensible. Which is correct, and what would you have to know about the data to decide?

Exercise 5 — sort them properly. Sort the list {100 9 10 1000} so it reads as numbers. Then sort it as text. Explain why validation would not have caught the second one.

Exercise 6 — accumulator type. Run the §7 script, then change 0.0 to 0 and run it again. Confirm you get 16.67 and 16.00. Now add one decimal value to the data and run both again — what happens, and why does that make the bug harder to find?

Exercise 7 — write the whole thing. Given the list {"45.2" "" "38.9" "bad" "41.0"}, print the mean of the usable values to one decimal place, the count used, and the count skipped. Use -strict, brace your expression, and start the accumulator correctly.

10. What You Can Now Do

That closes Module 3. Looking back at the eleven chapters, you can now:

• say what a Tcl value is, and why there is no type to ask for
• create, read, overwrite and remove variables, and test whether one exists
• decide whether some text will work as a number, and write the guard correctly
• say what counts as true and false, and why anything else is an error
• do arithmetic with expr, and explain why Tcl needs a command for it
• brace every expression, and say what the unbraced form does wrong
• count with incr
• divide without silently losing the fraction
• turn untrusted text into a number, and decide what happens when it will not convert

Module 2 gave you the model of how Tcl reads a command. Module 3 gave you the model of what the values in that command actually are. Module 4 takes the operators you have been using one at a time — comparison, logic, bitwise, precedence — and treats them properly, which is straightforward now that you know they all live inside expr.

11. Remember

• Text from outside your script is not a number until you have checked.
• The guard is string is double -strict — and -strict is required.
• Validating is the easy half. Deciding is the real work. Default, skip, stop, or record — choose deliberately.
• A script that silently drops data it did not understand is worse than one that crashes.
• If you skip, count what you skipped and say so in the output.
• Surrounding whitespace is accepted by the guard and by expr.
• A valid number can still be compared as text. lsort needs -integer or -real; expr compares numerically.
• Start an accumulator at 0.0 if you want a real result.