Regexes That Fail Silently

Part I · Module 13 · Chapter 13.9 · Regexes That Fail Silently

1. Failure one: the pattern never matches

Symptom: a loop that produces no output, or a count of zero, with no error.

The commonest cause is a pattern mangled by quoting (Chapter 13.5):

set lines {"width = 32" "depth = 16"}

set n 0
foreach line $lines {
    if {[regexp "^(\w+)\s*=\s*(\d+)$" $line -> k v]} { incr n }
}
puts "quoted pattern matched $n line(s)"

set n 0
foreach line $lines {
    if {[regexp {^(\w+)\s*=\s*(\d+)$} $line -> k v]} { incr n }
}
puts "braced pattern matched $n line(s)"

quoted pattern matched 0 line(s)
braced pattern matched 2 line(s)

How to see it: print the pattern as the engine receives it, and test it on one known-good string.

puts "quoted arrives as : '[set p "^(\w+)\s*=\s*(\d+)$"]'"
puts "braced arrives as : '[set q {^(\w+)\s*=\s*(\d+)$}]'"
puts "quoted on 'width = 32' : [regexp $p "width = 32"]"
puts "braced on 'width = 32' : [regexp $q "width = 32"]"

quoted arrives as : '^(w+)s*=s*(d+)$'
braced arrives as : '^(\w+)\s*=\s*(\d+)$'
quoted on 'width = 32' : 0
braced on 'width = 32' : 1

Fix: braces. Always.

2. Failure two: it matches the wrong thing

Symptom: output that looks right but contains the wrong values — a field from elsewhere in the line, or a shorter number than expected.

Two causes. Unanchored matching (Chapter 13.2 §2):

set line "v2 has 32 bits"

regexp {\d+} $line m
puts "unanchored captured : '$m'"

unanchored captured : '2'

It found the 2 in v2, not the 32. The pattern matched; it matched in the wrong place.

And a word matching inside another word:

set line "max_width = 8"

puts "plain  : [regexp {width} $line]"
puts "\\y...\\y : [regexp {\ywidth\y} $line]"

plain  : 1
\y...\y : 0

A script looking for the width setting would happily pick up max_width.

How to see it: capture the whole match and print it, even when you only want a group.

foreach line {"v2 has 32 bits" "max_width = 8"} {
    if {[regexp {\d+} $line whole]} {
        puts "  '$line' -> whole match was '$whole'"
    }
}

  'v2 has 32 bits' -> whole match was '2'
  'max_width = 8' -> whole match was '8'

Fix: anchor with ^ and $ when the pattern should describe the whole string, and \y when it should describe a whole word.

3. Failure three: stale capture variables

Symptom: a row of output repeating the previous row's values.

set lines {"width=32" "a comment" "depth=16"}

foreach line $lines {
    regexp {(\w+)=(\d+)} $line -> key value
    puts "  $line -> $key = $value"
}

  width=32 -> width = 32
  a comment -> width = 32
  depth=16 -> depth = 16

The middle line reports width = 32 because the failed match left the variables alone — the behaviour from Chapter 13.4 §5.

How to see it: clear the variables before each attempt and the duplicate becomes visible as blank:

foreach line $lines {
    set key "" ; set value ""
    regexp {(\w+)=(\d+)} $line -> key value
    puts "  $line -> '$key' = '$value'"
}

  width=32 -> 'width' = '32'
  a comment -> '' = ''
  depth=16 -> 'depth' = '16'

Fix: gate the read on the return value — section 4.

4. Failure four: an unchecked return value

This is failure three's cause, and it deserves its own name because the fix is a habit rather than a patch — and because the symptom changes depending on where the code sits.

In a loop in one scope, you get stale data (section 3's case):

foreach line {"width=32" "a comment"} {
    regexp {(\w+)=(\d+)} $line -> k v
    puts "  '$line' -> '$k=$v'"
}

  'width=32' -> 'width=32'
  'a comment' -> 'width=32'

Inside a procedure, you get an error instead — because each call gets a fresh frame (Chapter 11.1 §5), so there is no previous value to inherit:

proc parse_bad {line} {
    regexp {(\w+)=(\d+)} $line -> k v
    return "$k=$v"
}

foreach line {"width=32" "a comment"} {
    if {[catch {parse_bad $line} r]} {
        puts "  '$line' -> ERROR: $r"
    } else {
        puts "  '$line' -> '$r'"
    }
}

  'width=32' -> 'width=32'
  'a comment' -> ERROR: can't read "k": no such variable

The checked version is correct in both places:

proc parse_good {line} {
    if {![regexp {(\w+)=(\d+)} $line -> k v]} { return "(no match)" }
    return "$k=$v"
}

foreach line {"width=32" "a comment"} {
    puts "  good : '[parse_good $line]'"
}

  good : 'width=32'
  good : '(no match)'

5. All four have no error message

That is what makes them worth a chapter:

None raises. All four survive testing on data you wrote the pattern for.

6. A diagnostic routine

When a pattern misbehaves, four lines answer most questions:

proc diagnose {pattern subject} {
    puts "  pattern  : '$pattern' ([string length $pattern] chars)"
    puts "  subject  : '$subject'"
    set r [regexp -inline -- $pattern $subject]
    if {[llength $r] == 0} {
        puts "  result   : NO MATCH"
    } else {
        puts "  whole    : '[lindex $r 0]'"
        for {set i 1} {$i < [llength $r]} {incr i} {
            puts "  group $i  : '[lindex $r $i]'"
        }
    }
}

diagnose {(\w+)=(\d+)} "width=32"
diagnose {(\w+)=(\d+)} "a comment"
diagnose {\d+} "v2 has 32 bits"

  pattern  : '(\w+)=(\d+)' (11 chars)
  subject  : 'width=32'
  whole    : 'width=32'
  group 1  : 'width'
  group 2  : '32'
  pattern  : '(\w+)=(\d+)' (11 chars)
  subject  : 'a comment'
  result   : NO MATCH
  pattern  : '\d+' (3 chars)
  subject  : 'v2 has 32 bits'
  whole    : '2'

Printing the pattern's length catches the quoting failure at a glance — a pattern that should have backslashes and does not will be visibly shorter than you expect. -inline shows every group without needing to guess how many variables to supply.

7. The habit: test what should NOT match

The registry's own advice, and the most valuable thing in this chapter. A pattern tested only against inputs it should accept will pass while accepting far more than you intended.

proc check_pattern {pattern should_match should_not_match} {
    set failures 0

    foreach s $should_match {
        if {![regexp -- $pattern $s]} {
            puts "  FAIL: should have matched '$s'"
            incr failures
        }
    }
    foreach s $should_not_match {
        if {[regexp -- $pattern $s]} {
            puts "  FAIL: should NOT have matched '$s'"
            incr failures
        }
    }

    return $failures
}

puts "testing {^\\d+$} as 'is a whole number':"
set f [check_pattern {^\d+$} \
    {0 42 12345} \
    {"" "-1" "1.5" "12abc" "v2 has 32" " 12" "12 "}]
puts "  $f failure(s)"

puts "testing the unanchored {\\d+} as the same thing:"
set f [check_pattern {\d+} \
    {0 42 12345} \
    {"" "-1" "1.5" "12abc" "v2 has 32" " 12" "12 "}]
puts "  $f failure(s)"

testing {^\d+$} as 'is a whole number':
  0 failure(s)
testing the unanchored {\d+} as the same thing:
  FAIL: should NOT have matched '-1'
  FAIL: should NOT have matched '1.5'
  FAIL: should NOT have matched '12abc'
  FAIL: should NOT have matched 'v2 has 32'
  FAIL: should NOT have matched ' 12'
  FAIL: should NOT have matched '12 '
  6 failure(s)

Both patterns accept every valid input. Only the negative list separates them — and it separates them six times over.

8. A worked example

A validator with its tests beside it:

set pattern {^[A-Za-z_][A-Za-z0-9_]*(?:\[\d+\])?$}

set valid   {clk rst_n data_bus clk_div2 x1 bus[0] data[31] _private}
set invalid {"" "2fast" "has space" "bad-dash" "bus[]" "bus[x]" "bus[0" "a[0]b"}

set failures 0
foreach s $valid {
    if {![regexp -- $pattern $s]} { puts "  FAIL valid   : '$s'" ; incr failures }
}
foreach s $invalid {
    if {[regexp -- $pattern $s]} { puts "  FAIL invalid : '$s'" ; incr failures }
}
puts "[llength $valid] valid + [llength $invalid] invalid, $failures failure(s)"

8 valid + 8 invalid, 0 failure(s)

Sixteen inputs, half of them chosen to break it. The anchors stop trailing text, the first character class stops 2fast, and (?:\[\d+\])? makes the index optional while still requiring digits inside it.

9. Exercises

Exercise 1. Name the four silent failures.

Exercise 2. What is the commonest cause of a pattern that never matches?

Exercise 3. Why does regexp {\d+} "v2 has 32 bits" m put 2 in m?

Exercise 4. Why does a loop that reads captures without checking report the previous row?

Exercise 5. Give the three safe shapes for calling regexp when you will read captures.

Exercise 6. Why is a should-not-match list more valuable than a should-match list?

Answers

1. A pattern that never matches; one that matches the wrong thing; capture variables left over from a previous iteration; and an unchecked return value.

2. A pattern written in double quotes, so Tcl's backslash substitution removed the escapes before the regex engine saw it — \d becomes d and so on.

3. Because matching is unanchored and the engine takes the leftmost match, and the leftmost digits are the 2 in v2.

4. Because a failed regexp writes nothing — it returns 0 and leaves the capture variables holding whatever they held before, which on the second iteration is the first iteration's values.

5. if {[regexp ...]} { ... }, if {![regexp ...]} { continue }, or assigning the result with set ok [regexp ...] and testing it.

6. Because a pattern tested only on inputs it should accept will pass while accepting much more than intended — all four silent failures survive a positive-only test suite. The negative list is where the thinking is: what is nearly right but wrong.

10. Remember

• Four silent failures: never matches, matches the wrong thing, stale captures, unchecked return value. None raises an error.
• A pattern that never matches is usually one written in double quotes — check its length against what you expect.
• A pattern that matches the wrong thing is usually unanchored — add ^, $ or \y.
• A failed match writes nothing, so capture variables keep the previous iteration's values.
• Gate every capture read on the return value. A bare regexp statement followed by $a is the commonest regex bug in Tcl.
• Print the pattern's length and use -inline when diagnosing — both show you what the engine really got.
• Test against inputs that should NOT match, and write that list first.
• The empty string, a negative, a decimal, trailing text and leading whitespace belong in almost every negative list.