Choosing Between string match and regexp

Part I · Module 13 · Chapter 13.10 · Choosing Between string match and regexp

1. The difference that matters most

string match patterns are implicitly anchored at both ends. regexp patterns are anchored at neither.

set s "max_width = 8"

puts "string match 'width'   : [string match "width" $s]"
puts "string match '*width*' : [string match "*width*" $s]"
puts "regexp       'width'   : [regexp {width} $s]"
puts "regexp       '^width$' : [regexp {^width$} $s]"

string match 'width'   : 0
string match '*width*' : 1
regexp       'width'   : 1
regexp       '^width$' : 0

Read the pairs across. string match "width" asks "is the whole string exactly width" and needs explicit * to search inside. regexp {width} asks "does width occur anywhere" and needs explicit ^/$ to describe the whole string.

2. What glob patterns can do

More than people expect. The language from Chapter 5.5:

puts "any run      : [string match "clk*" "clk_div2"]"
puts "one char     : [string match "clk?" "clk1"]"
puts "a class      : [string match {*[0-9]*} "bus3"]"
puts "a range      : [string match {[a-z]*} "clk"]"
puts "no digits    : [string match {*[0-9]*} "clk"]"
puts "case (-nocase): [string match -nocase "CLK*" "clk_div2"]"

any run      : 1
one char     : 1
a class      : 1
a range      : 1
no digits    : 0
case (-nocase): 1

*, ?, [...] and -nocase. What it has not got: quantifiers beyond *, alternation, capture groups, anchors, and backreferences. If you need any of those, you need regexp.

3. The decision table

Every string-matching tool Part I has given you:

4. Four questions that pick the tool

Work down; stop at the first yes.

1. Is it a path? → the file commands.
2. Is the structure fixed and simple? → scan, split, string first, string range.
3. Am I only asking a yes/no about a name's shape? → string match.
4. Otherwise → regexp.

set cases {
    {"is summary.csv a csv?"          "string match"}
    {"what directory is it in?"       "file dirname"}
    {"split 'a:b:c' on colons"        "split"}
    {"read 'clk in 1' as 3 columns"   "scan"}
    {"is this field a valid number?"  "regexp ^...$"}
    {"pull key and value from 'a = b'" "regexp with groups"}
}
foreach c $cases {
    lassign $c question tool
    puts [format "  %-34s -> %s" $question $tool]
}

  is summary.csv a csv?              -> string match
  what directory is it in?           -> file dirname
  split 'a:b:c' on colons            -> split
  read 'clk in 1' as 3 columns       -> scan
  is this field a valid number?      -> regexp ^...$
  pull key and value from 'a = b'    -> regexp with groups

5. Translating between the two

When you do want to move a string match pattern to regexp, the mapping is mechanical — and shows how much the regex has to say explicitly:

foreach {glob re subject} {
    "clk*"    {^clk}     "clk_div2"
    "*_div2"  {_div2$}   "clk_div2"
    "*div*"   {div}      "clk_div2"
    "clk?"    {^clk.$}   "clk1"
} {
    puts [format "  %-9s / %-10s on %-10s -> %d %d" \
        $glob $re "'$subject'" [string match $glob $subject] [regexp $re $subject]]
}

  clk*      / ^clk       on 'clk_div2' -> 1 1
  *_div2    / _div2$     on 'clk_div2' -> 1 1
  *div*     / div        on 'clk_div2' -> 1 1
  clk?      / ^clk.$     on 'clk1'     -> 1 1

Note the regex column needs anchors in three of the four rows to say what the glob says by default — and the ? becomes ., which in a regex matches any character including ones a glob's ? would also match. The mapping is exact here; it stops being exact as soon as the data contains regex metacharacters.

6. Speed, measured — and it is a tie

Since people ask, and since the answer is not the one usually assumed. Best of five runs, 200,000 iterations each:

set s "clk_div2"
set N 200000

puts "string match : [time {string match "clk*" $s} $N]"
puts "regexp       : [time {regexp {^clk} $s} $N]"

string match : 0.0652 microseconds per iteration
regexp       : 0.0651 microseconds per iteration

They are the same. The ratio stayed between 0.99 and 1.00 across repeated runs on this machine.

That is not what most people expect, and the reason is that Tcl caches compiled regular expressions. The pattern is compiled once and the cached form is reused on every subsequent call, so a simple anchored pattern in a loop costs about what a glob costs. The compilation you might expect to pay for happens once.

7. When the simple tool is the wrong answer

Honesty requires the counter-case, or the chapter reads as "avoid regexes".

Validating structure. string match {[0-9]*} says "starts with a digit", not "is a number". There is no glob pattern for "one or more digits and nothing else" — the quantifier does not exist:

foreach s {42 4x 4 ""} {
    puts [format "  %-5s glob {[0-9]*}=%d   regexp {^\\d+$}=%d" \
        "'$s'" [string match {[0-9]*} $s] [regexp {^\d+$} $s]]
}

  '42'  glob {[0-9]*}=1   regexp {^\d+$}=1
  '4x'  glob {[0-9]*}=1   regexp {^\d+$}=0
  '4'   glob {[0-9]*}=1   regexp {^\d+$}=1
  ''    glob {[0-9]*}=0   regexp {^\d+$}=0

The glob accepts 4x. It cannot be fixed, because the language has no way to say "digits all the way to the end".

Extracting several pieces at once. regexp with groups does in one call what string first plus two string range calls does in four lines, and says more about the format while doing it.

Alternatives. (in|out|inout) has no glob equivalent short of three separate string match calls.

8. A worked example

One parser, each field using the tool that fits it:

proc parse_line {line} {
    set t [string trim $line]

    # a yes/no about shape -> string match
    if {$t eq "" || [string match "#*" $t]} { return [list skip ""] }

    # a known delimiter -> string first
    set i [string first "=" $t]
    if {$i < 0} { return [list unknown $t] }

    set key   [string trim [string range $t 0 [expr {$i - 1}]]]
    set value [string trim [string range $t [expr {$i + 1}] end]]

    # validating shape -> regexp, anchored
    if {[regexp {^-?\d+$} $value]}           { return [list integer "$key=$value"] }
    if {[regexp {^-?\d+\.\d+$} $value]}      { return [list decimal "$key=$value"] }
    return [list text "$key=$value"]
}

foreach line {"# a note" "" "width = 32" "scale = 1.5" "mode = fast" "nonsense"} {
    lassign [parse_line $line] kind detail
    puts [format "  %-16s %-8s %s" "'$line'" $kind $detail]
}

  '# a note'       skip     
  ''               skip     
  'width = 32'     integer  width=32
  'scale = 1.5'    decimal  scale=1.5
  'mode = fast'    text     mode=fast
  'nonsense'       unknown  nonsense

Three tools in eleven lines: string match for the comment check, string first for the split, and two anchored regexes for the shape tests. Each is the simplest thing that cannot be wrong about its own job.

9. What Module 13 covered

Ten chapters, deliberately placed after the simpler tools:

• When a regex is the right tool — and the four alternatives, compared on the same problems
• regexp, and the two defaults that surprise: unanchored, and leftmost
• Pattern syntax as three questions — what, where, how many
• Capture groups, and the two opposite kinds of empty
• Braces, and what a quoted pattern silently becomes
• The switches — -all, -inline, -line, -nocase, and the flat result list
• regsub, which returns a count rather than the result
• Greediness, and the two real fixes for it
• The four silent failures, and testing against what should not match
• The decision table, and the anchoring difference behind it

The thread is the one Chapter 13.1 opened with: the simplest tool that cannot be wrong wins. Regular expressions are powerful and they are also the tool with the most ways to be quietly incorrect — unanchored when you meant anchored, greedy when you meant lazy, mangled by quoting, read without checking. Knowing them properly is what lets you use them only where they earn it.

10. What comes next

Module 14 is Errors and Debugging, which formalises something you have now used in nearly every module. catch has appeared around open, around dict get, around pattern compilation and around procedure bodies; error has appeared in every validation example since Chapter 10.7. Module 14 sets out Tcl's actual error model — a command either returns a value or raises — along with return codes, errorInfo, the error messages you will meet in practice, and the debugging tools.

Module 15, Building Tcl Programs, then closes Part I.

11. Exercises

Exercise 1. What is the anchoring difference between string match and regexp?

Exercise 2. What does string match "width" "max_width = 8" return, and what about regexp {width} on the same string?

Exercise 3. Name four things a glob pattern cannot express.

Exercise 4. Give the four questions that pick a tool, in order.

Exercise 5. Translate *.csv, clk* and a?c into regexp patterns.

Exercise 6. Why can no glob pattern express "one or more digits and nothing else"?

Answers

1. A string match pattern is implicitly anchored at both ends — it describes the whole string, and you opt into searching inside with *. A regexp pattern is anchored at neither end — it searches anywhere, and you opt into describing the whole string with ^ and $.

2. string match returns 0, because the whole string is not exactly width. regexp returns 1, because width occurs inside max_width.

3. Quantifiers other than * (no +, no {n,m}), alternation (|), capture groups, anchors, and backreferences — any four of those.

4. Is it a path? Is the structure fixed and simple? Am I only asking yes/no about a name's shape? Otherwise, regexp.

5. {\.csv$}, {^clk}, and {^a.c$}.

6. Because the glob language has no quantifier meaning "one or more". [0-9]* means "a digit followed by anything", so it accepts 4x; there is no way to require that the digits continue to the end.

12. Remember

• string match is anchored at both ends; regexp at neither. That single difference explains most of the safety gap.
• Glob has *, ?, [...] and -nocase — and no quantifiers, alternation, groups, anchors or backreferences.
• Four questions in order: a path? fixed and simple? a yes/no about shape? otherwise a regex.
• Use the file commands for paths, scan for fixed columns, split for repeated separators, string first for one known delimiter.
• Use regexp to validate shape, to express optional or alternative structure, to capture a trailing free-form field, and to extract several pieces at once.
• A glob is not a regex and translating is not mechanical once metacharacters appear.
• string match and a simple regexp measured the same — Tcl caches compiled patterns, so speed is not a tiebreaker.
• No glob can say "digits and nothing else", which is exactly the kind of job a regex exists for.