Capture Groups, and Reading Them Out

Part I · Module 13 · Chapter 13.4 · Capture Groups, and Reading Them Out

1. Parentheses capture

set line "width=32"

if {[regexp {(\w+)=(\d+)} $line -> key value]} {
    puts "key   : $key"
    puts "value : $value"
}

key   : width
value : 32

Each pair of parentheses in the pattern becomes one more variable after the subject string, in left-to-right order.

2. The first variable is the whole match

The variable before the group variables receives everything the pattern matched:

regexp {(\w+)=(\d+)} "width=32" whole key value

puts "whole : $whole"
puts "key   : $key"
puts "value : $value"

whole : width=32
key   : width
value : 32

So with two groups you supply three names. That first one is usually not wanted, which is why Tcl code conventionally calls it ->:

regexp {(\w+)=(\d+)} "width=32" -> key value
puts "$key / $value"

width / 32

3. More variables than groups

Extra names are set to the empty string rather than left alone or raising:

regexp {(\d+)} "abc123" -> a b c
puts "a='$a'  b='$b'  c='$c'"

a='123'  b=''  c=''

Fewer names than groups is also fine — the uncollected groups are simply discarded.

4. An unmatched optional group is set to empty

Here is the first half of the pair this chapter is about:

set pattern {(\w+)(:\d+)?}

regexp $pattern "name" -> base suffix
puts "absent  : base='$base' suffix='$suffix'"

regexp $pattern "name:5" -> base suffix
puts "present : base='$base' suffix='$suffix'"

absent  : base='name' suffix=''
present : base='name' suffix=':5'

The match succeeded both times. In the first, the optional group did not participate, and Tcl set its variable to the empty string — it did not leave the previous value in place.

That is the behaviour you want: after a successful match, every capture variable holds a defined value, so you can test $suffix eq "" to ask whether the optional part was there.

5. A failed match leaves the variables untouched

And here is the opposite:

set value "PREVIOUS"

set matched [regexp {(\d+)} "no digits here" -> value]

puts "regexp returned : $matched"
puts "value is still  : '$value'"

regexp returned : 0
value is still  : 'PREVIOUS'

The pattern did not match, so regexp returned 0 and wrote nothing. value still holds whatever it held before.

6. The bug this causes, in a loop

set lines {"width=32" "a comment" "depth=16"}

puts "WRONG — the capture is read without checking:"
foreach line $lines {
    regexp {(\w+)=(\d+)} $line -> key value
    puts "  $line -> $key = $value"
}

WRONG — the capture is read without checking:
  width=32 -> width = 32
  a comment -> width = 32
  depth=16 -> depth = 16

The middle line reports width = 32, which came from the previous iteration. Nothing failed, nothing warned, and a report built this way silently duplicates rows.

puts "RIGHT — the return value gates the read:"
foreach line $lines {
    if {[regexp {(\w+)=(\d+)} $line -> key value]} {
        puts "  $line -> $key = $value"
    } else {
        puts "  $line -> (no match)"
    }
}

RIGHT — the return value gates the read:
  width=32 -> width = 32
  a comment -> (no match)
  depth=16 -> depth = 16

Chapter 13.9 collects this with the other quiet failures; it appears here because it is a direct consequence of how capturing works.

7. Nested and numbered groups

Groups are numbered by the position of their opening parenthesis, left to right, including nested ones:

regexp {((\w+)\.(\w+))} "file.csv" -> whole name ext
puts "whole : $whole"
puts "name  : $name"
puts "ext   : $ext"

whole : file.csv
name  : file
ext   : csv

The outer group is 1, the first inner is 2 and the second inner is 3 — so the variable order follows the opening brackets, not the nesting depth.

When a group is only there for grouping and you do not want it captured, Tcl supports a non-capturing form (?:...):

regexp {(?:0x)?([0-9a-f]+)} "0x3fa9" -> digits
puts "digits : $digits"

regexp {(?:0x)?([0-9a-f]+)} "3fa9" -> digits
puts "digits : $digits"

digits : 3fa9
digits : 3fa9

The 0x prefix is optional and grouped, but contributes no variable — so the digits land in the first group either way.

8. A worked example

Parsing port declarations with an optional width:

set pattern {^(\w+)\s+(in|out|inout)(?:\s*\[(\d+)\])?$}

set decls {
    "clk in"
    "data out [31]"
    "rst_n in"
    "addr inout [15]"
    "nonsense here"
    "bad sideways [3]"
}

foreach d $decls {
    if {[regexp $pattern $d -> name dir width]} {
        if {$width eq ""} { set width 1 }
        puts [format "  %-18s name=%-6s dir=%-5s width=%s" "'$d'" $name $dir $width]
    } else {
        puts [format "  %-18s rejected" "'$d'"]
    }
}

  'clk in'           name=clk    dir=in    width=1
  'data out [31]'    name=data   dir=out   width=31
  'rst_n in'         name=rst_n  dir=in    width=1
  'addr inout [15]'  name=addr   dir=inout width=15
  'nonsense here'    rejected
  'bad sideways [3]' rejected

Three things are doing work. The (?:...) groups the optional width clause without consuming a capture slot. The $width eq "" test distinguishes "no width given" from a width that was given — safe only because the match succeeded, which §4 guarantees sets the variable. And the if gates every read, so the two rejected lines cannot leak the previous row's values.

9. Exercises

Exercise 1. How many variables do you supply for a pattern with three groups, and what does each receive?

Exercise 2. What is -> and why is it used?

Exercise 3. What happens if you supply more variable names than there are groups?

Exercise 4. After a successful match where an optional group did not participate, what does its variable hold?

Exercise 5. After a failed match, what do the capture variables hold?

Exercise 6. Rewrite this so it cannot report a previous iteration's values:

foreach line $lines {
    regexp {(\w+)=(\d+)} $line -> k v
    puts "$k = $v"
}

Answers

1. Four. The first receives the whole match and the next three receive groups one, two and three in the order their opening parentheses appear.

2. An ordinary variable name, not syntax. It is used by convention for the whole-match slot that is usually not wanted, because it reads as "gives you" and lets a reader's eye skip it.

3. The extras are set to the empty string. It is not an error.

4. The empty string. Tcl writes every capture variable on a successful match, so you can test eq "" to ask whether the optional part was present.

5. Whatever they held before — regexp writes nothing when it returns 0. That is why a leftover value from a previous iteration can look exactly like a fresh capture.

6.

foreach line $lines {
    if {[regexp {(\w+)=(\d+)} $line -> k v]} {
        puts "$k = $v"
    }
}

10. Remember

• Each (...) adds one variable, after a first variable that receives the whole match.
• -> is an ordinary variable name, used by convention for the whole-match slot.
• Extra variable names are set to the empty string; extra groups are discarded.
• On a successful match, every capture variable is written — an optional group that did not participate becomes "".
• On a failed match, nothing is written — the variables keep their previous values.
• Those two look identical and are opposites. Always gate the read on the return value.
• if {[regexp ...]} { ... } is the only safe shape for reading captures in a loop.
• Groups are numbered by opening parenthesis, including nested ones; (?:...) groups without capturing.