Quoting and Substitution Bugs

Part I · Module 14 · Chapter 14.8 · Quoting and Substitution Bugs

Chapter 14.7 ended on the uncomfortable point that the loud errors are the easy ones. Quoting bugs are the quiet kind. Tcl's rules are simple and completely mechanical — Module 2 laid them all out — but when you get one wrong the usual symptom is not an error. It is a loop that runs forever, a body that prints the same value three times, or a comparison that is silently false.

What makes them tractable is that the rules are mechanical. You never have to guess. You can always ask Tcl what it actually received.

1. The method: ask what Tcl saw

This is the single most useful debugging procedure in Tcl, and it is four lines:

proc saw {args} {
    puts "    got [llength $args] argument(s):"
    set i 0
    foreach a $args { puts "      \[$i\] '$a'" ; incr i }
}

args collects every argument, so saw accepts any call and reports exactly how the words were divided. Put it in front of the call that is misbehaving — saw instead of the real command name — and the ambiguity disappears:

set path "my dir/out.txt"
set pins {clk data}

puts "  saw \$path:"       ; saw $path
puts "  saw \"\$path\":"   ; saw "$path"
puts "  saw \$pins:"       ; saw $pins
puts "  saw {*}\$pins:"    ; saw {*}$pins
puts "  saw \$path extra:" ; saw $path extra

  saw $path:
    got 1 argument(s):
      [0] 'my dir/out.txt'
  saw "$path":
    got 1 argument(s):
      [0] 'my dir/out.txt'
  saw $pins:
    got 1 argument(s):
      [0] 'clk data'
  saw {*}$pins:
    got 2 argument(s):
      [0] 'clk'
      [1] 'data'
  saw $path extra:
    got 2 argument(s):
      [0] 'my dir/out.txt'
      [1] 'extra'

Every quoting question you will ever have about a call is answerable this way in about fifteen seconds. Use it before theorising.

2. The space: where it actually bites

You will often be told that "a value containing a space becomes two arguments". Row one above already disproves it: saw $path got one argument. A substitution result is never re-parsed into words — that is the once-only rule from Chapter 2.11.

The accurate rule has three parts, and only two of them are hazards:

As a command argument: safe. myproc $path 1.0 passes two arguments, whatever is in $path.

As a list: it is two elements. This is real, and it is what the space actually costs you:

set files {}
set a "my dir/one.txt"
set b "two.txt"
lappend files $a
lappend files $b
puts "  lappend \$a then \$b  -> llength [llength $files]"
foreach f $files { puts "    element: '$f'" }

set files2 {}
set files2 "$files2 $a"
puts "  string-concatenated  -> llength [llength $files2]"
foreach f $files2 { puts "    element: '$f'" }

  lappend $a then $b  -> llength 2
    element: 'my dir/one.txt'
    element: 'two.txt'
  string-concatenated  -> llength 2
    element: 'my'
    element: 'dir/one.txt'

Both report length 2, and they mean completely different things. lappend added one element that happens to contain a space, and the path survived. Building the list by string concatenation produced a two-element list where the path was torn in half — my and dir/one.txt — and a later foreach over it will try to open a file called my.

Through eval or {*}: it splits. Those re-parse, which is the one way a space becomes two arguments. 14.7 §5 has the four cases side by side; the fix is eval [list cmd $arg ...].

3. A condition frozen by quotes

Here is a bug that produces no error and never ends. The correct version first:

set i 0
while {$i < 4} { puts "    i = $i" ; incr i }

    i = 0
    i = 1
    i = 2
    i = 3

Now the same loop with the condition in quotes instead of braces:

set i 0
set guard 0
while "$i < 4" {
    puts "    i = $i"
    incr i
    if {[incr guard] >= 6} { puts "    ...still running at i = $i; stopped by a guard" ; break }
}

    i = 0
    i = 1
    i = 2
    i = 3
    i = 4
    i = 5
    ...still running at i = 6; stopped by a guard

It never stops. "$i < 4" is substituted once, when the while command is being parsed, and $i was 0 then. So while received the fixed string 0 < 4, which it re-evaluates every iteration to the same answer: true.

Braces prevent that substitution, so while receives the literal text $i < 4 and evaluates it — $i and all — fresh on every pass. That is the entire reason while's condition must be braced.

4. A body substituted too early

The same mechanism, applied to a loop body, gives a different and more insidious symptom:

set pins {clk data reset}
set p "LEFTOVER"
foreach p $pins "puts \"    pin = $p\""

    pin = LEFTOVER
    pin = LEFTOVER
    pin = LEFTOVER

Three iterations, three times the wrong value. $p inside a quoted body was substituted when the foreach command was parsed — before the loop had run at all, when p still held LEFTOVER from earlier. foreach then dutifully set p to clk, data and reset and executed a body that no longer mentions p.

With braces, the body reaches foreach unsubstituted and $p is resolved on each pass:

foreach p $pins { puts "    pin = $p" }

    pin = clk
    pin = data
    pin = reset

The variant where the variable has no value at parse time at least fails loudly:

unset -nocomplain q
set rc [catch {foreach q {a b} "puts $q"} e]
puts "  rc=$rc  $e"

  rc=1  can't read "q": no such variable

That is 14.7 §1's no such variable with an unexpected cause: the loop variable is about to exist, but the body was built before the loop started. If you see can't read naming a loop variable, suspect a quoted body.

5. expr without braces

expr is the same story and the stakes are higher, because an unbraced expression is substituted by Tcl and then parsed and evaluated again by expr. Anything the value contains gets its own chance to run:

proc threshold_from_config {} {
    # imagine this came from a config file or a tool query
    return {[puts "    *** side effect ***"] + 2}
}
set limit [threshold_from_config]
set slack 1.0

puts "  the value: $limit"
puts "  braced   : expr {\$slack > \$limit}"
puts "    -> [catch {expr {$slack > $limit}} e] / $e"
puts "  unbraced : expr \$slack > \$limit"
puts "    -> [catch {expr $slack > $limit} e] / $e"

  the value: [puts "    *** side effect ***"] + 2
  braced   : expr {$slack > $limit}
    -> 0 / 0
  unbraced : expr $slack > $limit
    *** side effect ***
    -> 1 / can't use empty string as operand of "+"

The *** side effect *** line is the point. The unbraced form took the contents of a configuration value and executed it as Tcl code. Braced, the value was treated as data and the comparison was simply false.

This is not a contrived hazard. Any script that reads thresholds from a file, a command-line argument, or a tool query, and then uses them in an unbraced expr, will execute whatever those sources contain. The braces are the boundary between data and code.

6. The decision rule, applied

Everything in this chapter reduces to the rule from Chapter 2.9, stated as a question about when:

Braces mean "substitute later, if at all". Quotes mean "substitute now".

So the question is never "which looks nicer". It is: does this text need to be re-evaluated after the current command is parsed?

The one place people hesitate is a message: puts "slack is $s ns" is correct and puts {slack is $s ns} prints the literal $s. Quotes are right there precisely because you do want the substitution now.

7. Worked example: finding one of these from the symptom

A script that is supposed to report the worst slack per pin produces three identical, blank-looking lines. No error. Here is the loop:

set rows {{clk 0.42} {data -0.11} {reset 0.80}}
set pin ""
set slack ""
foreach row $rows "lassign \$row pin slack ; puts \"    $pin -> $slack\""

     -> 
     -> 
     ->

Three iterations happened — the row count is right — and both values are missing. Rather than reasoning about it, trace what the command received: replace foreach with saw.

saw $rows "lassign \$row pin slack ; puts \"    $pin -> $slack\""

    got 2 argument(s):
      [0] '{clk 0.42} {data -0.11} {reset 0.80}'
      [1] 'lassign $row pin slack ; puts "     -> "'

Argument [1] is the answer and it took no thought at all. The body that foreach will run is puts " -> " — $pin and $slack were substituted away to empty strings when the foreach line was parsed, because that is what they held at the time. lassign is still setting them correctly on every pass; the body simply no longer refers to them.

Brace the body and the same saw call shows the variables intact:

saw $rows {lassign $row pin slack ; puts "    $pin -> $slack"}

    got 2 argument(s):
      [0] '{clk 0.42} {data -0.11} {reset 0.80}'
      [1] 'lassign $row pin slack ; puts "    $pin -> $slack"'

And it works:

foreach row $rows {lassign $row pin slack ; puts "    $pin -> $slack"}

    clk -> 0.42
    data -> -0.11
    reset -> 0.80

The habit worth taking from this: when output is wrong rather than absent, print the arguments before you reason about the logic. Three identical lines could have been a lassign bug, a list bug, or a loop bug. One saw call eliminated all three in one step.

8. Exercises

Exercise 1. Write the four-line procedure that tells you how Tcl split a command's arguments.

Exercise 2. Does myproc $path with $path containing a space pass one argument or two?

Exercise 3. while "$i < 10" {...} — what happens, and exactly why?

Exercise 4. A foreach body in quotes prints the same value every iteration. What was substituted, and when?

Exercise 5. Give two independent reasons to brace every expr.

Exercise 6. set l "$l $item" to build a list. What breaks, and what should it be?

Exercise 7. Why is puts {slack is $s} wrong when puts "slack is $s" is right, given that braces are the usual advice?