Generating Tcl From Perl

Part II · Module 26 · Chapter 26.6 · Generating Tcl From Perl

1. Why a Flow Generates Tcl

Chapter 26.4's corner script was written by hand, and most of a tool script should be. But some of it depends on data that changes per project, per block or per run — and writing that part by hand means editing a script every time the data changes.

Typical generated pieces:

• Case analysis or constant ports — a list of test-mode and scan-enable ports to hold at a fixed value, taken from a design specification or a port spreadsheet.
• Per-corner settings — operating conditions and file paths per corner, from a table the project maintains.
• File lists — the sources or libraries to read, from a filelist (Module 19 built those).
• Exceptions — false paths and multicycle paths kept in a reviewed list rather than in a script.

In each case Perl reads structured data and writes Tcl that the tool will source. The Perl is simple. The difficulty is entirely in the values: each one has to arrive in the tool exactly as it was in the data, and Tcl has firm opinions about what certain characters mean.

2. The Canonical Line, Against a Stub

This is the line from this chapter's canonical example, the shape almost every generator starts with:

print {$tcl} "set_case_analysis 0 [get_ports $_]\n" for sort @ports;

To run what it generates without a timing tool, this chapter uses a stub: two commands, written in plain Tcl, that do nothing except report what they were called with. It is not a model of any real tool; it exists only so a generated file can be parsed and executed by real Tcl and the arguments it delivers can be seen.

# tool_stub.tcl -- a STUB, not a tool. It defines two commands that exist in
# timing tools, so that a generated Tcl file can be sourced by a plain tclsh
# and the arguments it delivers can be inspected. It models nothing else.
proc get_ports {pattern} {
    return "port:$pattern"
}
proc set_case_analysis {value object} {
    puts "  set_case_analysis value=$value object=$object"
}
if {[catch {source [lindex $argv 0]} msg]} {
    puts "  ERROR: $msg"
    exit 1
}

The port list is the realistic one: two control ports and two bits of a bus.

# ports to hold at 0 for functional timing
scan_en
test_mode
data_in[0]
data_in[1]

use strict;
use warnings;

open my $in, "<", "ports.txt" or die "ports.txt: $!\n";
my @ports = grep { !/^#/ && /\S/ } map { s/\s+\z//r } <$in>;
close $in;

open my $tcl, ">", "case_v1.tcl" or die "case_v1.tcl: $!\n";
print {$tcl} "set_case_analysis 0 [get_ports $_]\n" for sort @ports;
close $tcl;

#!/bin/bash
perl gen_case_v1.pl
echo "--- the generated file"
sed 's/^/  /' case_v1.tcl
echo "--- sourced by the stub"
tclsh tool_stub.tcl case_v1.tcl
echo "  exit $?"

--- the generated file
  set_case_analysis 0 [get_ports data_in[0]]
  set_case_analysis 0 [get_ports data_in[1]]
  set_case_analysis 0 [get_ports scan_en]
  set_case_analysis 0 [get_ports test_mode]
--- sourced by the stub
  ERROR: invalid command name "0"
  exit 1

The file looks exactly like what its author intended. Sorted, one command per port, readable. Then Tcl ran it, and the first bus bit stopped the file. Sorted, the two data_in lines come first, so the error happened on line 1 and nothing was applied — not even scan_en and test_mode, whose lines are perfectly valid.

What Tcl saw on that line was [get_ports data_in[0]]. Square brackets in Tcl mean command substitution: run the command inside and replace the brackets with its result. So inside get_ports's argument, [0] is a nested command — a command called 0, which does not exist.

Two details deserve attention, because they decide how this fails in practice:

The error is in the first port, so the whole file is lost here. If the bus ports had sorted last, the two control ports would have been applied and the bus bits silently not — a partial application with a single error line in a long tool log.

Whether the tool stops at such an error depends on the tool and on how the file is sourced. The stub exits 1, as a plain tclsh does. Chapter 26.4's warning applies: find out what your tool does, and never rely on it.

3. What Tcl Does to a Word

Everything in this chapter follows from three Tcl rules about a word in a command:

• $name is replaced by a variable's value. cfg_$mode becomes cfg_ followed by whatever mode holds — or an error, if there is no mode.
• [...] is replaced by the result of running a command. That is section 2's failure.
• \ starts an escape. \t is a tab, \n a newline, and \ at the very end of a line joins it to the next line.

And two rules about where words end:

• Whitespace separates words, so a path containing a space becomes two arguments.
• ; and a newline end a command, so either one inside a value ends the command early.

Tcl offers two ways to stop all of that. Double quotes group a word but still allow $, [ and \ substitution inside, so they solve only the whitespace problem. Braces group a word and turn off every substitution inside it — {data_in[0]} is exactly the eleven characters data_in[0]. Braces are the right tool almost every time. Almost, because braces themselves must balance, and a backslash just before the closing brace escapes it.

The one quoting method that handles every possible string is to put a backslash before each special character — a backslash makes the next character ordinary. The only exceptions are the newline and the tab, which have to be written as \n and \t, because a backslash followed by a real newline is Tcl's line continuation and turns into a single space.

4. Three Strategies, Seventeen Values

That is the theory. The measurement below takes seventeen values a real flow might substitute — bus bits, paths with spaces, user text with quotes and dollar signs, Windows paths, unbalanced braces, an empty string, an embedded newline — and sends each one through three strategies:

1. naive — paste the value in, as the canonical line does.
2. braces — wrap it in {...}, the most common fix.
3. tcl_word — a function that chooses the plainest form that is exactly right: nothing for a simple name, braces when they are safe, and backslash-escaping otherwise.

For every value and strategy, Perl writes a generated Tcl file that hands the value to a command which prints back the exact bytes it received, in hexadecimal. tclsh runs the file. The value is ok only if the bytes match exactly.

use strict;
use warnings;

# Values a flow really substitutes into Tcl: bus ports, paths, user text.
my @VALUES = (
    "scan_en",            "data_in[0]",         "/proj/run A/top.v",
    'cfg_$mode',          'say "hi"',           'C:\tmp',
    "{a}",                "a{b",                "a}b",
    "semi;colon",         "x]",                 "#not_a_comment",
    "two\nlines",         "tab\there",          "",
    'ends_with\\',        " leading",
);

my %STRATEGY = (
    "1 naive"    => sub { $_[0] },            # paste the value in
    "2 braces"   => sub { "{$_[0]}" },        # wrap it in braces
    "3 tcl_word" => \&tcl_word,               # quote it properly
);

# Turn any string into ONE Tcl word that reads back as exactly that string.
sub tcl_word {
    my ($s) = @_;
    return "{}" if $s eq "";
    return $s if $s =~ m{\A[A-Za-z0-9_./:+-]+\z};      # nothing special at all
    return "{$s}" if $s !~ /[{}\\]/;                    # braces stop all substitution
    (my $w = $s) =~ s/([\\\[\]{}\$";\s#])/\\$1/g;       # else escape each special char
    $w =~ s/\\\n/\\n/g;     # backslash-newline is Tcl's line continuation, so a
    $w =~ s/\\\t/\\t/g;     # newline must be written \n (and a tab \t) instead
    return $w;
}

printf "%-18s", "value";
printf " %-11s", $_ for sort keys %STRATEGY;
print "\n";
my %ok;
for my $v (@VALUES) {
    (my $show = $v) =~ s/\n/\\n/g;
    $show =~ s/\t/\\t/g;
    printf "%-18s", $show eq "" ? "(empty)" : $show;
    for my $name (sort keys %STRATEGY) {
        # A generated script that hands the value to a command which prints
        # back the exact bytes it received, in hex.
        open my $f, ">", "gen.tcl" or die "gen.tcl: $!\n";
        print {$f} 'proc record {args} { foreach a $args { binary scan $a H* h; puts $h } }', "\n";
        print {$f} "record ", $STRATEGY{$name}->($v), "\n";
        close $f;
        my $got  = `tclsh gen.tcl 2>&1`;
        my $want = unpack("H*", $v) . "\n";
        my $verdict = $? != 0 ? "ERROR" : $got eq $want ? "ok" : "WRONG";
        $ok{$name}++ if $verdict eq "ok";
        printf " %-11s", $verdict;
    }
    print "\n";
}
printf "%-18s", "exactly right";
printf " %-11s", ($ok{$_} // 0) . " of " . scalar(@VALUES) for sort keys %STRATEGY;
print "\n";

value              1 naive     2 braces    3 tcl_word 
scan_en            ok          ok          ok         
data_in[0]         ERROR       ok          ok         
/proj/run A/top.v  WRONG       ok          ok         
cfg_$mode          ERROR       ok          ok         
say "hi"           WRONG       ok          ok         
C:\tmp             WRONG       ok          ok         
{a}                WRONG       ok          ok         
a{b                ok          ERROR       ok         
a}b                ok          ERROR       ok         
semi;colon         ERROR       ok          ok         
x]                 ok          ok          ok         
#not_a_comment     ok          ok          ok         
two\nlines         ERROR       ok          ok         
tab\there          WRONG       ok          ok         
(empty)            WRONG       ok          ok         
ends_with\         WRONG       ERROR       ok         
 leading           WRONG       ok          ok         
exactly right      5 of 17     14 of 17    17 of 17

Pasting the value in is exactly right for 5 of 17, and the failures come in two kinds. Four are errors — Tcl refused the file. Eight are silently wrong — Tcl accepted the file and delivered a different value. C:\tmp arrived with a tab in it, because \t is an escape. The path with a space arrived as two separate arguments. The empty string arrived as no argument at all. None of those eight produces any message.

Brace-wrapping is exactly right for 14 of 17, and its three failures are all errors rather than silent damage: an unbalanced brace inside the value, and a value ending in a backslash, which escapes the closing brace. That is a much better failure profile than pasting — but a generator that crashes on 3 inputs in 17 is still a generator somebody will "fix" by hand-editing the output.

tcl_word is exactly right for all 17, and for the common cases its output is the most readable form: scan_en stays scan_en, data_in[0] becomes {data_in[0]}. Only values that braces cannot carry get backslashes.

Quoting a value correctly for Tcl is one problem. What the tool then does with a correctly delivered data_in[0] is another. A timing tool's get_ports treats its argument as a pattern, and in pattern matching [0] can mean a character class — "the character 0" — rather than a literal bracket. Many tools recognise bus notation and treat it specially, but the rule is the tool's and differs between tools. tcl_word guarantees the tool receives exactly the characters in your data; confirm in your tool's documentation how it then matches them.

5. Generate Data, Not Code

tcl_word makes any value safe to paste into generated code. There is a second approach that reduces how much code is generated at all, and it is usually better.

The idea comes from the third Tcl rule in section 3, read carefully: substitution happens once. When Tcl replaces $p with a variable's value, it does not then re-scan that value for brackets, dollar signs or backslashes. A value that is already in a variable is just a value.

So the generator writes only data — a Tcl list, with every element quoted once by tcl_word — and a hand-written Tcl procedure, kept in version control and reviewed like any other code, does the work:

# apply_case.tcl -- hand-written, reviewed, never generated.
# Hold every port in the list at the given value.
proc apply_case {value ports} {
    foreach p $ports {
        set_case_analysis $value [get_ports $p]
    }
    return [llength $ports]
}

use strict;
use warnings;
use Digest::MD5 qw(md5_hex);
use File::Temp qw(tempfile);

sub tcl_word {
    my ($s) = @_;
    return "{}" if $s eq "";
    return $s if $s =~ m{\A[A-Za-z0-9_./:+-]+\z};
    return "{$s}" if $s !~ /[{}\\]/;
    (my $w = $s) =~ s/([\\\[\]{}\$";\s#])/\\$1/g;
    $w =~ s/\\\n/\\n/g;
    $w =~ s/\\\t/\\t/g;
    return $w;
}

my $src = "ports.txt";
open my $in, "<", $src or die "$src: $!\n";
my $raw = do { local $/; <$in> };
close $in;
my @ports = sort grep { !/^#/ && /\S/ } map { s/\s+\z//r } split /^/, $raw;

my ($fh, $tmp) = tempfile("case_ports.tcl.XXXXXX", DIR => ".");
print {$fh} "# GENERATED by gen_case_v2.pl -- do not edit; change $src and regenerate\n";
print {$fh} "# source: $src  md5: ", md5_hex($raw), "  ports: ", scalar(@ports), "\n";
print {$fh} "set case_ports [list \\\n";
print {$fh} "    ", tcl_word($_), " \\\n" for @ports;
print {$fh} "]\n";
close $fh or die "cannot finish $tmp: $!\n";
rename $tmp, "case_ports.tcl" or die "rename: $!\n";

# sta_case.tcl -- the tool script: hand-written logic, generated data
source apply_case.tcl
source case_ports.tcl
puts "  applied to [apply_case 0 $case_ports] ports"

#!/bin/bash
perl gen_case_v2.pl
echo "--- the generated file"
sed 's/^/  /' case_ports.tcl
echo "--- sourced by the stub"
tclsh tool_stub.tcl sta_case.tcl
echo "  exit $?"
echo "--- generated twice, compared byte for byte"
cp case_ports.tcl first.tcl
perl gen_case_v2.pl
cmp first.tcl case_ports.tcl && echo "  identical"

--- the generated file
  # GENERATED by gen_case_v2.pl -- do not edit; change ports.txt and regenerate
  # source: ports.txt  md5: 942fcdaaa34df2e17bbdafb943941169  ports: 4
  set case_ports [list \
      {data_in[0]} \
      {data_in[1]} \
      scan_en \
      test_mode \
  ]
--- sourced by the stub
  set_case_analysis value=0 object=port:data_in[0]
  set_case_analysis value=0 object=port:data_in[1]
  set_case_analysis value=0 object=port:scan_en
  set_case_analysis value=0 object=port:test_mode
  applied to 4 ports
  exit 0
--- generated twice, compared byte for byte
  identical

All four ports applied, including both bus bits, and the bus bits reached set_case_analysis exactly as written in ports.txt.

Look at what is generated and what is not. The generated file contains one set command and a list of values — nothing that decides anything. The loop, the command names and the argument order live in apply_case.tcl, which is ordinary code with an ordinary history. When the tool misbehaves, there are two questions with two separate places to look: is the data right (read case_ports.tcl), and is the logic right (read apply_case.tcl). With the canonical line, both were tangled into one generated file.

The list command in the generated file is doing real work too. Tcl's list builds a well-formed list from separate words, so even a value containing spaces or braces becomes exactly one element. The trailing backslashes are line continuations, which keep one port per line so a reviewer can read the file and a diff between two versions shows exactly which ports changed.

6. A Generated File Is Evidence

When a tool run goes wrong, the generated Tcl file is often the most important thing in the run directory: it is the exact input the tool received. That only helps if the file can answer the questions people will ask of it. gen_case_v2.pl does five things for that reason.

It says it is generated, by what, and what to do instead of editing it. A generated file with no header gets hand-edited during a late-night debug, the edit is lost on the next run, and the bug "comes back".

It names its input and records a checksum of it. md5: ... identifies exactly which version of ports.txt produced this file. Two runs that disagree can be checked in one step: same checksum means same input. Digest::MD5 is part of core Perl. The checksum here is for identifying content, not for security.

It contains no timestamp. A date in the header would make every regeneration differ from the last even when nothing changed — the driver above shows the two runs are byte-identical, and that property is what lets a version-control diff or a cmp mean something. If the time matters, the run directory's own metadata records it (Chapter 26.8).

Its order is sorted (Chapter 17.8), so the same input always gives the same file, and the same reason for the second point.

It is written through a temporary file and renamed (Chapter 13.13), so a tool started while the generator is still writing never sources half a list.

Keep the generated file in the run directory, next to the tool log, and never only in a temporary location. When someone asks "what did the tool actually get?", it is the answer — and it can be re-run by hand: start the tool interactively, source the same file, and watch it happen one command at a time.

7. Two Gates Before the Tool

A tool licence and a design load are expensive. Two cheap checks catch most generated-Tcl problems before the tool ever starts.

Gate 1: is the file syntactically complete? Tcl has a command for exactly this question: info complete returns 1 if a string is one or more complete commands — braces, brackets and quotes balanced. It catches the brace-wrapping failures from section 4 (an unbalanced brace in a value). It does not catch section 2's failure, because [get_ports data_in[0]] is perfectly balanced: Tcl's syntax is fine with it, and the problem appears only when the command is run.

Gate 2: does it run against a stub? Sourcing the file with a stub that defines the tool's commands — as this chapter has done throughout — runs every substitution for real. It catches the command substitution, undefined variables, and wrong argument counts, without a licence.

#!/bin/bash
perl gen_case_v1.pl
perl gen_case_v2.pl
printf 'set_case_analysis 0 [get_ports {a{b}]\n' > unbalanced.tcl

for f in unbalanced.tcl case_v1.tcl sta_case.tcl; do
    complete=$(printf 'set f [open {%s}]; puts [info complete [read $f]]' "$f" | tclsh)
    tclsh tool_stub.tcl "$f" >/dev/null 2>&1 && run=ok || run=FAILED
    printf '  %-16s gate 1 info complete: %s   gate 2 dry run: %s\n' "$f" "$complete" "$run"
done

  unbalanced.tcl   gate 1 info complete: 0   gate 2 dry run: FAILED
  case_v1.tcl      gate 1 info complete: 1   gate 2 dry run: FAILED
  sta_case.tcl     gate 1 info complete: 1   gate 2 dry run: ok

Each gate catches something the other cannot. The unbalanced file fails the syntax gate; the bus-bit file passes it and fails only when run. The data-not-code file passes both. A generator that runs both gates before handing the file to the tool turns an expensive failure inside a licensed tool into a cheap one at generation time.

The stub catches only what it models. It does not know which ports exist in the design, which values a real set_case_analysis accepts, or how the tool matches patterns — that is still the tool's job, and the tool's log still has to be read (Chapter 26.7). The stub's job is narrower and still worth doing: to prove that the generated file parses and delivers the values you meant.

8. Exercises

1. Predict the stub. Without running it, write down exactly what tool_stub.tcl prints for a ports.txt containing cfg_$mode and C:\tmp, generated first by gen_case_v1.pl and then by gen_case_v2.pl. Run both and explain every difference from your prediction.

2. Break tcl_word. The round-trip script tests seventeen values. Find a value it does not cover — consider a carriage return, a NUL byte, and non-ASCII text — add it, and run the round trip. If tcl_word fails, fix it; if it passes, explain which rule of section 3 makes it pass.

3. Per-corner settings. Write a generator that turns this table into a Tcl dictionary the corner script from Chapter 26.4 could read: corner,voltage,temperature,lib_dir with rows ss_0p72v_125c,0.72,125,/proj/libs/ss slow and ff_0p88v_m40c,0.88,-40,/proj/libs/ff. Note the space in one path and the minus sign in a temperature. Keep data and logic separate as in section 5, and prove with the stub that every value arrives unchanged.

4. A third gate. Write a Perl check that runs after the stub dry run and compares the list of objects the stub received with the list in ports.txt. What can it catch that the first two gates cannot? Construct an input on which it fires.

5. The timestamp argument. A colleague insists that every generated file must carry the time it was generated. Write a short reply that keeps the useful part of that request — knowing when a file was produced — without breaking byte-identical regeneration. Where does the time go instead?

9. What To Remember

• Flows generate Tcl for data that changes: port lists, per-corner settings, file lists, exceptions.
• The canonical "[get_ports $_]" line fails on a bus bit: Tcl read [0] as a command. Measured: the file aborted on its first line.
• Tcl substitutes $var, [cmd] and \ escapes; whitespace, ; and newline end words and commands.
• Braces turn off all substitution — but must balance, and a trailing backslash escapes the closing brace.
• Measured over 17 values: pasting is exactly right for 5 (8 silently wrong, 4 errors), braces for 14, an escaping tcl_word for 17.
• A newline must be written \n — backslash-newline is a line continuation.
• Correct quoting delivers the exact characters; the tool's pattern matching may still treat [0] specially. Check the tool's rules.
• Generate data, not code. Substitution happens once, so a list built with tcl_word and a hand-written foreach is safe for any value.
• A generated file is evidence: a header naming the generator and input, an input checksum, no timestamp, sorted, written atomically, kept in the run directory.
• Gate 1, info complete, catches imbalance; gate 2, a stub dry run, catches substitution. Neither replaces reading the tool log.

Chapter 26.7 finally runs a real EDA tool from Perl — the Icarus Verilog simulator, the one tool available here — and measures what its exit status does and does not tell you. The answers are less reassuring than most flows assume.