Make with Bash, Tcl and Python

Everything in this module so far has been about tools that are not installed here. This chapter is the opposite: every tool in it is real. Bash 3.2.57, Tcl 8.5.9, Python 3.13.5 and GNU Make 3.81 are all on this machine, and every transcript below is genuine.

The subject is the boundary. A recipe line is a shell command, and the shell is excellent at running programs and terrible at everything else — the moment you need a dictionary, a sort with a custom key, or a number formatted to three decimal places, you want a real language. Flows reach for Bash, Tcl or Python at exactly that point, and the handoff is where things go wrong quietly.

1. What Belongs Where

A rule of thumb that holds up well:

Keep it in the recipe when it is one command, or a short chain of commands whose only logic is "and then". Invoking a tool, making a directory, copying a file, a single grep test.

Move it to a script as soon as there is a data structure, arithmetic, more than one branch, or any formatting. A recipe with a four-level if nested in a for is a script that has not been given a filename yet, and it is far harder to test, because the only way to run it is through Make.

Which language is mostly decided for you. Tcl, because the EDA tools speak it — dc_shell, pt_shell and Questa's do-files all take Tcl, so a Tcl helper can be sourced inside the tool as well as run outside it. Python for anything that parses, aggregates or reports, because the standard library is where the work already is. Bash for short glue that is mostly running other programs.

There is one rule the choice does not change: a script must report failure through its exit status, because that is the only channel Make listens on.

2. Values Going In

Two channels: the command line and the environment. They behave differently and the difference is about whitespace.

TEST_WITH_SPACE := axi smoke test
export TESTNAME := $(TEST_WITH_SPACE)

unquoted:
	@python3 scripts/echoargs.py $(TEST_WITH_SPACE)
quoted:
	@python3 scripts/echoargs.py "$(TEST_WITH_SPACE)"
viaenv:
	@python3 scripts/echoargs.py

argc=3
  arg1=[axi]
  arg2=[smoke]
  arg3=[test]
  env TESTNAME=[axi smoke test]

argc=1
  arg1=[axi smoke test]
  env TESTNAME=[axi smoke test]

argc=0
  env TESTNAME=[axi smoke test]

Make expands its variable, then the shell splits the result on whitespace. Unquoted, one Make variable became three arguments. The script cannot tell that apart from three separate values, so it does something reasonable with the wrong input.

The environment channel has no such problem, because nothing re-splits it. That makes export the better choice for anything that might contain a space — a path, a message, a list of flags — and the command line better for short, known-shape values where seeing them in make -n output is worth more.

3. Failures Coming Back

Make listens on exactly one channel: the exit status. A script that prints ERROR and exits 0 has not failed, as far as the flow is concerned.

So each language needs its convention. Here is the same job — read a log, decide pass or fail — written three times.

#!/bin/bash
# Usage: check.sh <logfile>  -> prints a verdict, exits 0 pass / 1 fail / 2 unusable
set -euo pipefail
log="${1:?usage: check.sh <logfile>}"
[ -f "$log" ] || { echo "check.sh: no such log: $log" >&2; exit 2; }
if grep -q '^UVM_ERROR :  0' "$log"; then echo "PASS"; exit 0; else echo "FAIL"; exit 1; fi

build/icarus/axi_smoke/1/run.log  -> PASS (rc=0)
build/icarus/axi_flaky/5/run.log  -> FAIL (rc=1)
/tmp/nope.log                     -> check.sh: no such log: /tmp/nope.log (rc=2)

Three statuses, three meanings, and the set -euo pipefail line is what makes the rest trustworthy: -e stops at the first failing command, -u makes an unset variable an error rather than an empty string, and -o pipefail makes a pipeline fail if any stage does. ${1:?usage...} is the shell's own way of requiring an argument.

The Python version returns the same three statuses and does the aggregation a shell would struggle with:

#!/usr/bin/env python3
"""Usage: report.py <rundir> ...   -> a summary table, exit 1 if anything is not PASS."""
import sys, pathlib
dirs = sys.argv[1:]
if not dirs:
    print("usage: report.py <rundir> ...", file=sys.stderr); sys.exit(2)
rows, bad = [], 0
for d in dirs:
    log = pathlib.Path(d) / "run.log"
    if not log.exists():
        rows.append((d, "NOTRUN")); bad += 1; continue
    txt = log.read_text(errors="replace")
    if any(l.startswith("UVM_ERROR :  0") for l in txt.splitlines()):
        rows.append((d, "PASS"))
    else:
        rows.append((d, "FAIL")); bad += 1
w = max(len(d) for d, _ in rows)
for d, st in rows:
    print(f"{d:<{w}}  {st}")
print(f"{len(rows)-bad} passed, {bad} not passed, of {len(rows)}")
sys.exit(1 if bad else 0)

build/icarus/axi_burst/1  PASS
build/icarus/axi_flaky/1  PASS
build/icarus/axi_smoke/1  PASS
build/icarus/axi_burst/5  PASS
build/icarus/axi_flaky/5  FAIL
build/icarus/axi_smoke/5  PASS
5 passed, 1 not passed, of 6

sys.exit(1 if bad else 0) is the whole contract. Note also errors="replace" on the read: a simulator log can contain a stray byte that is not valid UTF-8, and a UnicodeDecodeError in your report script is a confusing way to learn that.

4. The Bug Tcl Gave Me

Here is the Tcl version of the same job, as I first wrote it:

set fh [open $log r]; set txt [read $fh]; close $fh
if {[regexp {^UVM_ERROR :  0} $txt]} {
    puts [format "%-28s %s" $d PASS]
} else {
    puts [format "%-28s %s" $d FAIL]; incr fail
}

build/icarus/axi_burst/1     FAIL
build/icarus/axi_flaky/1     FAIL
build/icarus/axi_smoke/1     FAIL
build/icarus/axi_burst/5     FAIL
build/icarus/axi_flaky/5     FAIL
build/icarus/axi_smoke/5     FAIL

Everything failed, including runs that Python had just called PASS on the same directories. The design was fine. The regular expression was wrong:

set txt "UVM_INFO foo\nUVM_ERROR :  0\n"
  without -line: 0
  with    -line: 1
  tcl version  : 8.5.9

In Tcl, ^ matches the start of the whole string, not the start of each line, unless you pass -line. The script read the entire log into one string, so ^UVM_ERROR could only ever match if the log's very first character began that line.

if {[regexp -line {^UVM_ERROR :  0} $txt]} {

build/icarus/axi_burst/1     PASS
build/icarus/axi_flaky/1     PASS
build/icarus/axi_smoke/1     PASS
build/icarus/axi_burst/5     PASS
build/icarus/axi_flaky/5     FAIL
build/icarus/axi_smoke/5     PASS

Now Tcl and Python agree exactly — five passes and one failure, at axi_flaky seed 5, which is the seed-dependent failure Chapter 6.2 §5 measured. Both scripts exit 1.

5. The Pipe That Ate the Status

Having fixed the Tcl, I checked its exit status:

tclsh scripts/report.tcl build/icarus/*/5 | sed 's/^/  /'
echo "rc=$?"

  build/icarus/axi_burst/5     PASS
  build/icarus/axi_flaky/5     FAIL
  build/icarus/axi_smoke/5     PASS
rc after the pipe        : 0

Zero. One of the three runs failed, the script was returning 1, and $? said 0 — because $? is the status of the last command in the pipeline, which was sed, which succeeded at indenting.

This is Chapter 4.3 §4's hazard, and knowing about it was not enough to stop me writing it. Worse, my first attempt at reading the real status was also wrong:

  pipestatus[1] (the real one): 0

pipestatus is reset by the next command, so by the time an echo asked for it, it described the echo. Captured on the line immediately after the pipeline it is correct:

  pipestatus = 1 0   (tclsh=1, sed=0)
  $? alone  = 0

  PIPESTATUS = 1 0   (tclsh=1, sed=0)

Note the shells differ in both name and base: zsh's pipestatus is 1-based, bash's PIPESTATUS is 0-based. And a Make recipe uses neither by default — it uses /bin/sh:

  direct        rc=1
  through cat   rc=0
  with pipefail rc=1

Three forms, three answers. The lesson in one line: do not put a pipe between a script and the status you need.

# 1. Best: no pipe. Capture to a file, then print it.
	@tclsh scripts/report.tcl $(DIRS) > $(REPORT); st=$$?; \
	 cat $(REPORT); [ $$st -eq 0 ] || exit 1

# 2. Ask the shell to propagate it.
	@set -o pipefail; tclsh scripts/report.tcl $(DIRS) | tee $(REPORT)

# 3. Let the script do its own formatting, so no pipe is needed at all.
	@tclsh scripts/report.tcl --indent $(DIRS)

The third is usually the right answer. If a script's output needs reshaping, the script is the place to reshape it — that is why it is a script and not a recipe.

6. Values Coming Out

$(shell ...) brings a command's stdout back into Make, and it edits it on the way:

	@echo "  one line  : [$(shell printf 'hello\n')]"
	@echo "  two lines : [$(shell printf 'a\nb\n')]"
	@echo "  trailing  : [$(shell printf 'x\n\n\n')]"
	@echo "  spaces    : [$(shell printf '  padded  ')]"

  one line  : [hello]
  two lines : [a b]
  trailing  : [x]
  spaces    : [  padded  ]

Three rules, all visible there. Trailing newlines are stripped — all of them. Interior newlines become single spaces, so a\nb arrives as a b. Spaces are left alone, including leading ones.

The newline-to-space rule is the one that decides the design. For a list, it is exactly what you want: a script that prints one filename per line becomes a Make list you can hand straight to $(words) or a prerequisite list. For anything where the line structure is the information — a report, a log extract, a table — $(shell) destroys it, and you want a file instead.

# A LIST: lines become words. Use $(shell).
SOURCES := $(shell cat $(FILELIST))

# A REPORT: lines matter. Use a file, and a rule that produces it.
$(BUILD)/report.txt: $(RESULTS)
	@python3 scripts/report.py $(dir $^) > $@

And remember $(shell ...) runs when the variable is expanded, so assign it with := (Chapter 3.1) or it re-runs at every use — Chapter 6.2 §7 measured a seed being re-drawn three times from one assignment.

7. Generating Make Code From a Script

The most useful thing a script can hand back is not a value but Make itself. A script that prints variable assignments, included by the Makefile, turns an arbitrary input format into something Make can use natively.

The input here is a test list with group comments — a format Make would find awkward:

# group: smoke
axi_smoke
axi_burst

# group: random
axi_random
axi_flaky

# group: error
axi_decode_err

The script reads it and prints Make:

# generated by scripts/gen_tests_mk.py - do not edit
ALL_TESTS := axi_smoke axi_burst axi_random axi_flaky axi_decode_err
TESTS_smoke := axi_smoke axi_burst
TESTS_random := axi_random axi_flaky
TESTS_error := axi_decode_err
TEST_GROUPS := smoke random error

And the Makefile includes it, with a rule to build it:

TESTLIST ?= tests/grouped.f
GEN      := $(BUILD)/tests.mk

-include $(GEN)

$(GEN): $(TESTLIST) scripts/gen_tests_mk.py
	@mkdir -p $(@D)
	@echo "  [gen] $(TESTLIST) -> $@"
	@python3 scripts/gen_tests_mk.py $(TESTLIST) > $@.tmp
	@mv $@.tmp $@

This works on a clean tree because of the mechanism Chapter 4.2 §9 established: Make builds a missing included file if it has a rule for it, then restarts itself and re-reads everything. Measured, in sequence:

  [gen] tests/grouped.f -> build/tests.mk
  ALL_TESTS   = axi_smoke axi_burst axi_random axi_flaky axi_decode_err
  TEST_GROUPS = smoke random error
  TESTS_smoke = axi_smoke axi_burst
  TESTS_random= axi_random axi_flaky
  count       = 5

  ALL_TESTS   = axi_smoke axi_burst axi_random axi_flaky axi_decode_err
  TEST_GROUPS = smoke random error
  count       = 5

  [gen] tests/grouped.f -> build/tests.mk
  ALL_TESTS   = axi_smoke axi_burst axi_random axi_flaky axi_decode_err axi_throughput
  TEST_GROUPS = smoke random error perf
  count       = 6

The payoff is that a group is now selectable by name, with no parsing in the Makefile at all:

$ make group G=random
  running group 'random': axi_random axi_flaky
$ make group G=smoke
  running group 'smoke': axi_smoke axi_burst

$(TESTS_$(G)) — a variable whose name is computed — is doing that, and it only works because the script emitted one variable per group.

The > $@.tmp then mv is deliberate: a generated file must never exist half-written. If the script dies partway, $@.tmp is the casualty and the old $@ is untouched, so the next make retries instead of including a truncated fragment.

8. Guarding a Generated Include

-include is designed to be silent about a missing file, and that is exactly what makes this pattern dangerous. Measured, with the input list misspelled so the script's input does not exist:

  ALL_TESTS   = axi_smoke axi_burst axi_random axi_flaky axi_decode_err axi_throughput
  TEST_GROUPS = smoke random error perf
  count       = 6
  rc=0

Exit 0, and every value is from the previous list. The flow is running yesterday's tests and reporting success.

Remove the stale file and it gets worse in a different way:

  ALL_TESTS   = 
  TEST_GROUPS = 
  TESTS_smoke = 
  count       = 0
  rc=0

Exit 0, everything empty, zero tests. That is Chapter 6.3 §2's most dangerous shape arriving by a new route: a regression that ran nothing, reported as fine.

Neither case is Make misbehaving. The dash on -include says "carry on if this file is missing", and Make carried on. The fix is to guard the input, which the dash says nothing about:

TESTLIST ?= tests/grouped.f
$(if $(wildcard $(TESTLIST)),,$(error test list "$(TESTLIST)" does not exist))

Makefile:3: *** test list "tests/nosuch.f" does not exist.  Stop.
  rc=2

9. Common Mistakes

Letting a recipe grow logic. A nested if inside a for inside a recipe is an untestable script without a filename (§1).

Passing a value with spaces unquoted. Measured: one Make variable arrived as three arguments (§2).

Reaching for export with no arguments. Everything is exported to every child, including tools that should not see it (§2).

Tcl's ^ without -line. It anchors to the start of the whole string, so a whole-file read never matches a mid-file line — measured, every run reported FAIL (§4).

Never running a verdict script against a known failure. A script that has only seen passing logs is not known to work; two inputs and two different answers is the whole test (§4).

Putting a pipe between a script and its status. $? becomes the formatter's status — measured 0 while the script returned 1 (§5).

Sampling pipestatus one command too late. It describes the most recent command, so capture it on the very next line (§5).

Assuming the recipe shell is bash. A recipe uses /bin/sh; PIPESTATUS is bash, pipestatus is zsh and 1-based (§5).

Using $(shell ...) for multi-line output. Interior newlines become spaces, so a report arrives as one long line (§6).

Writing a generated file in place. A script that dies partway leaves a truncated include; generate to .tmp and mv (§7).

Trusting -include to protect a generator's inputs. Measured twice: exit 0 with stale values, and exit 0 with zero tests (§8).

10. Check Your Understanding

Question 1. A recipe runs python3 scripts/report.py $(RUNDIRS) | tee report.txt and the regression is always green, although individual runs are known to fail. Where is the bug?

Question 2. A Tcl script reads a whole log with read and tests it with regexp {^ERROR}. It never matches. What is wrong, and why would the same code be more dangerous if it searched for a success string instead?

Question 3. SOURCES := $(shell cat filelist.f) works nicely, but SUMMARY := $(shell python3 report.py builds/*) produces one unreadable line. Explain both.

Question 4. A Makefile has -include build/tests.mk and a rule to generate it from tests/list.f. Somebody renames tests/list.f. Describe both failure modes, and the guard that prevents them.

Question 5. Why does a script's own formatting flag (--indent) solve the exit-status problem better than set -o pipefail?

11. Exercise

All four tools are real, so every step here runs as shown.

Step 1. Prove the quoting difference:

make unquoted
make quoted

argc=3
  arg1=[axi]
  arg2=[smoke]
  arg3=[test]
argc=1
  arg1=[axi smoke test]

Step 2. Check the three statuses of the Bash helper:

./scripts/check.sh build/icarus/axi_smoke/1/run.log ; echo "rc=$?"
./scripts/check.sh build/icarus/axi_flaky/5/run.log ; echo "rc=$?"
./scripts/check.sh /tmp/nope.log                    ; echo "rc=$?"

PASS
rc=0
FAIL
rc=1
check.sh: no such log: /tmp/nope.log
rc=2

Step 3. Run the Tcl and Python reports over the same directories and confirm they agree:

tclsh  scripts/report.tcl build/icarus/*/1 build/icarus/*/5 > /tmp/t.out; echo "tcl rc=$?"
python3 scripts/report.py build/icarus/*/1 build/icarus/*/5 > /tmp/p.out; echo "py  rc=$?"

tcl rc=1
py  rc=1

Both exit 1, and both name axi_flaky/5 as the one failure.

Step 4. Reproduce the Tcl bug on purpose. Remove -line from report.tcl and run step 3 again — every directory becomes FAIL.

Then make it dangerous. Change the test to search for the failure string and invert the branches, still without -line:

if {[regexp {^UVM_ERROR : [1-9]} $txt]} {
    puts [format "%-28s %s" $d FAIL]; incr fail
} else {
    puts [format "%-28s %s" $d PASS]
}

  build/icarus/axi_smoke/1     PASS
  build/icarus/axi_flaky/5     PASS
  rc=0

axi_flaky/5 is the run that genuinely fails, and it is reported as a pass with exit status 0. Same bug, opposite direction, and nothing anywhere in the flow complains — which is why §4's two-input test is not optional.

Step 5. Lose a status, then keep it:

tclsh scripts/report.tcl build/icarus/*/5 | sed 's/^/  /'
PS=(${pipestatus[@]}); echo "pipestatus = ${PS[@]}"
echo "\$? alone = $?"

pipestatus = 1 0
$? alone = 0

Step 6. See what $(shell) does to structure:

make capture

  one line  : [hello]
  two lines : [a b]
  trailing  : [x]
  spaces    : [  padded  ]

Step 7. Generate and include, then break the input:

make clean && make show
make show TESTLIST=tests/nosuch.f

The first generates and prints six tests. The second — with the guard from §8 — stops with one clear line. Remove the guard and try again twice: once with build/tests.mk present, once after deleting it. Predict each result before you run it; one serves stale values and one serves nothing, and both exit 0.

12. Remember

• Keep a recipe to commands; move logic to a script. A nested loop in a recipe is a script with no filename and no way to test it.
• Tcl because the tools speak it, Python for parsing and reporting, Bash for short glue.
• Values go in as arguments or environment, and the shell splits arguments on whitespace: measured, one Make variable became three arguments unquoted.
• The environment survives spaces without quoting, which makes export the safer channel for paths and messages.
• Failures come back only as an exit status. A script that prints ERROR and exits 0 has not failed.
• Tcl's ^ needs -line to match the start of a line in a multi-line string — measured, the whole report read FAIL without it.
• Test a verdict script against a known pass and a known failure. The same bug pointing the other way reports PASS forever, and nobody investigates green.
• Never put a pipe between a script and the status you need. Measured rc=0 while the script returned 1.
• Capture pipestatus on the very next line, and remember zsh's is 1-based, bash's PIPESTATUS is 0-based, and a recipe is /bin/sh.
• Better still, remove the pipe — let the script format its own output.
• $(shell) strips trailing newlines and turns interior ones into spaces. Right for a list, destructive for a report.
• A script can hand back Make code. Generate variable assignments, -include them, and let Make's restart mechanism handle the clean tree.
• Generate to .tmp and mv, so a dead generator never leaves a half-written include.
• -include protects the generated file, never the generator's inputs. Guard those explicitly, or measure the two silent failures yourself: stale values at exit 0, and zero tests at exit 0.

The boundaries are mapped, and the languages behind them are the same ones the EDA tools themselves run on. That is the opening for the last chapter of this module. Synthesis and timing analysis are not driven by command-line options at all — they are driven by Tcl scripts, with the tool as an interpreter, and their answer arrives as a report file rather than an exit status. Chapter 7.4 builds that flow, and measures a tool reporting a timing violation while exiting 0.