Test Lists as Data

Part II · Module 31 · Chapter 31.2 · Test Lists as Data

Chapter 31.1 built the unit and said it must not decide what to run. This chapter is where that decision lives: a file, one test per line, which anybody can change without touching a script.

1. Why a File and Not an Array

# inside the script: changing the regression is a code change
TESTS=(smoke fairness random burst)

# testlist.txt: changing the regression is a data change
smoke
fairness
random
burst

The second form costs a few lines of reading code and buys four things that matter at scale: the list can be generated (Chapter 31.4), reviewed as a diff of its own, varied per flow without copying the driver, and recorded with the results — so a regression's report can say which list produced it.

The reading code is the registry's hint, and it has three traps in four lines:

while IFS= read -r test; do
  run_one.sh "$test"
done < testlist.txt

2. A File Without a Trailing Newline Loses Its Last Test

$ printf 'smoke\nfairness\nrandom'   > logs/nonl.txt      # no final newline
$ printf 'smoke\nfairness\nrandom\n' > logs/withnl.txt
$ for f in logs/withnl.txt logs/nonl.txt; do
>   n=0; while IFS= read -r l; do n=$((n+1)); done < "$f"
>   printf '  %-14s while read saw %d; wc -l says %s; grep -c . says %s\n' \
>     "$(basename $f)" "$n" "$(wc -l < $f | tr -d ' ')" "$(grep -c . $f)"
> done
  withnl.txt     while read saw 3; wc -l says 3; grep -c . says 3
  nonl.txt       while read saw 2; wc -l says 2; grep -c . says 3

Two of three. read returns false when it hits end-of-file without a delimiter, so the loop body never runs for the final line — and wc -l agrees with the loop, because it counts newlines rather than lines.

3. What IFS= and -r Actually Protect

A file with a leading-space line, a literal \t, and a backslash:

$ while IFS= read -r l; do printf '    [%s]\n' "$l"; done < logs/odd.txt
    [  leading space	a\tb]
    [back\slash]
$ while read l; do printf '    [%s]\n' "$l"; done < logs/odd.txt
    [leading space	atb]
    [backslash]

Two separate corruptions in the bare form:

IFS= preserves leading and trailing whitespace. Without it, read strips it — so a line with accidental indentation silently becomes a different test name, and a deliberately indented list (a grouped file) loses its structure.

-r makes backslashes literal. Without it, read processes them as escapes: a\tb became atb and back\slash became backslash. A test name containing a backslash is unusual; a path containing one is not, and a list of filelists or configuration paths is exactly where this bites.

So IFS= read -r is not a ritual. It is two specific protections, and the measured corruption shows what each one is for.

4. The Loop That Runs One Test and Stops

This is the failure that looks like a broken regression and is a broken loop:

$ printf 'smoke\nfairness\nrandom\n' > logs/three.txt
$ n=0
$ while IFS= read -r t; do n=$((n+1)); cat >/dev/null; done < logs/three.txt
$ printf '  saw %d of 3\n' "$n"
  saw 1 of 3

One of three. The body's cat reads standard input — and standard input is the test list, because that is where < logs/three.txt attached it. The first iteration's body consumed the remaining two lines, so read found end-of-file.

cat is standing in for anything that reads standard input, and the real culprits are everywhere: ssh, a simulator in interactive mode, a tool that reads a script from stdin, mail, and anything whose arguments you got slightly wrong.

Four arrangements, measured:

# the list on a dedicated descriptor, and the body given nothing to read
while IFS= read -r -u 9 test; do
  run_one.sh "$test" < /dev/null
done 9< testlist.txt

5. Comments, Blanks and Disabled Tests

A real list is not bare names. It has sections, and tests disabled with a reason:

# the nightly smoke set
smoke
fairness

# randomised, several seeds
random

#disabled while the arbiter fix lands
#burst

$ grep -c . testlist_commented.txt
7
$ grep -v '^[[:space:]]*#' testlist_commented.txt | grep -c .
3

Seven lines, three tests. The filter is one pipeline, and it is worth writing as a function because it is used twice — once to count for the report, once to iterate:

# the list, stripped of blanks, comments and surrounding whitespace
testlist() {
  sed -e 's/[[:space:]]*#.*$//' -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' "$1" \
    | grep -v '^$'
}

$ testlist testlist_commented.txt
smoke
fairness
random
$ testlist testlist_commented.txt | grep -c .
3

Three details in that sed.

s/[[:space:]]*#.*$// strips a trailing comment, not just a whole-line one — so smoke # flaky on node 7 yields smoke. That is the form people actually write.

The whitespace strips come after, so a line that was only a comment becomes empty and is then dropped by the grep -v '^$'.

[[:space:]] rather than \s. BSD sed has no \s — Chapter 14.2's portability point — and the way it fails is worth measuring rather than assuming.

$ printf 'smoke   # note\nsmokes  # note\nsss# note\n' > /tmp/s.txt
$ sed 's/[[:space:]]*#.*$//' /tmp/s.txt | sed -n l
smoke$
smokes$
sss$
$ sed 's/\s*#.*$//' /tmp/s.txt | sed -n l
smoke   $
smokes  $
$

\s* means zero or more literal s. On the first two lines it matches zero of them, so the comment is still removed and the whitespace before it survives. On sss# note it matches all three — and the line becomes empty.

Then the rest of the filter hides most of the evidence:

$ # the full three-stage filter, with each comment pattern
    with [[:space:]] : smoke|smokes|sss|
    with \s          : smoke|smokes|

The later whitespace strips clean up the surviving spaces, so the only remaining symptom is that the test named sss has disappeared from the regression. No error, no warning, and a count that is short by one — which Chapter 31.6's reconciliation would catch and a wc -l check would not.

A name ending in s immediately before a comment is not a contrived case: regs, flags, addrs, bursts.

6. A Name the Testbench Does Not Have

Section 5 removed comments and blanks from the list. What survives that is a non-empty string, and nothing so far has asked whether it names a test that exists.

A list with one name misspelt — randm for random — run exactly as the driver would run it:

=== a test list with one misspelt name, run as usual ===
  smoke        rc=0  TEST PASSED
  fairness     rc=0  TEST PASSED
  randm        rc=1  ERROR unknown test 'randm' TEST FAILED

The run failed, which is the right thing for the testbench to do — a run that cannot find its test must not report success. Its log:

TB  test=randm seed=1 cycles=120000 verbosity=LOW
TB  ERROR unknown test 'randm'
TB  TEST FAILED  test=randm checks=0 errors=1
tb/tb_arb.v:83: $finish called at 25 (1s)

Now read that as it arrives in the morning: one failing test in a regression of three. The report says randm FAIL. Somebody opens a waveform, finds no activity, and eventually reads the second line of the log. The information was there, and the cost of it being there rather than in the launcher's output is a person's hour.

The giveaways in the log are checks=0 and $finish called at 25 — zero checks and twenty-five time units. A test that did nothing is distinguishable from a test that found a bug, which is why Chapter 29.4 argues for reading a count alongside a verdict. But that is detection after the fact.

Validating the list against the testbench

The testbench knows exactly which names it implements, and it says so in its source:

$ grep -oE 'testname == "[a-z0-9_]+"' tb/tb_arb.v | sed 's/.*"\(.*\)"/\1/' | sort -u
fairness
random
smoke

Three names. So the list can be checked before anything is launched:

#!/bin/bash
# checklist LIST TBSRC -- refuse a list that names a test the bench does not have.
set -u
list=${1:-}; tb=${2:-}
[ -r "$list" ] || { echo "checklist: cannot read '$list'" >&2; exit 2; }
[ -r "$tb" ]   || { echo "checklist: cannot read '$tb'" >&2; exit 2; }
known=$(grep -oE 'testname == "[a-z0-9_]+"' "$tb" | sed 's/.*"\(.*\)"/\1/' | sort -u)
[ -n "$known" ] || { echo "checklist: found no test names in $tb -- refusing to guess" >&2; exit 2; }
bad=0
while IFS= read -r t; do
  case $t in ''|\#*) continue;; esac
  t=${t%%#*}; t=$(printf '%s' "$t" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
  [ -n "$t" ] || continue
  if printf '%s\n' "$known" | grep -qx "$t"; then
    printf '  %-14s known\n' "$t"
  else
    printf '  %-14s NOT A TEST in %s\n' "$t" "$(basename "$tb")"; bad=$((bad+1))
  fi
done < "$list"
[ "$bad" -eq 0 ] || { printf 'checklist: %d name(s) the testbench does not implement\n' "$bad" >&2; exit 1; }

On the misspelt list:

$ checklist typo.txt tb/tb_arb.v
  smoke          known
  fairness       known
  randm          NOT A TEST in tb_arb.v
checklist: 1 name(s) the testbench does not implement
  exit status: 1

On a correct one, including a commented entry:

$ cat good.txt
smoke
fairness
random   # the long one
$ checklist good.txt tb/tb_arb.v
  smoke          known
  fairness       known
  random         known
  exit status: 0

grep -qx is doing real work there, and it is worth measuring rather than assuming. Against the three known names:

  random    grep -qx: match   grep -q (no -x): match
  rand      grep -qx: no      grep -q (no -x): match
  andom     grep -qx: no      grep -q (no -x): match
  smoke     grep -qx: match   grep -q (no -x): match
  smok      grep -qx: no      grep -q (no -x): match

Without -x, every one of the three typos passes — not only the prefix rand but the suffix andom and the truncation smok, because an unanchored pattern matches a substring anywhere in the line. All three would then be caught in simulation instead, which is the entire failure this check exists to prevent. -x requires the whole line to match, and the real list confirms it:

$ printf 'smoke\nrand\n' > prefix.txt
$ checklist prefix.txt tb/tb_arb.v
  smoke          known
  rand           NOT A TEST in tb_arb.v
checklist: 1 name(s) the testbench does not implement
  exit status: 1

7. The Driver

#!/bin/bash
# runlist LIST — run every test in LIST, collecting every result.
set -u
list=${1:-}
[ -n "$list" ] || { echo "usage: runlist LIST" >&2; exit 2; }
[ -r "$list" ] || { echo "runlist: cannot read $list" >&2; exit 2; }
root=${ROOT:-}; [ -n "$root" ] || { echo "runlist: set ROOT" >&2; exit 2; }
runs=${RUNS:-}; [ -n "$runs" ] || { echo "runlist: set RUNS" >&2; exit 2; }
mkdir -p "$runs" || exit 2

testlist() {
  sed -e 's/[[:space:]]*#.*$//' -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' "$1" \
    | grep -v '^$'
}

n=$(testlist "$list" | grep -c .)
[ "$n" -gt 0 ] || { echo "runlist: $list contains no tests" >&2; exit 2; }
printf 'runlist: %s tests from %s\n' "$n" "$list"

# the list that produced these results, kept with them
testlist "$list" > "$runs/testlist.used"

pass=0 fail=0 noverdict=0 error=0
while IFS= read -r -u 9 spec || [ -n "$spec" ]; do
  "$root/bin/run_one.sh" "$spec" < /dev/null
  case "$?" in
    (0) pass=$((pass+1));      v=PASS ;;
    (1) fail=$((fail+1));      v=FAIL ;;
    (3) noverdict=$((noverdict+1)); v=NO-VERDICT ;;
    (*) error=$((error+1));    v=ERROR ;;
  esac
  printf '%-16s %s\n' "$spec" "$v"
done 9< "$runs/testlist.used"

printf '\n%d of %d tests: %d passed, %d failed, %d no verdict, %d error\n' \
       "$((pass+fail+noverdict+error))" "$n" "$pass" "$fail" "$noverdict" "$error"
[ $((fail + noverdict + error)) -eq 0 ]

Five decisions.

It counts first and refuses an empty list. A driver that runs zero tests and reports success is the most convincing false green available — nothing failed.

It reconciles the count. %d of %d tests prints what it ran against what it found, so §2's lost-last-line defect would show as 2 of 3 rather than as a clean pass.

It writes testlist.used and then iterates that, which gives two things: the results record which list produced them, and the filter runs once instead of per iteration.

|| [ -n "$spec" ] for the unterminated last line, and fd 9 plus < /dev/null for §4. Three traps, three guards.

The guards exit 2, per Chapter 31.1 §4 — 1 already means a failing test.

Practice

1. Write a list with no trailing newline and count it three ways: while read, wc -l, grep -c ..
2. Decide which of the three is right, and explain why the majority is not.
3. Add || [ -n "$l" ] and confirm the count changes.
4. Build a sanity check on wc -l and work out why it agrees with the bug.
5. Put a leading space and a backslash in a list and read it with and without IFS= read -r.
6. Say which protection each of IFS= and -r provides.
7. Write the loop whose body runs cat and record how many iterations happen.
8. Explain where the body's standard input was pointing.
9. Fix it with < /dev/null, then with fd 9, then with both.
10. Run the fd-9-only version from an interactive shell and see what it does.
11. Write a list with sections, blank lines and a disabled test, and count total versus actual.
12. Write the filter and confirm it handles a trailing comment on a test line.
13. Replace [[:space:]] with \s and run it on smoke # x, smokes # x and sss# x. Find which line disappears.
14. Put a # inside a test name and watch the filter truncate it.
15. Write runlist and confirm it refuses an empty list and reconciles its count.
16. Misspell one name in your test list and run the regression; see what the report calls it.
17. Read the failing run's log and find the two fields that say it never tested anything.
18. Extract the names your testbench actually implements from its source.
19. Write checklist and run it on the misspelt list.
20. Drop the -x from its grep and confirm that a prefix, a suffix and a truncation all pass.
21. Point it at an empty file and confirm it refuses rather than reporting every test unknown.
22. Say what it would do to a testbench whose test names are in a case statement.

Check Your Understanding

1. What four things does a file buy over an in-script array?
2. How many tests did while read see in a file of three without a trailing newline?
3. Which of the three counting methods is correct, and why is it the odd one out?
4. Why will a wc -l sanity check fail to catch the problem?
5. What does IFS= protect, and what does -r protect?
6. What did bare read do to a\tb and back\slash?
7. Why did the loop with cat in its body run once?
8. Why is the fd-9 form alone insufficient, and what is the worse failure it produces?
9. Why are -u 9 and < /dev/null both needed?
10. In the commented list, how many lines and how many tests?
11. Why do the whitespace strips come after the comment strip?
12. What does \s*# do on BSD sed, and which test disappears because of it?
13. What happens if a test name contains #, and what makes that failure bad?
14. Why does runlist refuse an empty list?
15. What does printing %d of %d catch?
16. A list names a test the testbench does not implement. What does the run do, and what does the morning report say?
17. Which two fields in that log distinguish it from a test that found a bug?
18. Where does the testbench state which names it implements?
19. What does grep -qx prevent that grep -q does not — and which three typos were measured passing?
20. Why must checklist exit 2 when it finds no test names at all?
21. What would a naive version do with a testbench using a case statement, and why is one of the two failure modes worse?
22. What is the robust alternative to parsing the testbench's source?

8. Remember

• A non-empty list entry is not a test. A misspelt name fails in simulation and arrives in the report looking exactly like a design bug.
• Validate the list against the names the testbench implements, before launching anything.
• grep -qx, never bare grep -q: unanchored, a prefix, a suffix and a truncation all matched.
• A checker that cannot find any known names must exit 2, not 0 and not "everything is unknown".
• Parsing a testbench's source is a stopgap. Having the testbench print its own test list is the real fix — ask the tool rather than infer from its inputs.
• The list is data. Generated, diffable, per-flow, and recorded with the results.
• A file without a trailing newline loses its last test: while read 2, wc -l 2, grep -c . 3 — and the majority is wrong.
• A wc -l sanity check agrees with the bug. Count with grep -c ..
• || [ -n "$var" ] in the loop; grep -c . in the report.
• IFS= keeps whitespace, -r keeps backslashes. Bare read turned a\tb into atb.
• A body that reads stdin eats the list — 1 of 3. Use fd 9 and < /dev/null.
• fd 9 alone leaves the body on the terminal, so it works in cron and hangs for a human.
• Strip trailing comments, then whitespace, then blanks, with [[:space:]]. BSD sed's \s means literal s, and s/\s*#.*$// deleted a test named sss from the list entirely — silently.
• Decide what # means in a name and write it down; a truncated test name runs the default and passes.
• Refuse an empty list, and print %d of %d.

9. What Is Next

The list of tests is data. Chapter 31.3 adds the other axis — the seed — and establishes why an unrecorded one turns a real failure into a story nobody can check.