Testing an awk Script

Part II · Module 8 · Chapter 8.7 · Testing an awk Script

1. Why This Matters

Everything in Module 8 so far makes a script harder to break. This chapter is about noticing when it has broken anyway.

The specific risk is that an awk script is small enough to feel self-evident. Nobody writes a test for a seven-line program — and then a tool upgrade adds a column, or somebody reorders two printf arguments, and the script keeps running and produces a number that is wrong by a factor nobody can see.

A test here is not a framework. It is a sample input, a file holding the output that input should produce, and one command that compares them.

2. The Script Under Test

A script with the three-status contract from Chapter 8.3:

# summarise.awk — reduce a regression log to a verdict.
#
# Usage:  awk -f summarise.awk run.log
# Exit:   0 no failures
#         1 at least one failure
#         2 could not evaluate (bad header, or no data)

NR == 1 && $1 != "test" { print "ERROR: unexpected header" > "/dev/stderr"; exit 2 }
NR == 1 { next }
$2 == "FAIL" { fails++ }
{ n++ }
END {
    if (!n) { print "ERROR: no data records" > "/dev/stderr"; exit 2 }
    printf "%d of %d failed\n", fails + 0, n
    if (fails) exit 1
}

test result time
alu PASS 120
fifo FAIL 95
uart PASS 88

awk -f summarise.awk tests/sample.log
echo "status: $?"

1 of 3 failed
status: 1

One failure of three, status 1. That pair — the line and the status — is the script's contract.

3. The Expected Output File

Store the output that the fixture should produce:

1 of 3 failed

awk -f summarise.awk tests/sample.log | diff -u tests/expected.txt - && echo "PASS: output matches"

PASS: output matches

diff -u compares the stored file against standard input (-), and says nothing when they match. The && prints the confirmation only on success.

That is the whole test. Two files and one command — and it will catch any change to the output, including ones nobody thought to look for.

4. Watching the Test Fail

A test that has never failed is a comment. Breaking the script deliberately:

sed 's/fails + 0, n/n, fails + 0/' summarise.awk > broken.awk
awk -f broken.awk tests/sample.log | diff -u --label expected --label actual tests/expected.txt -
echo "diff status: $?"

--- expected
+++ actual
@@ -1 +1 @@
-1 of 3 failed
+3 of 1 failed
diff status: 1

The --label options replace the filenames in the header, which by default carry modification timestamps — so the diff is stable enough to compare between runs.

The sed swapped the two printf arguments, so the counts are reported the wrong way round. 3 of 1 failed is nonsense to a human and perfectly plausible to a machine — three failures out of one test.

The diff names exactly what changed, with - for expected and + for actual, and diff exits 1. That status is what a test runner branches on.

This failure is worth dwelling on because of how it would arrive in practice: not from a sed, but from somebody adding a column to the printf and miscounting the arguments. Chapter 7.1 section 6 established that too many printf arguments are discarded silently, so an edit of that kind produces no error at all.

5. The Status Is Part of the Contract

The output is half the contract. Four fixtures, four outcomes:

test result time
alu PASS 120

name result
alu PASS

test result time

for f in tests/sample.log tests/clean.log tests/badhdr.log tests/hdronly.log; do
  printf "%-20s " "$f"
  out=$(awk -f summarise.awk "$f" 2>&1); rc=$?
  printf "rc=%d  [%s]\n" "$rc" "$(echo "$out" | tr '\n' '|')"
done

tests/sample.log     rc=1  [1 of 3 failed|]
tests/clean.log      rc=0  [0 of 1 failed|]
tests/badhdr.log     rc=2  [ERROR: unexpected header|ERROR: no data records|]
tests/hdronly.log    rc=2  [ERROR: no data records|]

Three of the four are exactly as designed. The third is not.

6. A Defect a Status-Only Test Would Pass

Look at tests/badhdr.log: two error messages for one fault.

The header rule printed its message and called exit 2. Chapter 5.6 section 7 established that exit still runs END, where the !n check found no data records — because exit had stopped the input — and printed its own error. The second message is a consequence of the first and is actively misleading: a reader may conclude the file is both malformed and empty.

The status is 2 either way, so a test that checks only the status cannot see it:

rc_only() { awk -f "$1" tests/badhdr.log >/dev/null 2>&1; echo "$?"; }
printf "unfixed script rc: %s\n" "$(rc_only summarise.awk)"

unfixed script rc: 2

Correct status, defective behaviour. The fix is Chapter 8.3 section 7's bail flag:

# fixed.awk — summarise.awk with one message per fault.
NR == 1 && $1 != "test" { print "ERROR: unexpected header" > "/dev/stderr"; bail = 2; exit 2 }
NR == 1 { next }
$2 == "FAIL" { fails++ }
{ n++ }
END {
    if (bail) exit bail
    if (!n) { print "ERROR: no data records" > "/dev/stderr"; exit 2 }
    printf "%d of %d failed\n", fails + 0, n
    if (fails) exit 1
}

for f in tests/sample.log tests/clean.log tests/badhdr.log tests/hdronly.log; do
  printf "%-20s " "$f"
  out=$(awk -f fixed.awk "$f" 2>&1); rc=$?
  printf "rc=%d  [%s]\n" "$rc" "$(echo "$out" | tr '\n' '|')"
done

tests/sample.log     rc=1  [1 of 3 failed|]
tests/clean.log      rc=0  [0 of 1 failed|]
tests/badhdr.log     rc=2  [ERROR: unexpected header|]
tests/hdronly.log    rc=2  [ERROR: no data records|]

One message per fault, same statuses.

7. A Runner

Putting it together — a shell script checking both halves of the contract for every fixture:

#!/bin/sh
# run_tests.sh — check summarise.awk against its fixtures.
# Each case names the fixture, the expected status and the expected output.

pass=0; fail=0

check() { # name  fixture  expected_rc  expected_out
  out=$(awk -f fixed.awk "$2" 2>/dev/null); rc=$?
  if [ "$rc" = "$3" ] && [ "$out" = "$4" ]; then
    pass=$((pass + 1)); printf "  ok    %-12s rc=%s\n" "$1" "$rc"
  else
    fail=$((fail + 1))
    printf "  FAIL  %-12s rc=%s (want %s)  out=[%s] (want [%s])\n" "$1" "$rc" "$3" "$out" "$4"
  fi
}

check with-fails  tests/sample.log  1 "1 of 3 failed"
check clean       tests/clean.log   0 "0 of 1 failed"
check bad-header  tests/badhdr.log  2 ""
check header-only tests/hdronly.log 2 ""

printf "%d passed, %d failed\n" "$pass" "$fail"
[ "$fail" = 0 ]

chmod +x run_tests.sh
./run_tests.sh
echo "runner status: $?"

  ok    with-fails   rc=1
  ok    clean        rc=0
  ok    bad-header   rc=2
  ok    header-only  rc=2
4 passed, 0 failed
runner status: 0

Four cases, both halves checked, and the runner's own exit status usable by CI.

Three properties worth noting. The expected stdout for the error cases is the empty string, because those messages go to stderr — which is itself a tested property. The runner exits non-zero if anything failed, so it is a gate rather than a report. And each case is one line, so adding a fixture is cheap enough that people actually do it.

8. Proving the Runner Can Fail

A green test suite means nothing until you have seen it go red:

cp fixed.awk keep.awk
sed 's/fails + 0, n/n, fails + 0/' keep.awk > fixed.awk
./run_tests.sh
echo "runner status: $?"
cp keep.awk fixed.awk

  FAIL  with-fails   rc=1 (want 1)  out=[3 of 1 failed] (want [1 of 3 failed])
  FAIL  clean        rc=0 (want 0)  out=[1 of 0 failed] (want [0 of 1 failed])
  ok    bad-header   rc=2
  ok    header-only  rc=2
2 passed, 2 failed
runner status: 1

Two cases caught, not one — the swapped arguments break every fixture that reaches the printf, which is both of the ones with data. The two status-2 fixtures still pass, because they never reach it.

Three things to notice in that output. Each failure names the actual and expected output, so the change is identifiable without re-running anything. The statuses are still correct (rc=1 (want 1)), which is the section 6 point again — a status-only runner would have reported all four as passing. And the runner exits 1, so it works as a CI gate rather than a report.

The last cp restores the good script, which matters in a worked example and matters more in a real test directory: a runner that leaves the thing it was testing broken is its own kind of defect.

9. What to Keep in the Repository

A tested awk script is a small directory:

summarise.awk          the program
run_tests.sh           the runner
tests/sample.log       a fixture per outcome
tests/clean.log
tests/badhdr.log
tests/hdronly.log
tests/expected.txt     expected output where a diff is clearer than a string

Three guidelines for the fixtures.

Keep them small. Three or four records is enough to exercise a rule, and a small fixture makes a diff readable. A 10,000-line real log as a fixture means a failure you cannot interpret.

Keep them synthetic. A real log may contain customer or project names, and it will certainly change when the tool is upgraded. A hand-written fixture is stable and shareable — which is the same reason every example in this track is synthetic.

One fixture per outcome, not per input. The four here cover status 0, status 1 and two distinct status-2 paths. A fifth fixture that also produces status 1 adds maintenance and no coverage.

10. When the Format Legitimately Changes

The test will fail when the tool's output format changes, and that is the test working rather than breaking.

The procedure is to update the fixture and the expectation together, in one commit, so the diff shows both — which makes the change reviewable:

printf 'test result time cfg\nalu PASS 120 base\nfifo FAIL 95 base\nuart PASS 88 base\n' > tests/sample_v2.log
awk -f keep.awk tests/sample_v2.log
echo "status: $?"

1 of 3 failed
status: 1

A fourth column was added and the script still works, because it refers to $1 and $2 rather than $NF. That is luck rather than design — Chapter 8.3 section 4's field-count check is what turns it from luck into a decision:

awk 'NR == 1 { want = NF; next } NF != want { printf "record %d has %d field(s), header has %d\n", NR, NF, want; bad++ } END { printf "%d malformed of %d\n", bad + 0, NR - 1 }' tests/sample_v2.log

0 malformed of 3

Consistent, so the new format is internally coherent and the script's assumptions still hold. Had the extra column appeared on only some rows, this check would have said so.

11. Common Mistakes

No test at all, because the script is short. Length is unrelated to consequence.

Testing only the exit status. It collapses many behaviours onto one integer — section 6.

Not testing stderr. Two different errors look identical if you capture only stdout.

Never watching the test fail. A green suite that has never gone red proves nothing — section 8.

A fixture per input rather than per outcome. Maintenance without coverage.

A real log as a fixture. It changes, and it may carry project names.

A huge fixture. The diff becomes unreadable.

Updating the expectation without the fixture. The commit then hides what changed.

A runner that reports and exits 0. It must be a gate — Chapter 5.6 section 9.

A runner whose summary disagrees with its gate. Compute both from the same variables — section 8.

12. Exercises

Exercise 1 — Run the test

awk -f keep.awk tests/sample.log | diff -u tests/expected.txt - && echo "PASS"

PASS

Silence from diff means the output matches.

Exercise 2 — Watch it fail

Break the script and confirm the test catches it.

sed 's/fails + 0, n/n, fails + 0/' keep.awk > b2.awk
awk -f b2.awk tests/sample.log | diff -u --label expected --label actual tests/expected.txt -
echo "diff status: $?"

--- expected
+++ actual
@@ -1 +1 @@
-1 of 3 failed
+3 of 1 failed
diff status: 1

Exercise 3 — Build the status matrix

Report the status and output for every fixture.

for f in tests/sample.log tests/clean.log tests/badhdr.log tests/hdronly.log; do
  printf "%-20s " "$f"
  out=$(awk -f keep.awk "$f" 2>&1); rc=$?
  printf "rc=%d  [%s]\n" "$rc" "$(echo "$out" | tr '\n' '|')"
done

tests/sample.log     rc=1  [1 of 3 failed|]
tests/clean.log      rc=0  [0 of 1 failed|]
tests/badhdr.log     rc=2  [ERROR: unexpected header|]
tests/hdronly.log    rc=2  [ERROR: no data records|]

Four fixtures, four distinct outcomes, one message each.

Exercise 4 — Debug a doubled message

The script below passes a status-only test and misbehaves.

awk -f summarise.awk tests/badhdr.log 2>&1
echo "status: $?"

ERROR: unexpected header
ERROR: no data records
status: 2

exit 2 in the header rule still ran END, where the no-data check fired because the input had been stopped — Chapter 5.6 section 7.

awk -f keep.awk tests/badhdr.log 2>&1
echo "status: $?"

ERROR: unexpected header
status: 2

The bail flag suppresses the END path. Both versions return 2, which is why only an output comparison finds this.

Exercise 5 — Engineering application

Write a runner that checks stdout, stderr and status separately, so all three parts of the contract are independently verified.

#!/bin/sh
# three_part.sh — check stdout, stderr and status separately.
p=0; f=0
chk() { # name fixture rc out err
  o=$(awk -f keep.awk "$2" 2>/dev/null); r=$?
  e=$(awk -f keep.awk "$2" 2>&1 >/dev/null)
  if [ "$r" = "$3" ] && [ "$o" = "$4" ] && [ "$e" = "$5" ]; then
    p=$((p + 1)); printf "  ok    %s\n" "$1"
  else
    f=$((f + 1)); printf "  FAIL  %s rc=%s/%s out=[%s]/[%s] err=[%s]/[%s]\n" "$1" "$r" "$3" "$o" "$4" "$e" "$5"
  fi
}
chk with-fails  tests/sample.log  1 "1 of 3 failed" ""
chk clean       tests/clean.log   0 "0 of 1 failed" ""
chk bad-header  tests/badhdr.log  2 "" "ERROR: unexpected header"
chk header-only tests/hdronly.log 2 "" "ERROR: no data records"
printf "%d passed, %d failed\n" "$p" "$f"
[ "$f" = 0 ]

chmod +x three_part.sh
./three_part.sh
echo "status: $?"

  ok    with-fails
  ok    clean
  ok    bad-header
  ok    header-only
4 passed, 0 failed
status: 0

Each case now pins three things: the status, the stdout, and the exact stderr message. The two status-2 cases are distinguishable because their stderr differs — precisely the property section 6's defect violated.

The cost is running the script twice per case, once per stream. For a fast script that is irrelevant; for a slow one, redirect both streams to temporary files in a single run instead.

Exercise 6 — Challenge: prove the stderr check catches section 6's defect

Point the three-part runner at the unfixed script and show it failing where a status-only check passes.

sed 's/keep.awk/summarise.awk/' three_part.sh > three_part_unfixed.sh
chmod +x three_part_unfixed.sh
./three_part_unfixed.sh
echo "status: $?"

  ok    with-fails
  ok    clean
  FAIL  bad-header rc=2/2 out=[]/[] err=[ERROR: unexpected header
ERROR: no data records]/[ERROR: unexpected header]
  ok    header-only
3 passed, 1 failed
status: 1

Caught. Read the failing line closely: rc=2/2 and out=[]/[] — the status matches and the stdout matches. Only the stderr differs, showing both messages where one was expected.

The status-only check on the very same script passes:

awk -f summarise.awk tests/badhdr.log >/dev/null 2>&1
echo "status-only check: rc=$? (expected 2, so it passes)"

status-only check: rc=2 (expected 2, so it passes)

The same defect: invisible to one test, named by the other — and the rc=2/2 out=[]/[] in the failure line is the proof that neither of the other two checks could have found it.

That is the chapter in one pair of commands, and it generalises past awk. A test asserts on what it compares; this contract has three parts, so a test comparing one part verifies a third of it. The same reasoning appeared in Chapter 7.5 section 8 — a guard must assert on something the failure cannot corrupt — and in Chapter 8.6 section 7, where a two-key test could not distinguish a working sort from an inert one.

13. Summary

• A test here is a fixture, an expected output and one diff — not a framework.
• diff -u expected - compares a stored file against the script's output and exits 1 on a difference.
• Watch the test fail before trusting it. A suite that has never gone red proves nothing.
• The exit status is half the contract, and 0, 1 and 2 mean different things — Chapter 8.3 §10.
• Test the output, not just the status. A status collapses many behaviours onto one integer.
• Test stderr separately, or two distinct errors become indistinguishable.
• A status-only test passed a script emitting two errors for one fault — an observed defect, not a hypothetical.
• One fixture per outcome, small and synthetic, kept in the repository.
• The runner must be a gate, exiting non-zero, with its printed summary computed from the same variables.
• When the format legitimately changes, update the fixture and the expectation in one commit.

That completes Module 8 and Part II. Across Modules 5 to 8 you have gone from a one-liner that selects records to a script a team can depend on: state that survives a record, arrays that count and group without being told the keys, numbers whose denominators are stated, output in a shape and order something else can consume, and the validation, conversion, portability and testing habits that keep all of it honest.

Part III applies the whole of it to real semiconductor data. It opens with Module 9 — Simulation and UVM Log Analysis: reading the structure of a simulator log before writing a pattern for it, extracting UVM message records with their severity and id, counting warnings and errors without confusing message lines with distinct failures, grouping errors for triage so a hundred identical messages collapse into one row, and turning a directory of logs into the per-test table a verification lead opens first.