Engineering Review Checklist

Part IV · Module 14 · Chapter 14.6 · Engineering Review Checklist

1. Why This Matters

Chapter 14.5 ended with a linter that found six real defects in a five-line program and two limits it could not pass: it reads text rather than meaning, and reformatting defeats it. This chapter is what covers the rest.

A review needs an order, not a list, for the same reason Chapter 14.1's diagnostic sequence did: the questions are not independent. If a script selects the wrong records, every question about its arithmetic is wasted — the arithmetic may be flawless and the answer is still wrong. So the order runs from the question that invalidates the most to the question that invalidates the least, and each stage is one command.

Three scripts are reviewed against it below. All three run without error, produce well-formatted output, and are wrong.

2. The Review Order

Each stage is one question, with a test that answers it in a single command. The stages are not weighted equally: stage 2 has found more real defects across this track than the other five together.

3. The Three Specimens, As Submitted

cat > timing.rpt <<'EOF'
  slack (VIOLATED)   -0.018
  slack (MET)         0.067
  slack (VIOLATED)   -0.112
  slack (MET)         0.238
  slack (VIOLATED)   -0.041
  slack (MET)         0.005
EOF
cat > cov.rpt <<'EOF'
u_alu   75.00%   120  160
u_ctrl  40.00%    20   50
u_io   100.00%   310  310
EOF
mkdir -p logs empty
cat > logs/dma.log <<'EOF'
UVM_INFO ./tb/top.sv(44) @ 100: starting test
UVM_ERROR ./env/sb.sv(148) @ 5200: MISMATCH exp=32'hDEAD got=32'hBEEF
UVM_ERROR ./env/sb.sv(148) @ 5600: MISMATCH exp=32'hCAFE got=32'h0000
UVM_INFO ./tb/top.sv(91) @ 9000: test done
--- UVM Report Summary ---
UVM_ERROR :    2
UVM_WARNING :    0
EOF
cat > logs/cdc.log <<'EOF'
UVM_INFO ./tb/top.sv(44) @ 100: starting test
--- UVM Report Summary ---
UVM_ERROR :    0
UVM_WARNING :    1
EOF
printf 'dma\ncdc\nio_loopback\n' > tests.txt
printf "  %s records, %s coverage rows, %s logs, %s tests\n" \
  "$(awk 'END { print NR }' timing.rpt)" "$(awk 'END { print NR }' cov.rpt)" \
  "$(ls logs | wc -l | tr -d ' ')" "$(awk 'END { print NR }' tests.txt)"

  6 records, 3 coverage rows, 2 logs, 3 tests

Three specimens, submitted as working code:

/slack/ {
    v = $3
    if (v < "-0.050") { n++; printf "  VIOLATION %s\n", v }
}
END { printf "  %d path(s) below -0.050\n", n+0 }

{ sum += $2; n++
  if ($2 < 90) printf "  BELOW GATE: %s at %s\n", $1, $2 }
END { printf "  overall coverage: %.2f%%\n", sum / n }

#!/bin/sh
total=$(cat logs/*.log | awk '/^UVM_ERROR/ { n++ } END { print n+0 }')
echo "  total UVM_ERROR: $total"
if [ "$total" -gt 0 ]; then echo "  REGRESSION FAILED"; else echo "  REGRESSION CLEAN"; fi

All three are short, readable, and do approximately the right thing. Read them now and note what you would comment on; the stages below find six defects between them.

4. Stage 1 — What Input Does It Assume?

None of the three says. That is the finding, and it takes no command to make it.

subA.awk assumes slack lives in $3 of any line containing slack. subB.awk assumes four fields with a percentage in $2 and counts in $3 and $4. subC.sh assumes every .log in logs/ is a complete UVM log.

An assumption that is not written down is not checked, and Chapter 14.5 section 9 measured what writing it down costs: one comparison on record 1. Stage 1 is the cheapest stage because the answer is visible without running anything, and it sets up every later stage — you cannot ask "which field" without first asking "which field of what".

5. Stage 2 — Which Records Does It Select?

awk -f subA.awk timing.rpt
printf "  --- every slack value in the report ---\n"
awk '/slack/ { printf "  NR=%d v=%s\n", NR, $3 }' timing.rpt
printf "  --- numeric comparison ---\n"
awk '/slack/ { if ($3 + 0 < -0.050) { n++; printf "  VIOLATION %.3f\n", $3 } } END { printf "  %d path(s)\n", n+0 }' timing.rpt

  VIOLATION -0.018
  VIOLATION -0.041
  2 path(s) below -0.050
  --- every slack value in the report ---
  NR=1 v=-0.018
  NR=2 v=0.067
  NR=3 v=-0.112
  NR=4 v=0.238
  NR=5 v=-0.041
  NR=6 v=0.005
  --- numeric comparison ---
  VIOLATION -0.112
  1 path(s)

The same stage, applied to subC.sh:

sh subC.sh
printf "  --- the logs' own summary lines ---\n"
awk '/^UVM_ERROR :/ { s += $3 } END { printf "  summed from summaries: %d\n", s }' logs/*.log
printf "  --- corrected pattern, per log ---\n"
awk '$1 == "UVM_ERROR" && $2 ~ /\(/ { c[FILENAME]++ }
     END { n = split("logs/cdc.log logs/dma.log", o, " ")
            for (i = 1; i <= n; i++) printf "  %-14s %d\n", o[i], c[o[i]] + 0 }' logs/*.log

  total UVM_ERROR: 4
  REGRESSION FAILED
  --- the logs' own summary lines ---
  summed from summaries: 2
  --- corrected pattern, per log ---
  logs/cdc.log   0
  logs/dma.log   2

Four against two — exactly double. /^UVM_ERROR/ matches the two real error lines and the two UVM_ERROR : summary lines that UVM writes at the end of every log, including cdc.log's UVM_ERROR : 0.

Chapter 9.3 measured this pattern and the remedy: require something that only a real severity line has. $1 == "UVM_ERROR" && $2 ~ /\(/ requires the first field to be exactly the severity and the second to contain a source location, which the summary line's : does not.

The factor of two is a coincidence of this fixture — one summary line per log, two logs, two real errors. On a regression with forty passing tests and two failures, the same defect inflates two errors to forty-two. The error is proportional to the number of tests, not to the number of failures, so it gets worse as the regression gets healthier.

6. Stage 3 — Which Field, and Is It Numeric?

awk -f subB.awk cov.rpt
printf "  --- is \$2 numeric? ---\n"
awk '{ printf "  %-8s [%s] +0=%-8s numeric?=%s\n", $1, $2, $2 + 0, ($2 + 0 == $2 ? "yes" : "no") }' cov.rpt

  BELOW GATE: u_alu at 75.00%
  BELOW GATE: u_ctrl at 40.00%
  BELOW GATE: u_io at 100.00%
  overall coverage: 71.67%
  --- is $2 numeric? ---
  u_alu    [75.00%] +0=75       numeric?=no
  u_ctrl   [40.00%] +0=40       numeric?=no
  u_io     [100.00%] +0=100      numeric?=no

7. Stage 4 — What Is the Denominator?

The same run holds the second subB.awk defect, and it is in the last line rather than the gate.

awk '{ v = $2; sub(/%$/, "", v); v += 0; cov += $3; tot += $4
       if (v < 90) printf "  BELOW GATE: %s at %.2f%%\n", $1, v }
     END { printf "  aggregate: %.2f%% (%d of %d)\n", 100 * cov / tot, cov, tot }' cov.rpt

  BELOW GATE: u_alu at 75.00%
  BELOW GATE: u_ctrl at 40.00%
  aggregate: 86.54% (450 of 520)

71.67% against 86.54% — a gap of 14.87 points on identical data.

subB.awk computes sum / n, the mean of three percentages: (75 + 40 + 100) / 3 = 71.67. The corrected version computes the aggregate: 450 covered items of 520 coverable, 86.54%. Chapter 13.3 measured the same divergence (68.22% against 72.31%) and Chapter 11.8 measured the case where the two moved in opposite directions across two runs.

The mean is not arithmetically wrong — it is the answer to a different question, and the difference is driven entirely by u_ctrl, which is the weakest block and has the smallest population (50 of 520). An unweighted mean gives it the same vote as u_io's 310.

So the stage 4 question is not "is the arithmetic correct" but "what population is this a figure over, and is it the population the reader will assume?" The corrected line answers it by printing (450 of 520) beside the percentage — Chapter 13.5's rule that a stored figure needs its numerator and denominator, applied to a figure that is merely displayed.

subC.sh has a denominator problem too, and it is more serious because the denominator does not exist at all:

awk 'NR == FNR { nt++; next }
     $1 == "UVM_ERROR" && $2 ~ /\(/ { f = FILENAME; sub(/.*\//, "", f); sub(/\.log$/, "", f); e[f]++ }
     END { printf "  %d test(s) in tests.txt\n", nt
            while ((getline t < "tests.txt") > 0) {
              r = (t in e) ? "errors=" e[t] : "?"
              if (!(t in e)) r = (system("test -r logs/" t ".log") == 0 ? "clean" : "NO LOG")
              printf "  %-12s %s\n", t, r } }' tests.txt logs/*.log

  3 test(s) in tests.txt
  dma          errors=2
  cdc          clean
  io_loopback  NO LOG

Three tests were supposed to run and there are two logs. io_loopback produced no log at all — it never started, or it died before writing one — and subC.sh cannot see that, because it enumerates logs and a test with no log contributes no records.

Chapter 9.5 established the rule this illustrates, and Chapter 12.6 built on it: the row count comes from the test list, not from the logs. A script that counts what it found can never report what is missing, and a regression where a third of the suite failed to launch is exactly the case where the reported error count is least meaningful.

Note the attribution mechanism, which subC.sh also destroys:

cat logs/*.log | awk '$1 == "UVM_ERROR" && $2 ~ /\(/ { c[FILENAME]++ }
     END { for (k in c) printf "  FILENAME=[%s] -> %d\n", k, c[k] }'

  FILENAME=[] -> 2

FILENAME is empty. Piping through cat gives awk one anonymous stream, so every record loses the file it came from and all attribution collapses into a single empty key. Passing the files to awk directly — awk … logs/*.log — is what made the per-log breakdown in stage 2 possible, and it costs nothing.

8. Stage 5 — Does It Refuse?

cat > subC2.sh <<'SH'
#!/bin/sh
total=$(cat empty/*.log 2>/dev/null | awk '/^UVM_ERROR/ { n++ } END { print n+0 }')
echo "  total UVM_ERROR: $total"
if [ "$total" -gt 0 ]; then echo "  REGRESSION FAILED"; else echo "  REGRESSION CLEAN"; fi
SH
sh subC2.sh > o.txt 2>&1; printf "  rc=%s\n" "$?"; sed 's/^/  /' o.txt

  rc=0
    total UVM_ERROR: 0
    REGRESSION CLEAN

An empty directory reports REGRESSION CLEAN and exits 0. A regression that produced nothing at all is indistinguishable, to any caller, from a regression that passed completely.

This is the single most consequential defect in the three specimens, because it fails in the direction of good news. Chapter 12.6 measured a related case where a nightly reduction reported 60% passing against a true 1 of 6 attempted, and the remedy there is the remedy here — grade the state:

#!/bin/sh
# fixedC.sh — subC.sh with stages 2, 4 and 5 addressed.
#
# Exit:  0 clean, 1 errors found, 2 nothing can be concluded.

d=${1:-logs}
n=$(ls "$d"/*.log 2>/dev/null | wc -l | tr -d ' ')
[ "$n" -gt 0 ] || { echo "  NO LOGS in $d -- nothing can be concluded" >&2; exit 2; }

# the files go to awk directly, so FILENAME survives (section 7)
# and the pattern requires a source location, not just the severity word
e=$(awk '$1 == "UVM_ERROR" && $2 ~ /\(/ { n++ } END { print n+0 }' "$d"/*.log)
printf "  %d log(s), %d error(s) -> %s\n" "$n" "$e" "$([ "$e" -gt 0 ] && echo FAILED || echo CLEAN)"
[ "$e" -eq 0 ]

sh fixedC.sh logs > o.txt 2>&1; printf "  logs  rc=%s\n" "$?"; sed 's/^/  /' o.txt
sh fixedC.sh empty > o.txt 2>&1; printf "  empty rc=%s\n" "$?"; sed 's/^/  /' o.txt

  logs  rc=1
    2 log(s), 2 error(s) -> FAILED
  empty rc=2
    NO LOGS in empty -- nothing can be concluded

rc=1 for a regression with errors, rc=2 for one that cannot be assessed, and the log count printed beside the error count so a reader can see the denominator. Note the status is captured through a redirect rather than a pipe, for the reason Chapter 12.1 section 7 measured.

9. Stage 6 — Can the Next Engineer Change It?

for f in subA.awk subB.awk; do
  printf "  %-10s comments=%s  named-consts=%s  contract-check=%s\n" "$f" \
    "$(grep -c '^ *#' "$f")" \
    "$(awk '/^BEGIN *\{/ && /=/ { n++ } END { print n+0 }' "$f")" \
    "$(awk '/NF *[!=]= *[0-9]/ { n++ } END { print n+0 }' "$f")"
done

  subA.awk   comments=0  named-consts=0  contract-check=0
  subB.awk   comments=0  named-consts=0  contract-check=0

Zero across the board. This stage is last because it is the only one whose findings do not make the current output wrong — subA.awk would be equally wrong with perfect comments — and because Chapter 14.5 section 8 measured the limit of what it buys: a named field constant did not prevent a collapsed grouping, it only made the fix a one-line edit.

So the stage 6 question is narrow and answerable: when the report format changes, will this script fail loudly or quietly? Stage 1's contract check is what decides that, which is why stages 1 and 6 are the same question asked at opposite ends of the review.

10. Why the Order Is Fixed

The six stages are ordered by how much a wrong answer invalidates:

Reading that table bottom-up is the usual review order — style first, because it is visible without running anything — and it is the order that found none of the six defects in these specimens. Every one of them required running the script against a fixture and comparing records, fields, denominators or statuses.

That is the argument for the order, and for the specimens. Chapter 14.5's linter can ask stage 6 and parts of stage 3. Stages 2, 4 and 5 need a fixture, an expected answer worked out independently, and a comparison of sets rather than totals.

11. Common Mistakes

Reviewing style before correctness. Stage 6 found zero defects that affect any current output; stages 2 through 5 found six.

Accepting a plausible count. subA.awk's 2 and the true 1 are both plausible; the sets are disjoint.

Comparing totals instead of records. The only check that finds a disjoint selection is printing NR.

Assuming a numeric-looking field is numeric. 100.00% failed a < 90 gate with no quoted constant in sight.

Trusting a mean of percentages. 71.67% against an aggregate 86.54% on three rows.

Enumerating outputs instead of inputs. subC.sh cannot see io_loopback, which produced no log.

Piping through cat. FILENAME becomes empty and all per-file attribution collapses.

A severity pattern with no shape requirement. /^UVM_ERROR/ also matched the summary line, doubling the count — and the inflation grows with the number of passing tests.

Exiting 0 when nothing ran. An empty log directory reported REGRESSION CLEAN.

Reasoning about a defect instead of running it. The exit status I expected subC.sh to lose, it does not lose.

12. Exercises

Exercise 1 — Stage 2 on the slack extractor

awk -f subA.awk timing.rpt | sed 's/^/  as submitted:/'
awk '/slack/ { if ($3 + 0 < -0.050) printf "  corrected: VIOLATION NR=%d %.3f\n", NR, $3 }' timing.rpt

  as submitted:  VIOLATION -0.018
  as submitted:  VIOLATION -0.041
  as submitted:  2 path(s) below -0.050
  corrected: VIOLATION NR=3 -0.112

Exercise 2 — Stage 3 on the coverage gate

awk '{ v = $2; sub(/%$/, "", v); v += 0
       printf "  %-8s raw-compare=%-6s numeric-compare=%s\n", $1, ($2 < 90 ? "BELOW" : "ok"), (v < 90 ? "BELOW" : "ok") }' cov.rpt

  u_alu    raw-compare=BELOW  numeric-compare=BELOW
  u_ctrl   raw-compare=BELOW  numeric-compare=BELOW
  u_io     raw-compare=BELOW  numeric-compare=ok

The two columns disagree on exactly one row, and it is the row at 100.00%.

Note that the conversion is three statements — v = $2, sub(), v += 0 — and not one expression. awk has no comma operator, so ( v = $2, sub(...), v + 0 ) is a syntax error rather than a sequence; the first draft of this exercise was written that way and did not run at all.

Exercise 3 — Debug the doubled error count

awk '/^UVM_ERROR/ { printf "  matched FNR=%d in %s: [%s]\n", FNR, FILENAME, $0 }' logs/*.log

  matched FNR=3 in logs/cdc.log: [UVM_ERROR :    0]
  matched FNR=2 in logs/dma.log: [UVM_ERROR ./env/sb.sv(148) @ 5200: MISMATCH exp=32'hDEAD got=32'hBEEF]
  matched FNR=3 in logs/dma.log: [UVM_ERROR ./env/sb.sv(148) @ 5600: MISMATCH exp=32'hCAFE got=32'h0000]
  matched FNR=6 in logs/dma.log: [UVM_ERROR :    2]

Printing the matched records answers it in one command. Two are severity lines with a source path; two are summary lines, and the first line of the output is cdc.log's UVM_ERROR : 0 — a log reporting zero errors contributed one to the count.

That is the detail worth keeping: the inflation is not "the summary line is counted once per real error". It is once per log, including logs with nothing wrong. FNR rather than NR is what makes the per-file positions readable, and the shell expanded the glob in sorted order, so cdc.log is processed first.

Exercise 4 — Stage 4: build the denominator

awk 'NR == FNR { t[$1]; nt++; next } { f = FILENAME; sub(/.*\//, "", f); sub(/\.log$/, "", f); seen[f] }
     END { printf "  %d test(s) listed, %d produced a log\n", nt, length(seen)
            while ((getline x < "tests.txt") > 0) if (!(x in seen)) printf "  NO LOG: %s\n", x }' tests.txt logs/*.log

  3 test(s) listed, 2 produced a log
  NO LOG: io_loopback

Note length(seen) here, which Chapter 14.5 section 10 flags as a non-POSIX extension — acceptable in a one-off review command run on a known awk, and not acceptable in a committed script.

Exercise 5 — Engineering application

#!/bin/sh
# review.sh — run the six-stage review against a script and a fixture.
#
# Usage:  sh review.sh <script> <fixture> <expected-record-list>
#
# This does not judge the script. It prints what the script selected beside
# what it should have selected, so stage 2 is answered by comparison rather
# than by reading. Stages 1, 4 and 6 are reported as questions, because they
# cannot be decided without knowing what the figure is for.

prog=$1
fix=$2
want=$3

[ -r "$prog" ] || { echo "UNREADABLE: $prog" >&2; exit 2; }
[ -r "$fix" ]  || { echo "UNREADABLE: $fix"  >&2; exit 2; }

echo "STAGE 1 -- stated input contract"
if grep -q 'NF *[!=]= *[0-9]' "$prog"; then
    echo "  a field-count check is present"
else
    echo "  NONE -- the script does not state what it expects (14.5 s9)"
fi

echo "STAGE 2 -- records selected"
got=$(sh -c "awk -f '$prog' '$fix'" 2>/dev/null | awk 'END { print NR }')
printf "  the script produced %s output line(s)\n" "$got"
printf "  records it should have selected: %s\n" "$want"
echo "  (compare the records, never the counts -- 14.3 s3)"

echo "STAGE 3 -- numeric fields"
awk '{ for (i = 1; i <= NF; i++)
         if ($i ~ /[0-9]/ && $i + 0 != $i) { k[i] = k[i] " " $i; bad[i] = 1 } }
     END { n = 0
           for (i = 1; i <= 12; i++) if (i in bad) { n++
               printf "  field %d is NOT numeric:%s\n", i, k[i] }
           if (n == 0) print "  every digit-bearing field converts cleanly" }' "$fix"

echo "STAGE 4 -- denominator"
if grep -q '/ *n *}' "$prog" || grep -q 'sum */ *n' "$prog"; then
    echo "  divides by a RECORD COUNT -- is a mean the intended figure? (13.3)"
else
    echo "  no mean-of-rows division found"
fi

echo "STAGE 5 -- refusal"
if grep -q 'exit 2' "$prog"; then echo "  exits 2 somewhere"; else echo "  NO exit 2 -- it cannot say 'cannot conclude' (12.6)"; fi

echo "STAGE 6 -- changeability"
printf "  comments=%s named-field-consts=%s\n" \
    "$(grep -c '^ *#' "$prog")" \
    "$(awk '/^BEGIN *\{/ && /=/ { n++ } END { print n+0 }' "$prog")"

sh review.sh subA.awk timing.rpt "NR=3 only" > r.txt 2>&1; printf "  subA rc=%s\n" "$?"; sed 's/^/  /' r.txt

  subA rc=0
  STAGE 1 -- stated input contract
    NONE -- the script does not state what it expects (14.5 s9)
  STAGE 2 -- records selected
    the script produced 3 output line(s)
    records it should have selected: NR=3 only
    (compare the records, never the counts -- 14.3 s3)
  STAGE 3 -- numeric fields
    every digit-bearing field converts cleanly
  STAGE 4 -- denominator
    no mean-of-rows division found
  STAGE 5 -- refusal
    NO exit 2 -- it cannot say 'cannot conclude' (12.6)
  STAGE 6 -- changeability
    comments=0 named-field-consts=0

Read stage 3 carefully: every digit-bearing field converts cleanly, and that is correct. timing.rpt's non-numeric field is (VIOLATED), which contains no digit, so the check does not consider it — and there genuinely is no field in that file that looks numeric and is not.

So stage 3 passes and subA.awk is still wrong. Its defect is a string constant in the program — $3 < "-0.050" — not a property of the data, and no amount of inspecting the fixture can reveal it. Only stage 2's comparison against an independently derived expected set does, and the tool deliberately prints the expected set beside the result rather than deciding.

On the other specimen the mechanisable stages do fire:

sh review.sh subB.awk cov.rpt "none -- every row is reported" > r2.txt 2>&1; printf "  subB rc=%s\n" "$?"; sed 's/^/  /' r2.txt

  subB rc=0
  STAGE 1 -- stated input contract
    NONE -- the script does not state what it expects (14.5 s9)
  STAGE 2 -- records selected
    the script produced 4 output line(s)
    records it should have selected: none -- every row is reported
    (compare the records, never the counts -- 14.3 s3)
  STAGE 3 -- numeric fields
    field 2 is NOT numeric: 75.00% 40.00% 100.00%
  STAGE 4 -- denominator
    divides by a RECORD COUNT -- is a mean the intended figure? (13.3)
  STAGE 5 -- refusal
    NO exit 2 -- it cannot say 'cannot conclude' (12.6)
  STAGE 6 -- changeability
    comments=0 named-field-consts=0

Stage 3 names the field and its values, and stage 4 names the mean — both of subB.awk's real defects, found mechanically. The contrast with the subA run is the finding of the whole exercise and Exercise 6 quantifies it.

Exercise 6 — Challenge: measure what the mechanised stages can and cannot find

Six defects were established in sections 5 through 8. Determine which of them review.sh can find, and what that implies about the stages a tool can own.

printf "  stage 1 subA: "; grep -q 'NF *[!=]= *[0-9]' subA.awk && echo "contract present" || echo "NO contract"
printf "  stage 2 subA: "; awk -f subA.awk timing.rpt | awk 'END { printf "%d output line(s)\n", NR }'
printf "  stage 2 truth: "; awk '/slack/ { if ($3 + 0 < -0.050) { s = s sep "NR=" NR; sep = " " } } END { printf "[%s]\n", s }' timing.rpt
printf "  stage 3 timing.rpt: "; awk '{ for (i = 1; i <= NF; i++) if ($i ~ /[0-9]/ && $i + 0 != $i) b[i] }
     END { n = 0; for (i in b) n++; printf "%d non-numeric digit-bearing field(s)\n", n }' timing.rpt
printf "  stage 3 cov.rpt:    "; awk '{ for (i = 1; i <= NF; i++) if ($i ~ /[0-9]/ && $i + 0 != $i) b[i] }
     END { n = 0; for (i in b) n++; printf "%d non-numeric digit-bearing field(s)\n", n }' cov.rpt
printf "  stage 5 subA: "; grep -q 'exit 2' subA.awk && echo "exits 2" || echo "NO exit 2"

  stage 1 subA: NO contract
  stage 2 subA: 3 output line(s)
  stage 2 truth: [NR=3]
  stage 3 timing.rpt: 0 non-numeric digit-bearing field(s)
  stage 3 cov.rpt:    1 non-numeric digit-bearing field(s)
  stage 5 subA: NO exit 2

Six lines, each unambiguous, and stage 3 reporting 0 for timing.rpt and 1 for cov.rpt — the check is working in both cases, and in one of them the answer is "nothing here", which is true.

Now the tally. Of the six defects:

Three of six. And the split is not arbitrary: the three it found are properties of the text — a field that fails a conversion, a division by n, an absent exit 2. The three it missed each require something the tool does not have: the set of records that should have been selected, the count that should have been produced, and the knowledge that a list of expected tests exists somewhere else.

That is the precise boundary Chapter 14.5 predicted from the other side, and it has a practical consequence for how to spend review time. Stages 3, 4 and 5 can largely be automated; stage 2 cannot, and stage 2 is where the two worst defects in these specimens live — the one that reported two passing paths as violations, and the one that doubled an error count in a way that gets worse as the regression gets healthier.

So the review that matters is the one where the reviewer brings an independently derived answer to the fixture. Everything else is a linter's job.

A final honest limit, visible in the output above: review.sh fails stage 4 of its own checklist. Its stage 2 line prints 3 output line(s) — a count, with no statement of the population it came from and no comparison performed. It prints the expected set beside it and leaves the comparison to a human, which is the correct design for a tool that cannot know the answer, but it means the one number the tool emits is exactly the kind of number this chapter spent six sections warning about.

13. Summary

• A checklist is an order, not a list. Six stages, ordered by how much a wrong answer invalidates everything after it.
• Stage 1 — what input does it assume? All three specimens stated nothing, and an unstated assumption is an unchecked one.
• Stage 2 — which records? subA.awk reported -0.018 and -0.041 as below -0.050; both pass, and the only real violation -0.112 was missing. Counts 2 and 1, disjoint sets.
• Print NR. It is the only check that finds a disjoint selection, and it is one command.
• /^UVM_ERROR/ gave 4 against a true 2, matching UVM's summary line once per log — including a log reporting zero errors. The inflation scales with the number of passing tests.
• Stage 3 — is the field numeric? A block at 100.00% failed a < 90 gate, with no quoted constant anywhere; the gate fails worst on the best data.
• Stage 4 — what is the denominator? A mean of three percentages gave 71.67% against an aggregate 86.54%, driven by the weakest block having the smallest population.
• Enumerate inputs, not outputs. Three tests were listed and two produced logs; io_loopback is invisible to any script that counts what it found.
• cat file… | awk empties FILENAME, collapsing all per-file attribution into one key. Pass the files to awk.
• Stage 5 — does it refuse? An empty log directory reported REGRESSION CLEAN and exited 0. The graded version gives rc=1 for errors and rc=2 for nothing to conclude.
• Stage 6 — can it be changed? Last, because its findings do not make today's output wrong — and it buys repairability, not robustness.
• Run the defect; do not reason it. The exit status I expected subC.sh to lose, it keeps. The real defect was that it is produced and ignored.
• A tool found three of the six defects, and the three it missed each needed something it does not have: the expected record set, the expected count, or the existence of a test list. Stages 3, 4 and 5 automate; stage 2 does not, and the two worst defects live there.
• A review tool gets reviewed too. review.sh emits a bare 3 output line(s) with no population — it fails stage 4 of its own checklist.

Chapter 14.7 — Interview Problems: Beginner starts the three-chapter close of the track: the problems that are actually asked, with the answer that is expected and the follow-up that separates a memorised one-liner from understanding.