Perl Part II · Module 29 · Practical VLSI Automation Projects
Project: Maintaining a Legacy Flow Script
Most Perl an engineer touches at work was written by someone else, years ago, without tests, and other tools depend on exactly what it prints. The job is rarely to rewrite it. It is to change one thing and leave everything else as it was. This last project does that to an old regression-report script: it records everything the script does, measures why the obvious clean-ups are behaviour changes, makes a clarity edit that provably changes nothing, then predicts the exact diff of the requested feature before writing it — and checks that the change is that diff, line for line.
Advanced24 min readPerlPart IIProjectsLegacy Code
Part II · Module 29 · Chapter 29.7 · Project: Maintaining a Legacy Flow Script
1. The Brief
This project's deliverable is not a new tool. It is a change to an existing one, with the evidence that the change is safe.
| The request | Runs whose simulation died before finishing are being reported as PASS. Report them as CRASH, and count them as failures. |
| The script | legacy.pl, an old report script that other tools read. Its header says do not change the output format, the dashboard reads it!! |
| The rule | Nothing a caller can observe may change, except what the request asks for |
| Evidence required | a baseline of everything observable; an empty diff after any tidying; a written prediction of the feature's diff; proof that the actual diff equals the prediction |
The request is Chapter 21.2's finding, arriving as a bug report against code that predates it: a log with no errors and no finish marker is a run that stopped, not a run that passed.
The method assembles three earlier chapters into one change: reading an inherited script (28.1), writing down what it actually does (28.2), and refactoring a script you must not break (27.6). This project applies them to a requested behaviour change, and adds a written prediction of that change's exact diff, checked line for line.
2. The Script As Found
The script below was written for this chapter in the style of an old flow script, and it is not anyone's real code. Every habit in it is common in inherited Perl: no use strict, global variables everywhere, bareword filehandles, two-argument open, output in hash order, and a file written as a side effect.
#!/usr/bin/perl
#
# regress_report -- summarise the nightly sim logs
#
# usage: legacy.pl [-q] logdir/sim.log ...
# -q quiet, only print the totals
#
# (c) the verification team. do not change the output format,
# the dashboard reads it!!
#
$quiet = 0;
if ($ARGV[0] eq "-q") { $quiet = 1; shift; }
$total = 0; $npass = 0; $nfail = 0; $nfat = 0;
foreach $f (@ARGV) {
open(F, $f) || die "cant open $f\n";
$f =~ /([^\/]+)\/[^\/]*$/;
$t = $1; # the run name is the directory
$errs = 0; $fat = 0; $warns = 0;
while (<F>) {
if (/^UVM_(ERROR|FATAL) .*\[/) {
$errs++;
$fat++ if $1 eq "FATAL";
}
if (/^UVM_WARNING .*\[/) { $warns++; }
if (/^UVM_ERROR .*\[(\w+)\]/ && !$first{$t}) {
$first{$t} = $1;
}
}
close(F);
$total++;
if ($fat) {
$status{$t} = "FATAL"; $nfat++;
} elsif ($errs) {
$status{$t} = "FAIL"; $nfail++;
} else {
$status{$t} = "PASS"; $npass++;
}
$count{$t} = $errs;
$wcount{$t} = $warns;
}
if (!$quiet) {
print "REGRESSION REPORT\n";
print "-----------------\n";
foreach $t (keys %status) {
printf("%-14s %-6s %3d %3d %s\n", $t, $status{$t}, $count{$t},
$wcount{$t}, $first{$t});
}
}
print "total $total pass $npass fail $nfail fatal $nfat\n";
# the dashboard picks this up
open(O, ">last_run.txt");
foreach $t (keys %status) {
print O "$t $status{$t}\n" if $status{$t} ne "PASS";
}
close(O);
exit($nfail + $nfat > 0 ? 1 : 0);Read it in the order its callers experience it — arguments, the main loop, everything it emits, the exit — and write down every observable behaviour, because those are what must not change:
- Two ways to call it: with
-qit prints only the totals line. - STDOUT: a header, one line per run (name, status, error count, warning count, first error ID) and a totals line.
- A file:
last_run.txt, written into the current directory, listing every run that is notPASS. The comment says the dashboard picks it up. - The exit status: 1 if any run failed or was fatal.
- STDERR: nothing, today.
Two quirks are visible once the code is read slowly, and both are behaviour that callers may depend on. The errors column counts fatal lines too — the first pattern matches UVM_ERROR or UVM_FATAL. And the second pattern, which looks almost the same, matches errors only.
3. The Corpus
A baseline is only as good as the inputs it runs on. The corpus has to exercise every branch the change will touch and every branch it might touch by accident.
// tb_corpus.v -- the logs the legacy script reads: UVM-SHAPED lines (not
// UVM, which cannot run here), real simulator output, one run per
// +TEST/+SEED. What a run does depends only on its seed:
// seed 1, 2 pass (one warning)
// seed 3 two errors
// seed 4 an error, then a fatal
// seed 5 dies part-way: no report summary
`timescale 1ns/1ps
module tb_corpus;
integer seed, n_err, n_warn, n_fatal;
reg [8*16-1:0] test;
task summary;
begin
$display("--- UVM Report Summary ---");
$display("UVM_WARNING :%5d", n_warn);
$display("UVM_ERROR :%5d", n_err);
$display("UVM_FATAL :%5d", n_fatal);
end
endtask
initial begin
n_err = 0; n_warn = 0; n_fatal = 0;
if (!$value$plusargs("TEST=%s", test)) test = "smoke";
if (!$value$plusargs("SEED=%d", seed)) seed = 1;
$display("UVM_INFO @ 0: reporter [RNTST] Running test %0s...", test);
#10 $display("UVM_WARNING tb(0) @ %0t: env [LATE] response 2 cycles late", $time);
n_warn = n_warn + 1;
if (seed == 3) begin
#10 $display("UVM_ERROR tb(0) @ %0t: env.scbd [SCBD] data mismatch exp=%0d got=0", $time, seed * 7);
#10 $display("UVM_ERROR tb(0) @ %0t: env.scbd [SCBD] data mismatch exp=%0d got=0", $time, seed * 9);
n_err = n_err + 2;
end
if (seed == 4) begin
#10 $display("UVM_ERROR tb(0) @ %0t: env.axi [AXI] burst crossed a 4KB boundary", $time);
#10 $display("UVM_FATAL tb(0) @ %0t: env.axi [AXI] cannot continue", $time);
n_err = n_err + 1; n_fatal = n_fatal + 1;
end
if (seed == 5) begin
#10 $display("UVM_INFO tb(0) @ %0t: env [PHASE] main_phase executing", $time);
$finish; // dies: no summary
end
#10 summary;
$finish;
end
endmodule#!/bin/bash
# make_corpus.sh DIR -- simulate 4 tests x 5 seeds; DIR/<test>_s<seed>/sim.log
set -u
out=${1:?usage: make_corpus.sh DIR}
iverilog -o corpus.vvp tb/tb_corpus.v || exit 1
for test in smoke burst csr fifo; do
for seed in 1 2 3 4 5; do
mkdir -p "$out/${test}_s$seed"
vvp -n corpus.vvp +TEST=$test +SEED=$seed > "$out/${test}_s$seed/sim.log"
done
done
rm -f corpus.vvpFour tests, five seeds each, and each seed is one kind of run: two passes, two errors, an error followed by a fatal, and a run that dies before its report summary. The logs are real Icarus output, UVM-shaped as in Module 21. The seed-5 runs are the ones the request is about, so they are in the corpus; the fatal runs are what the two patterns disagree on, so they are in it too.
4. Making The Output Comparable
The method is to run the script before and after a change and compare. That only works if the script repeats itself.
#!/bin/bash
# order.sh -- run the unchanged script five times: does it repeat itself?
# (Only the verdict is printed: HOW MANY orders appear is itself random.)
bash make_corpus.sh corpus > /dev/null || exit 1
verdict() { [ "$1" -eq 1 ] && echo "one output" || echo "more than one different output"; }
raw=$(for i in 1 2 3 4 5; do perl legacy.pl corpus/*/sim.log | cksum; done | sort -u | wc -l)
srt=$(for i in 1 2 3 4 5; do perl legacy.pl corpus/*/sim.log | sort | cksum; done | sort -u | wc -l)
echo "five runs, output as printed: $(verdict $raw)"
echo "five runs, each output sorted: $(verdict $srt)"
rm -f last_run.txtfive runs, output as printed: more than one different output
five runs, each output sorted: one outputThe unchanged script, on unchanged input, does not repeat itself: it lists runs with foreach $t (keys %status), and hash order changes from one process to the next (Chapter 6.6). A naive diff against a baseline would report differences on a change that changed nothing, and an engineer who sees a noisy diff every time learns to stop reading it.
So the snapshot sorts every output before recording it. That is safe for one reason worth stating: since the order was random in production, no caller can have relied on any particular order. The snapshot then records everything section 2 listed — both ways of calling the script, the exit status, STDOUT, STDERR, and every file the script leaves in its working directory, so a change that stops writing last_run.txt, or starts writing something new, shows up.
#!/bin/bash
# snapshot.sh SCRIPT CORPUS -- everything a caller can observe of SCRIPT,
# for each way it is called: exit status, STDOUT, STDERR, and every file it
# leaves in its working directory. Lines are SORTED, because this script's
# output order changes from run to run (see order.sh).
script=$(cd "$(dirname "$1")" && pwd)/$(basename "$1")
corpus=$(cd "$2" && pwd)
for args in "" "-q"; do
work=$(mktemp -d) # the script's working directory
cap=$(mktemp -d) # what it printed
( cd "$work" && perl "$script" $args "$corpus"/*/sim.log > "$cap/stdout" 2> "$cap/stderr" )
status=$?
echo "### legacy.pl ${args:-(no options)}"
echo "exit $status"
echo "--- stdout"; sort "$cap/stdout"
echo "--- stderr"; sort "$cap/stderr"
for f in $(cd "$work" && ls -A | sort); do
echo "--- file $f"; sort "$work/$f"
done
rm -rf "$work" "$cap"
done5. The Method
Figure 29.7.1 — Two edits, each proved before the next
Two edits, never one. A clarity edit and a behaviour change made together produce one diff, and nobody can tell which lines came from which. Made separately, the first must produce no diff at all, and the second must produce exactly the predicted one.
6. Three Clean-Ups That Are Behaviour Changes
Everyone who opens a script like this wants to make the same three improvements first. Each is measured here against the baseline.
#!/bin/bash
# tidyups.sh -- the three clean-ups everyone wants to make first, each
# measured against the baseline of the script as found.
bash make_corpus.sh corpus > /dev/null || exit 1
bash snapshot.sh legacy.pl corpus > baseline.txt
mkdir -p try
echo "== A. add 'use strict;'"
perl -pe 's/^(\$quiet = 0;)/use strict;\n$1/' legacy.pl > try/strict.pl
errors=$(perl -c try/strict.pl 2>&1 | grep -c 'Global symbol')
names=$(perl -c try/strict.pl 2>&1 | grep -o 'Global symbol "[^"]*"' | sort -u | wc -l | tr -d ' ')
echo " it no longer compiles: $errors errors, naming $names different variables"
echo "== B. turn on warnings (-w on the #! line)"
perl -pe 's{^#!/usr/bin/perl$}{#!/usr/bin/perl -w}' legacy.pl > try/warn.pl
bash snapshot.sh try/warn.pl corpus > warn.txt
diff baseline.txt warn.txt | grep '^[<>]' | sed 's/"[a-z]*_s[0-9]"/"RUN"/; s# at .*/try/# at try/#' |
sort | uniq -c | sed 's/^/ /'
echo "== C. make the two message patterns one (they look the same)"
perl -pe 's{^(\s+if \(/)\^UVM_\(ERROR\|FATAL\) \.\*\\\[/\)}{$1^UVM_ERROR .*\\[(\\w+)\\]/)}' legacy.pl > try/onepat.pl
grep -n 'if (/^UVM_ERROR' try/onepat.pl | sed 's/^/ /'
bash snapshot.sh try/onepat.pl corpus > onepat.txt
diff baseline.txt onepat.txt | grep '^[<>]' > onepat.diff
echo " $(grep -c . onepat.diff) changed lines; the first ten:"
head -10 onepat.diff | sed 's/^/ /'== A. add 'use strict;'
it no longer compiles: 60 errors, naming 14 different variables
== B. turn on warnings (-w on the #! line)
12 > Use of uninitialized value $first{"RUN"} in printf at try/warn.pl line 50.
== C. make the two message patterns one (they look the same)
23: if (/^UVM_ERROR .*\[(\w+)\]/) {
28: if (/^UVM_ERROR .*\[(\w+)\]/ && !$first{$t}) {
28 changed lines; the first ten:
< burst_s4 FATAL 2 1 AXI
> burst_s4 FAIL 1 1 AXI
< csr_s4 FATAL 2 1 AXI
> csr_s4 FAIL 1 1 AXI
< fifo_s4 FATAL 2 1 AXI
> fifo_s4 FAIL 1 1 AXI
< smoke_s4 FATAL 2 1 AXI
> smoke_s4 FAIL 1 1 AXI
< total 20 pass 12 fail 4 fatal 4
> total 20 pass 12 fail 8 fatal 0A. use strict stops it compiling. Sixty errors naming fourteen variables, because every variable in the script is a global — exactly what Chapter 1.10's strict mode exists to refuse. Strictness is the right goal, and converting the script to it is a real piece of work — every variable needs a declaration at the right scope — with its own baseline and its own empty diff. It is not something to do in passing while fixing a bug.
B. Warnings change STDERR. Twelve new lines of the kind Chapter 1.11 describes, one for every run with no first error, because printf is handed an undefined $first{...}. STDOUT is unchanged — but STDERR is observable too, and flows exist that treat any STDERR output as failure, or mail it to someone. Turning warnings on is a behaviour change, and a good one to make on purpose.
C. The two patterns are not the same. Replacing the counting pattern with the first-ID pattern "because they do the same thing" changes 28 lines: every fatal run becomes FAIL, with half its errors, and the totals line reports no fatal runs at all. The two patterns differed in one alternation, (ERROR|FATAL) versus ERROR, and that difference was carrying the script's whole notion of a fatal run. A difference between two copies of almost-identical code is a question to answer, not a typo to fix.
7. The Clarity Edit
"Leave the file a little clearer" means something specific here: make the next reader faster, and change nothing they can observe. This edit writes down what the script actually does — including the quirk and the bug — marks the two patterns as different on purpose, and sorts the run lines.
--- legacy.pl
+++ legacy.pl
@@ -8,6 +8,18 @@
# (c) the verification team. do not change the output format,
# the dashboard reads it!!
#
+# WHAT IT ACTUALLY DOES -- written down 2026-10-13 from a baseline over
+# corpus/ (snapshot.sh), before anything here was changed:
+# stdout a header, one line per run (name, status, errors, warnings,
+# first error ID), then "total N pass N fail N fatal N"
+# -q only the totals line
+# file last_run.txt in the CURRENT directory: every run not PASS
+# exit 1 if any run is FAIL or FATAL, else 0
+# quirks the errors column counts UVM_FATAL lines too;
+# a log that stops before its report summary counts as PASS;
+# runs were listed in hash order, different on every run --
+# they are sorted now, which no reader could have relied on
+#
$quiet = 0;
if ($ARGV[0] eq "-q") { $quiet = 1; shift; }
@@ -20,6 +32,8 @@
$t = $1; # the run name is the directory
$errs = 0; $fat = 0; $warns = 0;
while (<F>) {
+ # NOTE the two message patterns below are NOT the same, on purpose:
+ # this one counts errors AND fatals; the first-ID one takes errors only
if (/^UVM_(ERROR|FATAL) .*\[/) {
$errs++;
$fat++ if $1 eq "FATAL";
@@ -45,7 +59,7 @@
if (!$quiet) {
print "REGRESSION REPORT\n";
print "-----------------\n";
- foreach $t (keys %status) {
+ foreach $t (sort keys %status) {
printf("%-14s %-6s %3d %3d %s\n", $t, $status{$t}, $count{$t},
$wcount{$t}, $first{$t});
}
@@ -54,7 +68,7 @@
# the dashboard picks this up
open(O, ">last_run.txt");
-foreach $t (keys %status) {
+foreach $t (sort keys %status) {
print O "$t $status{$t}\n" if $status{$t} ne "PASS";
}
close(O);The sort is the only line that touches behaviour, and section 4 argued why it is safe: the old order was random, so no caller relied on it. Now the order is the same on every run, and the next person who maintains this script can compare outputs without sorting them first.
The comment block is the most valuable part of the edit. It records the side-effect file, both ways of calling the script, the exit rule and the quirks as observed, with the date and the corpus they were observed on — so they are facts with a source, not folklore.
8. Predict The Diff, Then Make The Change
Before touching the code, write down the exact diff the request should produce. Working it out forces every consequence into the open: the four seed-5 run lines change status; the totals move four runs from pass to fail; last_run.txt gains four lines — in both calling modes, because -q writes the file too, which is easy to forget until the prediction has to list it.
# prediction.txt -- every line expected to leave (-) or join (+) the
# snapshot when crash.patch is applied, written BEFORE applying it.
# Each line ends in $ so that a trailing space can be seen.
-burst_s5 PASS 0 1 $
-csr_s5 PASS 0 1 $
-fifo_s5 PASS 0 1 $
-smoke_s5 PASS 0 1 $
-total 20 pass 12 fail 4 fatal 4$
-total 20 pass 12 fail 4 fatal 4$
+burst_s5 CRASH 0 1 $
+csr_s5 CRASH 0 1 $
+fifo_s5 CRASH 0 1 $
+smoke_s5 CRASH 0 1 $
+total 20 pass 8 fail 8 fatal 4$
+total 20 pass 8 fail 8 fatal 4$
+burst_s5 CRASH$
+csr_s5 CRASH$
+fifo_s5 CRASH$
+smoke_s5 CRASH$
+burst_s5 CRASH$
+csr_s5 CRASH$
+fifo_s5 CRASH$
+smoke_s5 CRASH$Every line ends in $, as cat -e shows line ends. My first prediction did not, and it failed: typed by hand, it had lost the trailing space that printf leaves after the empty first-ID field. A trailing space is invisible in an editor and real in a byte comparison, so the comparison makes it visible instead of ignoring it.
The change itself is small. It reads the summary marker, decides CRASH before anything else, counts it under fail so that the totals line keeps its exact shape, and updates the comment block from section 7, because the behaviour it documents has changed.
--- legacy.pl
+++ legacy.pl
@@ -14,9 +14,10 @@
# first error ID), then "total N pass N fail N fatal N"
# -q only the totals line
# file last_run.txt in the CURRENT directory: every run not PASS
-# exit 1 if any run is FAIL or FATAL, else 0
+# exit 1 if any run is FAIL, FATAL or CRASH, else 0
# quirks the errors column counts UVM_FATAL lines too;
-# a log that stops before its report summary counts as PASS;
+# a log that stops before its report summary is CRASH, and
+# is counted under "fail" (until 2026-10-13 it counted as PASS);
# runs were listed in hash order, different on every run --
# they are sorted now, which no reader could have relied on
#
@@ -30,7 +31,7 @@
open(F, $f) || die "cant open $f\n";
$f =~ /([^\/]+)\/[^\/]*$/;
$t = $1; # the run name is the directory
- $errs = 0; $fat = 0; $warns = 0;
+ $errs = 0; $fat = 0; $warns = 0; $summary = 0;
while (<F>) {
# NOTE the two message patterns below are NOT the same, on purpose:
# this one counts errors AND fatals; the first-ID one takes errors only
@@ -39,13 +40,16 @@
$fat++ if $1 eq "FATAL";
}
if (/^UVM_WARNING .*\[/) { $warns++; }
+ $summary = 1 if /^--- UVM Report Summary ---/;
if (/^UVM_ERROR .*\[(\w+)\]/ && !$first{$t}) {
$first{$t} = $1;
}
}
close(F);
$total++;
- if ($fat) {
+ if (!$summary) { # it never finished: not a pass
+ $status{$t} = "CRASH"; $nfail++; # counted with the failures
+ } elsif ($fat) {
$status{$t} = "FATAL"; $nfat++;
} elsif ($errs) {
$status{$t} = "FAIL"; $nfail++;One part of the request is a format change that no snapshot can approve: CRASH is a new value in the status column and in last_run.txt. The totals line keeps its shape on purpose, but whoever owns the dashboard that reads last_run.txt must confirm it accepts a status it has never seen. A maintainer names the consumers that could not be checked rather than assuming they are fine.
check.sh compares the lines that changed between two snapshots with the prediction, and maintain.sh runs the whole method:
#!/bin/bash
# check.sh BEFORE AFTER PREDICTION -- do the lines that changed between two
# snapshots match, exactly, the lines predicted before the edit was made?
# Every line gets a $ at its end, as `cat -e` shows it, so that a trailing
# space is part of the comparison AND visible in any message.
tmp=$(mktemp -d)
diff "$1" "$2" | grep '^[<>]' | sed 's/^< /-/; s/^> /+/; s/$/$/' | sort > "$tmp/actual"
grep -v '^#' "$3" | sort > "$tmp/wanted"
if cmp -s "$tmp/actual" "$tmp/wanted"; then
echo " the change is exactly the prediction: $(grep -c . "$tmp/wanted") line(s)"
status=0
else
echo " the change is NOT the prediction:"
diff "$tmp/wanted" "$tmp/actual" | grep '^[<>]' |
sed 's/^< / predicted, did not happen: /; s/^> / happened, not predicted: /'
status=1
fi
rm -rf "$tmp"
exit $status#!/bin/bash
# maintain.sh -- baseline, the clarity edit, then the one feature, each
# step proved against the snapshot before the next one starts.
bash make_corpus.sh corpus > /dev/null || exit 1
mkdir -p work && cp legacy.pl work/legacy.pl
echo "== 1. baseline of the script as found"
bash snapshot.sh work/legacy.pl corpus > baseline.txt
grep -c . baseline.txt | sed 's/^/ lines recorded: /'
echo "== 2. the clarity edit"
patch -s work/legacy.pl < clarity.patch || exit 1
bash snapshot.sh work/legacy.pl corpus > after_clarity.txt
if diff baseline.txt after_clarity.txt > /dev/null; then
echo " no observable change"
else
echo " CHANGED:"; diff baseline.txt after_clarity.txt | sed 's/^/ /'; exit 1
fi
echo "== 3. the feature: a run that never finished is CRASH, not PASS"
patch -s work/legacy.pl < crash.patch || exit 1
bash snapshot.sh work/legacy.pl corpus > after_crash.txt
bash check.sh after_clarity.txt after_crash.txt prediction.txt== 1. baseline of the script as found
lines recorded: 50
== 2. the clarity edit
no observable change
== 3. the feature: a run that never finished is CRASH, not PASS
the change is exactly the prediction: 20 line(s)The baseline, an empty diff after the clarity edit, and a feature diff that is exactly the twenty predicted lines. That last line is a stronger statement than "the tests pass". It says the change did what was intended, everything it did was intended, and every caller-visible effect was listed in advance.
9. Proving The Check Can Fail
mutate.sh makes the same change five wrong ways, each a plausible slip, and asks the check about each.
#!/bin/bash
# mutate.sh -- five WRONG ways to make the change. Each is crash.patch plus
# one more edit; check.sh must refuse every one whose effect differs from
# the prediction.
bash make_corpus.sh corpus > /dev/null || exit 1
mkdir -p m
cp legacy.pl m/clear.pl && patch -s m/clear.pl < clarity.patch
bash snapshot.sh m/clear.pl corpus > m/before.txt
wrong() { # LABEL PERL-SUBSTITUTION
cp m/clear.pl m/try.pl && patch -s m/try.pl < crash.patch
perl -0pi -e "$2" m/try.pl
if ! perl -c m/try.pl > /dev/null 2>&1; then printf '%-46s INVALID\n' "$1"; return; fi
bash snapshot.sh m/try.pl corpus > m/after.txt
if bash check.sh m/before.txt m/after.txt prediction.txt > /dev/null; then
printf '%-46s ACCEPTED by the check\n' "$1"
else
printf '%-46s REFUSED (%s line(s) differ from the prediction)\n' "$1" \
"$(bash check.sh m/before.txt m/after.txt prediction.txt | grep -c 'predicted\|happened')"
fi
}
wrong "1 crash = no \$finish line" 's/\$summary = 1 if \/\^--- UVM Report Summary ---\//\$summary = 1 if \/\\\$finish called\//'
wrong "2 a new 'crash N' field in the totals" 's/(\$status\{\$t\} = "CRASH"; )\$nfail\+\+;/$1\$ncrash++;/; s/fatal \$nfat\\n"/fatal \$nfat crash \$ncrash\\n"/'
wrong "3 CRASH counted under nothing" 's/(\$status\{\$t\} = "CRASH"; )\$nfail\+\+;/$1/'
wrong "4 the pattern clean-up slipped in too" 's{/\^UVM_\(ERROR\|FATAL\) \.\*\\\[/}{/^UVM_ERROR .*\\[(\\w+)\\]/}'
wrong "5 crash tested after fatal" 's/ if \(!\$summary\) \{(.*?)\n \} elsif \(\$fat\) \{(.*?)\n \} elsif/ if (\$fat) {$2\n } elsif (!\$summary) {$1\n } elsif/s'1 crash = no $finish line REFUSED (20 line(s) differ from the prediction)
2 a new 'crash N' field in the totals REFUSED (4 line(s) differ from the prediction)
3 CRASH counted under nothing REFUSED (4 line(s) differ from the prediction)
4 the pattern clean-up slipped in too REFUSED (28 line(s) differ from the prediction)
5 crash tested after fatal ACCEPTED by the checkFour are refused, each for a reason worth knowing:
- "No
$finishline" changes nothing at all. Icarus prints its own$finish calledline even for a run that died, so that test never fires. All twenty predicted lines fail to happen. Chapter 21.2 explains why the summary block, not the simulator's finish line, is the evidence. - A new
crash Nfield changes the totals line's shape — exactly what the header warned against. - Counting a crash under nothing leaves the totals summing to less than the total.
- Slipping the pattern clean-up in brings section 6's 28 lines with it, and the check lists them.
The fifth is accepted, and it should be. Testing for a crash after testing for a fatal gives a different answer only for a run that logs a fatal and then dies. The corpus has no such run, so on this corpus the two orders are indistinguishable, and the check cannot prefer one. A characterisation check proves that the change is right on the corpus, and the corpus decides what can be proved. The fix belongs in the corpus — exercise 1.
10. What This Method Does Not Do
- It cannot see what the corpus does not exercise. Section 9's accepted mutant is the proof. Adding the run you are worried about to the corpus is part of the change.
- It cannot vouch for output order once the snapshot sorts. The sort was argued safe here, from the fact that the old order was random; in a script with a deterministic order, the snapshot must keep it.
- It cannot approve a new value that a downstream reader might reject. That needs the reader's owner.
- It is not a test suite. The snapshot and the prediction prove one change. Keeping the corpus, the snapshot script and a current baseline beside the script turns them into a regression check for the next change, which is the cheapest test suite a legacy script will ever get.
- It does not make the script modern.
use strict, warnings and real tests are each worth doing — as separate, deliberate changes, each with its own baseline and its own empty or predicted diff.
11. Exercises
1. Close the gap. Add a sixth seed that logs a fatal and then dies before its summary. Update the prediction, rerun mutate.sh, and show that mutant 5 is now refused. Which status should that run get, and why?
2. Strict, safely. Convert legacy.pl to use strict with my declarations. Your evidence is an empty diff against the post-clarity baseline. Which variable needs the most thought about its scope?
3. Warnings, on purpose. Turn warnings on and fix the twelve uninitialized warnings without changing STDOUT. Write the prediction first: what exactly should the diff of STDERR be?
4. A third caller. Suppose a cron job runs the script from / instead of from a run directory. What does last_run.txt do then, and how would you extend snapshot.sh to notice?
5. The dashboard's side. Write the ten-line reader that "the dashboard" might be — reading last_run.txt — so that it accepts CRASH. Then write the one that would reject it. What should the review comment on the crash patch say?
12. What To Remember
- The deliverable is the change plus its evidence: a baseline, an empty diff for any tidying, a written prediction, and a diff that equals it.
- List every observable behaviour first — every way the script is called, STDOUT, STDERR, exit status and every file it writes.
- Make the output comparable. Measured: five runs of the unchanged script did not agree, and sorted output did.
- The obvious clean-ups are behaviour changes. Measured:
use strictgave 60 compile errors,-wgave 12 new STDERR lines, and unifying two near-identical patterns changed 28 lines. - Two edits, never one: a clarity edit with an empty diff, then the behaviour change.
- Predict the diff before writing the code. Measured: the change was exactly the 20 predicted lines. Writing the prediction is where the forgotten consequences show up —
-qwrites the file too. - Make line ends visible. My own prediction lost a trailing space.
- The corpus decides what can be proved. One wrong implementation was indistinguishable on it, and the check rightly accepted it.
- Name the consumers you could not check.
That closes Module 29, and with it the set of projects this track builds from its Part I and Part II material. Each tool had a brief before it had code, an exit status a flow can trust, a refusal for every input it could not handle completely, an independent witness for its results — a compiler, a simulator, a generator's own truth table, a re-count — and a test suite that was shown to fail. And the last project made the point the other six depend on: most of the work is on code that already exists, and the professional skill is changing it without surprising anyone who relies on it.
Where this fits
Part of the Perl curriculum.