Byte-for-Byte Side-by-Side Validation

Part II · Module 28 · Chapter 28.8 · Byte-for-Byte Side-by-Side Validation

1. The Gate

A migration has one moment where it can go wrong for everybody at once: the switch. Up to that moment, the Python implementation is a file in a repository. After it, every consumer of the script gets the new behaviour. Side-by-side validation is the gate in front of that moment. It does not pass until the new implementation has done exactly what the old one does, on every input anyone thought to try.

Why "byte for byte", and not "the same numbers"? Because every consumer reads bytes. Chapter 28.2 found three consumers of the case-study script, and none of them reads numbers. The Makefile reads one byte, the exit status. post_wns.sh reads a CSV field and treats an empty one as clean. The weekly digest counts lines starting with exactly four spaces. A comparison that parses both outputs and compares the values would pass 0 against an empty field, -0.043 against -0.0430, and a two-space indent against a four-space one. Each of those has already broken a consumer in an earlier chapter.

2. The One-Line Version, and What It Misses

The registry entry for this chapter gives the minimal form of the idea, and it is a good place to start:

for f in corpus/*.rpt; do
  diff <(perl old.pl "$f"; echo "exit=$?") \
       <(python3 new.py "$f"; echo "exit=$?") || echo "DIFF $f"
done

It compares STDOUT and the exit status, and for a pure filter (one file in, text out, nothing else) that can be enough. For a script like sta_summary.pl it misses four things that the rest of this chapter adds:

1. STDERR is not compared. It is not even captured: it goes to the terminal and scrolls away. Chapter 28.2's B8 and B11 live there.
2. Files are not compared. tmp.csv is where the CSV defects of Chapter 28.5 showed up, and where the dashboard reads.
3. Both runs share one working directory. The second run overwrites the first one's tmp.csv, and neither run starts from a known state.
4. The inputs are only real files. The corpus contains what the flow happened to produce, which is mostly normal runs. Section 7 measures what that costs.

3. The Faithful Port

The port under test is written to reproduce all twelve behaviours of Chapter 28.2's inventory. That includes the ones nobody would choose, each marked with its inventory row. Fixing them comes later, one decision at a time, and only after this gate has passed.

#!/usr/bin/env python3
"""sta_summary.py - a FAITHFUL port of sta_summary.pl (Chapter 28.1).

It reproduces the twelve behaviours of the Chapter 28.2 inventory, defects
included, so that it can be validated byte for byte against the original.
Fixes come AFTER validation, one at a time, as recorded decisions.
"""
import os
import re
import sys

ENDPOINT = re.compile(r"^Endpoint\s*:\s*(\S+)")
SLACK = re.compile(r"slack \((\w+)\)\s+(\S+)")
NUMBER = re.compile(r"\s*([+-]?(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?)")


def perl_true(v):
    """Perl's truth rule: undef, '', '0' and 0 are false."""
    return v is not None and v != "" and v != "0" and v != 0


def perl_num(v):
    """Numify like Perl: undef is 0, text uses its leading number, else 0."""
    if v is None:
        return 0.0
    m = NUMBER.match(v)
    return float(m.group(1)) if m else 0.0


def get_options(argv):
    """The parts of Getopt::Long's behaviour this script relies on (B11):
    options anywhere, names case-insensitive, '--' ends them, a lone '-'
    is an argument, and an unknown option is reported and then ignored."""
    verbose, rest, i = False, [], 0
    while i < len(argv):
        arg = argv[i]
        i += 1
        if arg == "--":
            rest.extend(argv[i:])
            break
        if arg.startswith("-") and arg != "-":
            name, eq, _ = arg.lstrip("-").partition("=")
            if name.lower() == "v":
                if eq:
                    sys.stderr.write("Option v does not take an argument\n")
                else:
                    verbose = True
            else:
                sys.stderr.write(f"Unknown option: {name.lower()}\n")
            continue
        rest.append(arg)
    return verbose, rest


def main(argv):
    verbose, args = get_options(argv)
    d = args[0] if args else None
    if not perl_true(d):                     # B9: "0" becomes "."
        d = "."
    try:
        names = os.listdir(d)
    except OSError as e:                     # B8: STDERR, exit = errno
        sys.stderr.write(f"cannot open {d}: {os.strerror(e.errno)}\n")
        return e.errno
    files = [n for n in names if n.endswith(".rpt")]
    if not files:                            # B7: STDOUT, exit 0, tmp.csv untouched
        print(f"No reports found in {d}")
        return 0

    paths, viol, tns, wns, eps = {}, {}, {}, {}, {}
    ep = None                                # global in the original: never reset
    for f in files:
        corner = f[:-len(".rpt")]
        try:
            fh = open(os.path.join(d, f))
        except OSError:                      # B10: skipped silently
            continue
        with fh:
            for line in fh:
                m = ENDPOINT.search(line)
                if m:
                    ep = m.group(1)
                m = SLACK.search(line)
                if m:
                    status, text = m.group(1), m.group(2)
                    paths[corner] = paths.get(corner, 0) + 1
                    if status == "VIOLATED":
                        viol[corner] = viol.get(corner, 0) + 1
                        tns[corner] = tns.get(corner, 0.0) + perl_num(text)
                        eps.setdefault(corner, []).append(ep)
                    if perl_num(text) < perl_num(wns.get(corner)):   # B4
                        wns[corner] = text                           # B6: the text

    try:
        csv = open("tmp.csv", "w")           # B5: the CURRENT directory
    except OSError:
        csv = None                           # the original never checked
    bad = 0
    for c in sorted(paths):                  # B1: string sort of corner names
        print("%-16s paths=%-3d viol=%-3d wns=%.3f tns=%.3f"
              % (c, paths[c], viol.get(c, 0), perl_num(wns.get(c)), tns.get(c, 0.0)))
        if csv:
            v = viol.get(c)
            csv.write(f"{c},{'' if v is None else v},{wns.get(c, '')}\n")   # B6: undef is empty
        if verbose:                          # B2
            for e in eps.get(c, []):
                print(f"    {'' if e is None else e}")
        if perl_true(viol.get(c)):
            bad += 1
    if csv:
        csv.close()
    return bad                               # B12: a count; 256 wraps to 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

The original it must match is sta_summary.pl from Chapter 28.1, unchanged:

#!/usr/bin/perl
# sta_summary.pl - summarise per-corner timing reports
# usage: sta_summary.pl [-v] [report_dir]
#   prints one line per corner: paths, violations, WNS and TNS
#
# (A SAMPLE legacy script written for this module, in the style of
#  scripts that have run unchanged in a flow for a decade.)

use Getopt::Long;

$verbose = 0;
GetOptions("v" => \$verbose);

$dir = shift @ARGV;
$dir = "." unless $dir;

opendir(D, $dir) || die "cannot open $dir: $!\n";
@files = grep(/\.rpt$/, readdir(D));
closedir(D);

if (!@files) {
    print "No reports found in $dir\n";
    exit 0;
}

foreach $f (@files) {
    ($corner = $f) =~ s/\.rpt$//;
    open(F, "$dir/$f") || next;
    while (<F>) {
        if (/^Endpoint\s*:\s*(\S+)/) {
            $ep = $1;
        }
        if (/slack \((\w+)\)\s+(\S+)/) {
            $paths{$corner}++;
            if ($1 eq "VIOLATED") {
                $viol{$corner}++;
                $tns{$corner} += $2;
                push(@{$eps{$corner}}, $ep);
            }
            $wns{$corner} = $2 if $2 < $wns{$corner};
        }
    }
    close(F);
}

open(CSV, ">tmp.csv");
$bad = 0;
foreach $c (sort keys %paths) {
    printf("%-16s paths=%-3d viol=%-3d wns=%.3f tns=%.3f\n",
           $c, $paths{$c}, $viol{$c}, $wns{$c}, $tns{$c});
    print CSV "$c,$viol{$c},$wns{$c}\n";
    if ($verbose) {
        foreach $e (@{$eps{$c}}) { print "    $e\n"; }
    }
    $bad++ if $viol{$c};
}
close(CSV);

exit $bad;

Most of it applies what the last three chapters measured:

• perl_true for $dir = "." unless $dir, so that "0" becomes "." (Chapter 28.6's truth rule, B9).
• perl_num for every place the Perl code used a captured string as a number, and the captured text itself wherever the Perl code printed it (B4, B6, and Chapter 28.5's -0.0430).
• viol.get(c) written as an empty field when absent (B6), and ep kept across files exactly like the original's global.
• os.strerror(e.errno) and return e.errno for the missing directory, reproducing both Perl's $! text and die's exit status (B8, Chapter 15.4).
• return bad as a count. Python's sys.exit(256) also leaves an exit status of 0, the same one-byte wrap as Perl (B12), measured separately.

The least obvious part is get_options. The original calls GetOptions("v" => \$verbose) and ignores the result (B11), so the port must reproduce what Getopt::Long does, not what argparse does. Measured with a small test script, Getopt::Long with this one specification:

• accepts -v, --v and -V (option names are case-insensitive by default);
• accepts options after the directory (night_0911 -v);
• stops at --, and treats a lone - as an argument;
• reports -X as Unknown option: x (lower case), and -v=1 as Option v does not take an argument, and carries on.

get_options implements exactly those measured rules, and its docstring says so. Anything else Getopt::Long can do (bundling, abbreviations of longer names, + options) is not reproduced, because this script's specification cannot reach it. That limit belongs in the inventory as "not probed".

4. The Corpus

The corpus is the set of inputs both implementations will see. It has two halves, and both are necessary:

• Real runs. Here, three nightly run directories built from the sample reports: a normal night, a night with a four-decimal slack and a dead corner, and a night where every corner met timing. In a real migration, copy the last few weeks of run directories into the corpus as they are.
• The inventory's edges. One case per behaviour that real runs rarely produce: -v in three spellings and positions, an unknown option, an option given a value, an empty directory, a missing one, a directory named 0, an unreadable report, and 256 failing corners.

Report : timing (synthetic teaching format)
Design : chip_top
Corner : ss_0p72v_125c

Startpoint: u_core/r_reg[3]
Endpoint  : u_core/acc_reg[7]
  slack (VIOLATED)   -0.043

Startpoint: u_core/r_reg[1]
Endpoint  : u_core/acc_reg[2]
  slack (VIOLATED)   -0.011

Startpoint: u_core/cnt_reg[0]
Endpoint  : u_core/cnt_reg[5]
  slack (MET)         0.120

End of report

Report : timing (synthetic teaching format)
Design : chip_top
Corner : tt_0p80v_25c

Startpoint: u_core/r_reg[3]
Endpoint  : u_core/acc_reg[7]
  slack (MET)         0.087

Startpoint: u_core/cnt_reg[0]
Endpoint  : u_core/cnt_reg[5]
  slack (MET)         0.240

End of report

#!/bin/bash
# make_corpus.sh DIR - the validation corpus: three "nightly" run directories
# built from the sample reports, plus the edge cases from the inventory.
set -e
c=$1
mkdir -p "$c"/{night_0911,night_0912,night_0913,empty,0,unreadable,many}
cp reports/*.rpt "$c/night_0911/"
cp reports/*.rpt "$c/night_0912/"
printf 'Report : timing\n\nStartpoint: u_io/a_reg\nEndpoint  : u_io/q_reg\n  slack (VIOLATED)   -0.0430\n\nEnd of report\n' \
    > "$c/night_0912/ss_0p72v_m40c.rpt"
printf 'Report : timing\nError: license unavailable\n' > "$c/night_0912/ff_0p88v_m40c.rpt"
cp reports/tt_0p80v_25c.rpt "$c/night_0913/"            # a night where every corner met
cp reports/*.rpt "$c/0/"
cp reports/*.rpt "$c/unreadable/"      # made unreadable in each COPY, not here
for i in $(seq -w 1 256); do cp reports/ss_0p72v_125c.rpt "$c/many/c$i.rpt"; done

# name          arguments          (every case runs in its own copy of the corpus)
night_0911      night_0911
night_0912      night_0912
night_0913      night_0913
verbose         -v night_0911
opt_after_dir   night_0911 -v
upper_V         -V night_0911
unknown_opt     -x night_0911
opt_with_value  -v=1 night_0911
empty_dir       empty
missing_dir     nosuch
dir_named_0     0
unreadable      unreadable
256_corners     many

5. The Harness

sidebyside.sh runs every case in the list against both implementations and compares the four channels:

#!/bin/bash
# sidebyside.sh OLD NEW - run both implementations over every case in
# $CASES (default cases.list) and compare all four channels byte for byte.
# Each implementation runs in its OWN copy of the corpus, seeded with a stale
# tmp.csv, so a missing write or a write to the wrong place shows up.
old=$1 new=$2 here=$(pwd) fail=0
corpus=$(mktemp -d)/corpus
bash make_corpus.sh "$corpus"

transcript() {      # transcript IMPL ARGS... : the four channels, as text
    local impl=$1; shift
    local w; w=$(mktemp -d)
    cp -R "$corpus"/. "$w"/
    chmod 0 "$w/unreadable/tt_0p80v_25c.rpt"   # after the copy: cp cannot copy it
    echo "old,9,-9.999" > "$w/tmp.csv"
    (cd "$w" && $impl "$@" > out 2> err; echo $? > status)
    echo "--- stdout";  cat "$w/out"
    echo "--- stderr";  cat "$w/err"
    echo "--- exit";    cat "$w/status"
    echo "--- tmp.csv"; cat "$w/tmp.csv"
    chmod -R u+rw "$w"
}

while read -r name args; do
    case "$name" in ''|'#'*) continue ;; esac
    transcript "$old" $args > "$corpus.old"
    transcript "$new" $args > "$corpus.new"
    if cmp -s "$corpus.old" "$corpus.new"; then
        printf '  same  %s\n' "$name"
    else
        printf '  DIFF  %s\n' "$name"
        diff -u --label old --label new "$corpus.old" "$corpus.new" | sed -n '3,$p' | grep '^[-+]' | sed 's/^/          /'
        fail=1
    fi
done < "${CASES:-cases.list}"
chmod -R u+rw "$corpus"
exit $fail

Three of its choices are the difference between a harness and the one-liner.

A private copy of the corpus for every run. transcript copies the whole corpus into a fresh directory and runs the implementation there. Nothing one run writes can be seen by the other, and every run starts from the same state.

A stale tmp.csv in every copy. Each copy starts with old,9,-9.999 in tmp.csv. If an implementation writes the file, the transcript shows the new contents. If it does not write it, the transcript shows the stale line. Either way, a difference in whether the file is written becomes a visible difference. Without the seed, "did not write" and "wrote nothing" would look the same.

One transcript per run, in a fixed layout. STDOUT, STDERR, the exit status and the file are written one after another under fixed headings, and the two transcripts are compared with cmp. For the report, diff -u --label old --label new shows only the changed lines. The labels keep temporary paths and timestamps out of the output, so the harness's own output is reproducible too.

6. The First Port

Ports are rarely right the first time. This is how the first attempt at sta_summary.py differed from the final one. make_port1.py recreates it, and its four edits are the four mistakes:

"""make_port1.py - recreate the FIRST port attempt from the final one.

The first attempt differed from sta_summary.py in exactly four places,
each a plausible translation choice. Each edit is checked to apply.
"""
EDITS = [
    # 1. "0" is a valid directory name, surely
    ('''    d = args[0] if args else None
    if not perl_true(d):                     # B9: "0" becomes "."
        d = "."''',
     '''    d = args[0] if args else "."'''),
    # 2. Python's own exception text, and a conventional status of 1
    ('''        sys.stderr.write(f"cannot open {d}: {os.strerror(e.errno)}\\n")
        return e.errno''',
     '''        sys.stderr.write(f"cannot open {d}: {e}\\n")
        return 1'''),
    # 3. a count of zero written as 0
    ('''            csv.write(f"{c},{'' if v is None else v},{wns.get(c, '')}\\n")   # B6: undef is empty''',
     '''            csv.write(f"{c},{viol.get(c, 0)},{wns.get(c, '')}\\n")'''),
    # 4. option names compared exactly
    ('''            if name.lower() == "v":''',
     '''            if name == "v":'''),
]
src = open("sta_summary.py").read()
for old, new in EDITS:
    assert src.count(old) == 1, "edit does not apply: " + old.splitlines()[0]
    src = src.replace(old, new)
open("sta_summary_port1.py", "w").write(src)
print("wrote sta_summary_port1.py with", len(EDITS), "edits")

Each edit is a choice a competent translator might make. A directory named 0 is a perfectly good name. Python's own exception text is more informative than strerror. A count of zero is 0. And option names are compared exactly, because who types -V?

The validation run compares both ports with the original, first over a corpus of nightly runs only, then over the full list:

# only what the nightly flow happened to produce
night_0911      night_0911
night_0912      night_0912
night_0913      night_0913

#!/bin/bash
# The two ports against the original, over the nightly-only corpus and the
# full corpus. Exit status of each comparison is printed, not acted on.
p=$(pwd)
python3 make_port1.py
for port in sta_summary_port1.py sta_summary.py; do
    for list in cases_nightly.list cases.list; do
        echo "=== $port over $list"
        CASES=$list bash sidebyside.sh "perl $p/sta_summary.pl" "python3 $p/$port"
        echo "    harness exit: $?"
    done
done

wrote sta_summary_port1.py with 4 edits
=== sta_summary_port1.py over cases_nightly.list
  DIFF  night_0911
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  DIFF  night_0912
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  DIFF  night_0913
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
    harness exit: 1
=== sta_summary_port1.py over cases.list
  DIFF  night_0911
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  DIFF  night_0912
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  DIFF  night_0913
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  DIFF  verbose
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  DIFF  opt_after_dir
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  DIFF  upper_V
          -    u_core/acc_reg[7]
          -    u_core/acc_reg[2]
          +Unknown option: v
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  DIFF  unknown_opt
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  DIFF  opt_with_value
          -tt_0p80v_25c,,
          +tt_0p80v_25c,0,
  same  empty_dir
  DIFF  missing_dir
          -cannot open nosuch: No such file or directory
          +cannot open nosuch: [Errno 2] No such file or directory: 'nosuch'
          -2
          +1
  DIFF  dir_named_0
          -No reports found in .
          +ss_0p72v_125c    paths=3   viol=2   wns=-0.043 tns=-0.054
          +tt_0p80v_25c     paths=2   viol=0   wns=0.000 tns=0.000
          -0
          +1
          -old,9,-9.999
          +ss_0p72v_125c,2,-0.043
          +tt_0p80v_25c,0,
  same  unreadable
  same  256_corners
    harness exit: 1
=== sta_summary.py over cases_nightly.list
  same  night_0911
  same  night_0912
  same  night_0913
    harness exit: 0
=== sta_summary.py over cases.list
  same  night_0911
  same  night_0912
  same  night_0913
  same  verbose
  same  opt_after_dir
  same  upper_V
  same  unknown_opt
  same  opt_with_value
  same  empty_dir
  same  missing_dir
  same  dir_named_0
  same  unreadable
  same  256_corners
    harness exit: 0

Over the full corpus, the first port differs in 10 of 13 cases, and the diff lines name every mistake:

• The CSV row tt_0p80v_25c,0, instead of tt_0p80v_25c,, appears in every case that produces a clean corner. That is mistake 3, B6 again.
• upper_V shows -V treated as an unknown option: the endpoint lines disappear from STDOUT and Unknown option: v appears on STDERR. That is mistake 4.
• missing_dir shows Python's exception text in STDERR and an exit status of 1 instead of 2. That is mistake 2. A Makefile that treats status 2 as "tool problem" and 1 as "timing failed" would now misreport a missing run directory as a timing failure.
• dir_named_0 shows the port reading the directory 0 (and finding violations, exit 1) where the original read . and exited 0. That is mistake 1.

7. The Corpus Decides What Can Be Found

Now look at the first block of the output: the same first port over only the nightly runs. All three cases differ, so the gate fails, which looks reassuring. But every one of those diffs is the same CSV row. Only one of the four mistakes is visible. The other three live in a missing directory, an unusual option spelling and a directory named 0, and the nightly runs never exercise any of them.

Follow that forward one step. A developer sees three identical diffs, fixes the CSV line, and re-runs the nightly corpus. Every case is now same. The gate passes, the switch happens, and three behaviour changes ship with it. The first missing run directory is reported to the Makefile as a timing failure, with an exit status nobody has seen before.

That is the most important measurement in this chapter, and it generalises: a validation corpus of real inputs tests the behaviours real inputs exercise. Real inputs are mostly normal. Every edge in Chapter 28.2's inventory needs a case of its own, because nothing else will produce it. The inventory and the corpus are the same list, written twice.

8. The Corrected Port

The last two blocks of the output are sta_summary.py, the port shown in section 3, over both corpora: same in 3 of 3 and in 13 of 13. Every channel of every case is byte-identical to the original: the summary, the endpoint list in all three option spellings, the warning text for the unknown option and for the option with a value, the errno message and status, the untouched tmp.csv after an empty run, the . for a directory named 0, the silently skipped unreadable report, and an exit status of 0 for 256 failing corners.

This is the state Chapter 28.10 starts from. The gate has passed, and because the port reproduces the defects too, the switch itself changes nothing for any consumer. The defects can then be fixed in Python, one at a time. Each fix will show up as a difference in this same harness, on exactly the cases it is meant to change, which is how a fix is shown to be the only thing that changed.

9. The Harness Had a False Green

The first version of sidebyside.sh reported the unreadable case as same for both ports, and it was not testing what its name says. make_corpus.sh used to make the report unreadable in the corpus, with chmod 0. Then cp -R, copying the corpus into each private directory, could not read that file. It printed Permission denied and carried on, so every copy contained no tt_0p80v_25c.rpt at all. The case was testing "a corner with no report", not "a report the script cannot read".

For this script the verdict happens to be the same, because the original skips an unreadable report silently, which looks exactly like a missing one. For a script that reported unreadable files, the case would have passed while never exercising that code. The fix, now in the harness, makes the file unreadable after the copy, in each private directory.

The lesson is the one every chapter of this module has repeated about tools: a check whose setup can fail quietly must be checked itself. The giveaway here was on STDERR of the harness, and it is also why the harness's own output is worth reading on every run, not just its exit status.

10. What "Same" Does Not Prove

Thirteen identical cases are thirteen points. Be explicit about what the gate does not establish:

• Inputs not in the corpus. A new report format, a corner name with a space, a 2 GB report. Add a case when you think of one.
• Timing and resources. The port could be much slower on a large report and still pass every case. Measure separately if it matters.
• The environment. Both ran under one Perl, one Python and one filesystem. Another machine can change strerror texts, readdir order (which ep's global behaviour depends on), or the encoding of non-ASCII names.
• Concurrency. Each case ran alone in its own directory. Two runs sharing a directory are not tested.
• Behaviour the original never had. The gate proves the port does what the Perl did. Whether that was right is the inventory's question, not this one.

Write these limits into the migration's review record. Chapter 28.9 asks for them by name.

11. Mistakes

Comparing parsed values instead of bytes. Consumers read bytes, and every difference this module found would have survived a value comparison.

Leaving out STDERR, the exit status or the files. Three of the four mistakes above were only visible in those channels.

Running both implementations in one directory. The second overwrites the first's outputs, and neither starts from a known state.

A corpus of real runs only. Measured: it exposed 1 of 4 mistakes.

Fixing until the diffs disappear. Each diff is either a port bug or a recorded decision. Making a diff disappear by changing the corpus, or by filtering the comparison, is neither.

Not reading the harness's own output. Permission denied from cp was the only sign that a case was not testing what it claimed.

12. Exercises

1. A fourth channel nobody listed. The original creates tmp.csv with whatever permissions the user's umask allows. Does the port? Add the file's mode to the transcript and run the gate. If it differs, decide whether that is a port bug or a behaviour nobody depends on, and record the decision.

2. Break the port, one channel at a time. Make four one-line changes to sta_summary.py, each affecting only one channel (STDOUT spacing, STDERR text, the exit status, the CSV). Confirm the harness catches each one, and note which cases catch it. Is any single case enough to catch all four?

3. Validate a fix. Fix B4 in the port (report a clean corner's real worst slack). Run the gate and list exactly which cases change and how. Compare the list with Chapter 28.2's fix_wns mutant: did the harness find the same scenarios?

4. Real corpus coverage. For a script you are migrating, collect two weeks of real inputs. For each inventory row, mark whether any real input exercises it. Write constructed cases for the rest, and report the fraction of the inventory that real inputs alone covered.

5. Order as a channel. Change sta_summary.py to print corners in insertion order instead of sorted order. Run the gate several times. Then do the same to the Perl original (remove sort) and run it under PERL_HASH_SEED=0 PERL_PERTURB_KEYS=0. What does a byte-for-byte gate need to do when the original's order was never defined (Chapter 28.6)?

13. What To Remember

• Validation is a gate in front of the switch: both implementations, the same inputs, every output compared.
• Byte for byte, because consumers read bytes: an empty field, a trailing zero and an indent have each broken a consumer in this module.
• Four channels: STDOUT, STDERR, exit status, and every file written. The one-liner compares two.
• Give each implementation a private copy of the inputs, and seed every file it might write, so "not written" is visible.
• The port reproduces defects too. Fix them after the gate passes, one at a time, each visible in the same harness.
• Reproduce the library, not just the code: Getopt::Long accepts -V and options after arguments, and reports -X as x.
• Measured: the first port differed in 10 of 13 cases; its four mistakes were in the CSV, the option case, the errno status and the 0 directory.
• Measured: real nightly runs alone exposed 1 of the 4. The inventory's edge cases exposed the rest.
• The inventory and the corpus are the same list. Every edge needs a constructed case.
• Check the harness: a chmod in the wrong place made a case test the wrong thing, and only the harness's own STDERR showed it.
• "Same" covers the corpus, not timing, environment, concurrency or inputs nobody tried. Write those limits down.

Chapter 28.9 turns the module into a review: the questions a reviewer asks of a migration before approving the switch, each backed by an artefact from Chapters 28.1 to 28.8.