Writing Down What the Script Actually Does

Part II · Module 28 · Chapter 28.2 · Writing Down What the Script Actually Does

1. Reading Predicts; Probing Observes

Chapter 28.1 ended with a reading record: fifteen statements about sta_summary.pl, each tied to a line number, and one of them wrong until a run corrected it. A reading record is a good start, but it is a set of predictions. Its quality is the quality of the reading.

A migration needs something stronger: a list of observations, each produced by a command anyone can re-run. If the inventory says "an empty directory exits 0 and leaves the previous tmp.csv in place", there must be a probe that sets up an empty directory and an old tmp.csv, runs the script, and shows exactly that. Then the statement is no longer an opinion about the code. It is a fact about the program, and it can be checked again against any replacement.

This chapter's rule is the one in its canonical purpose: a migration has to reproduce what the script does, not what it was meant to do. That includes behaviour nobody would choose. A clean corner reporting wns=0.000 is almost certainly a bug. But somebody's dashboard may have been built on it, and a replacement that silently "fixes" it changes that dashboard on the night of the cutover, with nobody warned. So the inventory records it as it is. Whether to change it is a separate decision, made later, by the people who own the outputs.

2. The Four Channels a Caller Can Observe

Everything a calling program, a Makefile or a person can learn from a run of the script arrives through one of four channels. An inventory records all four for every scenario, because a replacement can differ in any of them.

Some things are not channels in this sense, and an inventory should say that it did not record them, not pretend they do not exist: how long the run takes, what happens when two runs share a working directory, and whether the output changes with the Perl version or the locale. Section 9 comes back to these.

3. Choosing the Scenarios

A scenario is one controlled input: a set of files in a fresh directory and one command line. Good scenarios do not come from imagination. They come from the reading record, one for each place where the code makes a choice. Go through Chapter 28.1's record and the script and ask, at every branch, default and early exit, what input makes this line run?

Two of those deserve a word, because they are the ones a person would not think of without reading the code.

S9 is about representation, not value. -0.0430 and -0.043 are the same number. The script stores the captured text and prints it to tmp.csv unchanged, but formats it with %.3f for STDOUT. A replacement that parses the number and prints it back can produce -0.043 in the CSV, which is a byte-level difference in a file another program reads.

S11 is about the size of a byte. An exit status is one byte, so it wraps at 256 (Chapter 15.4). A script that exits with a count exits 0 when the count is 256. Nobody has a block with 256 failing corners on a normal day, but a broken constraint file can produce one. And the probe costs one loop.

4. The Probe Harness

probe.pl runs one scenario at a time. Each scenario gets its own fresh scratch directory, so no run can see another's leftovers, and nothing real can be overwritten. For each run it records the four channels.

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Find;
use File::Copy qw(copy);
use File::Spec;
use Digest::MD5;

# probe.pl - record what a script DOES, one scenario at a time.
# usage: perl probe.pl SCRIPT      (a .py script is run with python3)
# Every scenario runs in its own fresh scratch directory. For each one we
# record the four channels a caller can observe: STDOUT, STDERR, the exit
# status, and every file created, changed or deleted in that directory.
my $script = File::Spec->rel2abs(shift // die "usage: probe.pl SCRIPT\n");
my @run = $script =~ /\.py\z/ ? ('python3', $script) : ($^X, $script);
my $fixtures = File::Spec->rel2abs('reports');     # the two sample reports

sub put {                                           # write a file, making dirs
    my ($path, $text) = @_;
    my ($vol, $dirs) = File::Spec->splitpath($path);
    mkdir $dirs unless -d $dirs;
    open my $fh, '>', $path or die "cannot write $path: $!\n";
    print {$fh} $text;
    close $fh;
}
sub sample_reports {                                # copy both samples into $to
    my ($to) = @_;
    mkdir $to;
    copy("$fixtures/$_", "$to/$_") or die "copy $_: $!\n"
        for qw(ss_0p72v_125c.rpt tt_0p80v_25c.rpt);
}
sub one_path {                                      # a minimal one-path report
    my ($slack, $status) = @_;
    return "Report : timing (synthetic teaching format)\n\n"
         . "Startpoint: u_core/a_reg\nEndpoint  : u_core/b_reg\n"
         . "  slack ($status)   $slack\n\nEnd of report\n";
}

# name, setup(sub receiving the scratch dir), arguments
my @scenarios = (
  [ 'S1  two reports',             sub { sample_reports("$_[0]/reports") }, 'reports' ],
  [ 'S2  two reports, -v',         sub { sample_reports("$_[0]/reports") }, '-v', 'reports' ],
  [ 'S3  no argument at all',      sub { sample_reports($_[0]) } ],
  [ 'S4  empty dir, old tmp.csv',  sub { mkdir "$_[0]/empty"; put("$_[0]/tmp.csv", "old,9,-9.999\n") }, 'empty' ],
  [ 'S5  missing dir',             sub { }, 'nosuch' ],
  [ 'S6  dir named 0',             sub { sample_reports("$_[0]/0") }, '0' ],
  [ 'S7  unreadable report',       sub { sample_reports("$_[0]/r"); chmod 0, "$_[0]/r/tt_0p80v_25c.rpt" }, 'r' ],
  [ 'S8  report with no slack',    sub { sample_reports("$_[0]/r");
                                         put("$_[0]/r/ff_0p88v_m40c.rpt", "Report : timing\nError: license unavailable\n") }, 'r' ],
  [ 'S9  slack with 4 decimals',   sub { put("$_[0]/r/ss_x.rpt", one_path('-0.0430', 'VIOLATED')) }, 'r' ],
  [ 'S10 unknown option -x',       sub { sample_reports("$_[0]/reports") }, '-x', 'reports' ],
  [ 'S11 256 failing corners',     sub { put(sprintf("%s/r/c%03d.rpt", $_[0], $_), one_path('-0.010', 'VIOLATED'))
                                             for 1 .. 256 }, 'r' ],
);

sub snapshot {                                      # relative path => md5
    my ($root) = @_;
    my %s;
    find({ no_chdir => 1, wanted => sub {
        return unless -f $_;
        my $rel = File::Spec->abs2rel($_, $root);
        if (open my $fh, '<', $_) { binmode $fh; $s{$rel} = Digest::MD5->new->addfile($fh)->hexdigest }
        else                      { $s{$rel} = 'unreadable' }
    } }, $root);
    return \%s;
}
sub slurp { open my $fh, '<', $_[0] or return ''; local $/; return scalar <$fh> }
sub show {                                          # print a channel, briefly
    my ($label, $text) = @_;
    my @l = split /\n/, $text;
    if (!@l) { printf "  %-7s: (nothing)\n", $label; return }
    my @shown = @l > 4 ? (@l[0 .. 1], sprintf "... %d lines in all", scalar @l) : @l;
    printf "  %-7s: %s\n", $label, shift @shown;
    printf "  %-7s  %s\n", '', $_ for @shown;
}

for my $sc (@scenarios) {
    my ($name, $setup, @argv) = @$sc;
    my $dir = tempdir(CLEANUP => 1);
    my $cap = tempdir(CLEANUP => 1);                # captures live OUTSIDE $dir
    $setup->($dir);
    my $before = snapshot($dir);

    my $pid = fork() // die "fork: $!\n";
    if ($pid == 0) {
        open STDOUT, '>', "$cap/out" or die;
        open STDERR, '>', "$cap/err" or die;
        chdir $dir or die "chdir: $!\n";
        exec @run, @argv or die "exec: $!\n";
    }
    waitpid $pid, 0;
    my ($code, $sig) = ($? >> 8, $? & 127);
    my $after = snapshot($dir);

    my @files;
    for my $f (sort keys %{{ %$before, %$after }}) {
        my ($was, $now) = ($before->{$f}, $after->{$f});
        my $what = !defined $was ? 'created'
                 : !defined $now ? 'deleted'
                 : $was ne $now  ? 'changed'
                 :                 '';
        next unless $what;
        my @body = $what eq 'deleted' ? () : split /\n/, slurp("$dir/$f");
        push @files, @body > 3 ? sprintf("%s %s: %d lines", $what, $f, scalar @body)
                               : ("$what $f:", map { "    $_" } @body);
    }
    push @files, "tmp.csv untouched" if exists $before->{'tmp.csv'} && !grep /tmp\.csv/, @files;

    print "$name    argv: ", (@argv ? "@argv" : '(none)'), "\n";
    printf "  %-7s: %d%s\n", 'exit', $code, $sig ? " (signal $sig)" : '';
    show('stdout', slurp("$cap/out"));
    show('stderr', slurp("$cap/err"));
    show('files',  join "\n", @files);
    chmod 0644, glob("$dir/*/*");                   # let CLEANUP remove S7's file
}

Walk through the parts that do the real work.

fork, then exec the script with STDOUT and STDERR reopened onto files. This is the most direct way to capture both streams separately and still get a true exit status. @run chooses the interpreter: a Perl script runs under $^X, the same Perl that is running probe.pl, and a .py script runs under python3. That one line is what lets the next two chapters point this harness at a Python replacement and compare the two inventories. The list form of exec passes @argv to the script without a shell, so an argument like 0 or a name with a space arrives exactly as written.

The capture files live in $cap, a second scratch directory, never in $dir. If the captures were written into the directory being snapshotted, every scenario would report two "created" files that the script did not create. A harness that observes itself is a classic way to get a wrong inventory.

snapshot records every file with an MD5 digest, before and after. Comparing the two lists gives created, changed and deleted. Comparing digests, rather than timestamps, means a file rewritten with identical contents does not count as changed. That is the right definition for a migration, because a caller cannot tell the difference either. Small files are printed in full, because their contents are part of the behaviour.

$? >> 8 and $? & 127 split the wait status into the exit code and the signal number, the same decoding Chapter 26.7 uses for EDA tools.

show abbreviates long output to its first two lines and a count. For S11, the count (256) is the observation that matters, not the 256 identical lines.

5. The Observations

The script under test is sta_summary.pl from Chapter 28.1, unchanged and repeated here so that this page runs on its own, with the same two sample reports.

#!/usr/bin/perl
# sta_summary.pl - summarise per-corner timing reports
# usage: sta_summary.pl [-v] [report_dir]
#   prints one line per corner: paths, violations, WNS and TNS
#
# (A SAMPLE legacy script written for this module, in the style of
#  scripts that have run unchanged in a flow for a decade.)

use Getopt::Long;

$verbose = 0;
GetOptions("v" => \$verbose);

$dir = shift @ARGV;
$dir = "." unless $dir;

opendir(D, $dir) || die "cannot open $dir: $!\n";
@files = grep(/\.rpt$/, readdir(D));
closedir(D);

if (!@files) {
    print "No reports found in $dir\n";
    exit 0;
}

foreach $f (@files) {
    ($corner = $f) =~ s/\.rpt$//;
    open(F, "$dir/$f") || next;
    while (<F>) {
        if (/^Endpoint\s*:\s*(\S+)/) {
            $ep = $1;
        }
        if (/slack \((\w+)\)\s+(\S+)/) {
            $paths{$corner}++;
            if ($1 eq "VIOLATED") {
                $viol{$corner}++;
                $tns{$corner} += $2;
                push(@{$eps{$corner}}, $ep);
            }
            $wns{$corner} = $2 if $2 < $wns{$corner};
        }
    }
    close(F);
}

open(CSV, ">tmp.csv");
$bad = 0;
foreach $c (sort keys %paths) {
    printf("%-16s paths=%-3d viol=%-3d wns=%.3f tns=%.3f\n",
           $c, $paths{$c}, $viol{$c}, $wns{$c}, $tns{$c});
    print CSV "$c,$viol{$c},$wns{$c}\n";
    if ($verbose) {
        foreach $e (@{$eps{$c}}) { print "    $e\n"; }
    }
    $bad++ if $viol{$c};
}
close(CSV);

exit $bad;

Report : timing (synthetic teaching format)
Design : chip_top
Corner : ss_0p72v_125c

Startpoint: u_core/r_reg[3]
Endpoint  : u_core/acc_reg[7]
  slack (VIOLATED)   -0.043

Startpoint: u_core/r_reg[1]
Endpoint  : u_core/acc_reg[2]
  slack (VIOLATED)   -0.011

Startpoint: u_core/cnt_reg[0]
Endpoint  : u_core/cnt_reg[5]
  slack (MET)         0.120

End of report

Report : timing (synthetic teaching format)
Design : chip_top
Corner : tt_0p80v_25c

Startpoint: u_core/r_reg[3]
Endpoint  : u_core/acc_reg[7]
  slack (MET)         0.087

Startpoint: u_core/cnt_reg[0]
Endpoint  : u_core/cnt_reg[5]
  slack (MET)         0.240

End of report

The run takes the inventory twice and compares the two copies, because an inventory that differs between two identical runs is not recording behaviour at all:

#!/bin/bash
perl probe.pl sta_summary.pl | tee first.inv
perl probe.pl sta_summary.pl > second.inv
echo
cmp -s first.inv second.inv && echo "second run identical: yes" || echo "second run identical: NO"

S1  two reports    argv: reports
  exit   : 1
  stdout : ss_0p72v_125c    paths=3   viol=2   wns=-0.043 tns=-0.054
           tt_0p80v_25c     paths=2   viol=0   wns=0.000 tns=0.000
  stderr : (nothing)
  files  : created tmp.csv:
               ss_0p72v_125c,2,-0.043
               tt_0p80v_25c,,
S2  two reports, -v    argv: -v reports
  exit   : 1
  stdout : ss_0p72v_125c    paths=3   viol=2   wns=-0.043 tns=-0.054
               u_core/acc_reg[7]
               u_core/acc_reg[2]
           tt_0p80v_25c     paths=2   viol=0   wns=0.000 tns=0.000
  stderr : (nothing)
  files  : created tmp.csv:
               ss_0p72v_125c,2,-0.043
               tt_0p80v_25c,,
S3  no argument at all    argv: (none)
  exit   : 1
  stdout : ss_0p72v_125c    paths=3   viol=2   wns=-0.043 tns=-0.054
           tt_0p80v_25c     paths=2   viol=0   wns=0.000 tns=0.000
  stderr : (nothing)
  files  : created tmp.csv:
               ss_0p72v_125c,2,-0.043
               tt_0p80v_25c,,
S4  empty dir, old tmp.csv    argv: empty
  exit   : 0
  stdout : No reports found in empty
  stderr : (nothing)
  files  : tmp.csv untouched
S5  missing dir    argv: nosuch
  exit   : 2
  stdout : (nothing)
  stderr : cannot open nosuch: No such file or directory
  files  : (nothing)
S6  dir named 0    argv: 0
  exit   : 0
  stdout : No reports found in .
  stderr : (nothing)
  files  : (nothing)
S7  unreadable report    argv: r
  exit   : 1
  stdout : ss_0p72v_125c    paths=3   viol=2   wns=-0.043 tns=-0.054
  stderr : (nothing)
  files  : created tmp.csv:
               ss_0p72v_125c,2,-0.043
S8  report with no slack    argv: r
  exit   : 1
  stdout : ss_0p72v_125c    paths=3   viol=2   wns=-0.043 tns=-0.054
           tt_0p80v_25c     paths=2   viol=0   wns=0.000 tns=0.000
  stderr : (nothing)
  files  : created tmp.csv:
               ss_0p72v_125c,2,-0.043
               tt_0p80v_25c,,
S9  slack with 4 decimals    argv: r
  exit   : 1
  stdout : ss_x             paths=1   viol=1   wns=-0.043 tns=-0.043
  stderr : (nothing)
  files  : created tmp.csv:
               ss_x,1,-0.0430
S10 unknown option -x    argv: -x reports
  exit   : 1
  stdout : ss_0p72v_125c    paths=3   viol=2   wns=-0.043 tns=-0.054
           tt_0p80v_25c     paths=2   viol=0   wns=0.000 tns=0.000
  stderr : Unknown option: x
  files  : created tmp.csv:
               ss_0p72v_125c,2,-0.043
               tt_0p80v_25c,,
S11 256 failing corners    argv: r
  exit   : 0
  stdout : c001             paths=1   viol=1   wns=-0.010 tns=-0.010
           c002             paths=1   viol=1   wns=-0.010 tns=-0.010
           ... 256 lines in all
  stderr : (nothing)
  files  : created tmp.csv: 256 lines

second run identical: yes

Read the output scenario by scenario, and note how much of it was not in the usage comment.

• S1 and S2: the normal case. Exit status 1, because one corner violates. tmp.csv is created in the directory the script was run from, and the clean corner's row is tt_0p80v_25c,,.
• S3: with no argument the script reads the current directory, so tmp.csv appears beside the reports.
• S4: an empty directory prints its message on STDOUT, exits 0, and does not touch tmp.csv. The old,9,-9.999 row from the earlier run is still there for the next reader.
• S5: a missing directory is the only scenario in which the script's own code writes to STDERR (S10's message comes from Getopt::Long). The exit status is 2, the errno for "No such file or directory", not 255.
• S6: the directory named 0 is never read. The script reads . instead, finds no reports, and exits 0. Chapter 2.8's false values, observed.
• S7: the unreadable report's corner simply disappears. No message, and an exit status of 1 that says nothing about the missing corner. (This probe depends on not running as root, which can read a file with no permissions.)
• S8: a report with no slack lines (here, an invented Error: license unavailable) also disappears without a word. A timing run that died before writing any paths looks identical to a corner that was never run.
• S9: STDOUT prints -0.043; tmp.csv keeps -0.0430, exactly as the report wrote it.
• S10: Unknown option: x goes to STDERR, and then the run carries on as if the option had not been given.
• S11: 256 failing corners exit 0. The Makefile in section 7 reads that as "timing clean".

6. The Inventory

The observations become the inventory: one row per behaviour, each pointing at the scenario that shows it. This is the document a migration is reviewed against in Chapter 28.9, so its columns are chosen for that. Kind says whether the behaviour looks intended, accidental, or impossible to tell from the code. Status is the same for every row today.

The registry entry for this chapter puts it in two lines that belong at the top of every inventory:

# observed: exits 0 even when the report is empty
# observed: writes $CWD/tmp.csv as a side effect
# both are now requirements until someone decides otherwise

"REQUIRED" does not mean "good". It means the replacement must reproduce it unless a named person signs off on changing it. B4, B9 and B10 are almost certainly defects, and a team may well decide to fix all three during the migration. That is a fine outcome, provided it is a decision: written down, announced to the consumers of the affected channel, and tested. It must not be an accident discovered in production. Chapter 28.3 shows what happens when a rewrite makes those decisions without knowing it is making them.

7. Who Depends on Which Behaviour

An inventory row matters in proportion to who relies on it. The way to find out is to search the flow for every reference to the script and to every file it writes. Searching only for the script's name misses the consumers that never call it and just read what it left behind.

The flow below is a sample: a Makefile step and two helper scripts, written to show the three common kinds of consumer.

# flow/Makefile (SAMPLE) - the signoff step of a block's flow
RUN   ?= run
TOOLS ?= ../tools

signoff: $(RUN)/sta_summary.txt
	@echo "timing clean: signoff may proceed"

$(RUN)/sta_summary.txt: $(wildcard $(RUN)/reports/*.rpt)
	cd $(RUN) && perl $(TOOLS)/sta_summary.pl reports > sta_summary.txt

#!/bin/bash
# post_wns.sh (SAMPLE) - feed per-corner WNS to the team dashboard.
# Reads the CSV that sta_summary.pl leaves in the run directory.
while IFS=, read -r corner viol wns; do
    echo "dashboard: $corner wns=${wns:-clean}"
done < "${1:-run}/tmp.csv"

#!/usr/bin/perl
# weekly_digest.pl (SAMPLE) - mail the failing endpoints once a week
use strict;
use warnings;
my @lines = `perl ../tools/sta_summary.pl -v run/reports`;
my @endpoints = grep { /^    \S/ } @lines;
print "failing endpoints this week: ", scalar(@endpoints), "\n";

#!/bin/bash
# Who depends on sta_summary.pl, and through which channel?
# Search for the script's name AND for the file it writes.
grep -rn -e 'sta_summary' -e 'tmp\.csv' flow | sort

flow/Makefile:5:signoff: $(RUN)/sta_summary.txt
flow/Makefile:8:$(RUN)/sta_summary.txt: $(wildcard $(RUN)/reports/*.rpt)
flow/Makefile:9:	cd $(RUN) && perl $(TOOLS)/sta_summary.pl reports > sta_summary.txt
flow/bin/post_wns.sh:3:# Reads the CSV that sta_summary.pl leaves in the run directory.
flow/bin/post_wns.sh:6:done < "${1:-run}/tmp.csv"
flow/bin/weekly_digest.pl:5:my @lines = `perl ../tools/sta_summary.pl -v run/reports`;

The | sort is there because grep -r makes no promise about the order in which it visits files. Each hit is a consumer, and each consumer depends on a different channel:

• The Makefile depends on the exit status (B12). Make stops on a non-zero status, so the script's exit status is the timing gate: a violating corner stops signoff. It also depends on STDOUT (B1), which it saves as sta_summary.txt. With 256 failing corners, B12 makes the gate pass.
• post_wns.sh depends on tmp.csv (B5, B6), and it reads the empty WNS field of a clean corner as clean. B6, which looked like an accident, is load-bearing. "Fixing" B4 so that tmp.csv gets the clean corner's real WNS (0.087) would change what the dashboard shows. That might be welcome, but only if the dashboard's owner knows it is coming.
• weekly_digest.pl depends on the -v format (B2): it counts lines that start with four spaces. Change the indent and the weekly count silently becomes zero.

Add a Consumers column to the inventory with what this search found: the Makefile on B1 and B12, post_wns.sh on B5 and B6, weekly_digest.pl on B2. A row with no known consumer is not automatically safe to change. It may have a consumer in another team's tree, or in a cron job nobody has listed. But a row with a consumer can never be changed without that consumer's owner being told.

8. Can the Inventory Fail?

An inventory is only useful if a change to the script shows up in it. A record that no change could alter would be recording nothing. So test it the same way you would test any check: change the script on purpose and see whether the inventory notices.

Here are three changes that a careful engineer might make on their first day with this script. Each is one line and each is arguably an improvement:

1. fix_wns: record the worst slack even for a corner that meets timing (B4).
2. fix_exit: exit 1 for "something failed" instead of a count (B12).
3. fix_opts: stop with a usage message on an unknown option instead of ignoring it (B11).

scenario_diff.pl splits two inventories at the scenario headers and lists the scenarios whose record changed:

use strict;
use warnings;

# scenario_diff.pl - which scenarios differ between two inventories?
# usage: perl scenario_diff.pl baseline.inv other.inv
sub load {
    my ($file) = @_;
    my (%by, $id);
    open my $fh, '<', $file or die "cannot read $file: $!\n";
    while (my $line = <$fh>) {
        $id = $1 if $line =~ /^(S\d+)\s/;          # a new scenario header
        $by{$id} .= $line if defined $id;
    }
    return \%by;
}
my ($base, $other) = map { load($_) } @ARGV;
my @changed = grep { ($other->{$_} // '') ne $base->{$_} }
              sort { substr($a, 1) <=> substr($b, 1) } keys %$base;
print @changed ? "@changed\n" : "none\n";

#!/bin/bash
# Three plausible "small fixes", each applied to a copy of the script.
# The inventory is re-taken for each and compared with the baseline.
perl probe.pl sta_summary.pl > baseline.inv

# 1. report the real worst slack of a corner that meets timing
sed 's/\$wns{\$corner} = \$2 if \$2 < \$wns{\$corner};/$wns{$corner} = $2 if !defined $wns{$corner} || $2 < $wns{$corner};/' \
    sta_summary.pl > fix_wns.pl
# 2. exit 1 for "some corner failed" instead of a count
sed 's/^exit \$bad;/exit($bad ? 1 : 0);/' sta_summary.pl > fix_exit.pl
# 3. stop on an unknown option instead of ignoring it
sed 's/^GetOptions("v" => \\\$verbose);/GetOptions("v" => \\$verbose) or die "usage: sta_summary.pl [-v] [dir]\\n";/' \
    sta_summary.pl > fix_opts.pl

for m in fix_wns fix_exit fix_opts; do
    if cmp -s sta_summary.pl $m.pl; then echo "$m: the edit did not apply"; continue; fi
    perl probe.pl $m.pl > $m.inv
    echo "$m changes: $(perl scenario_diff.pl baseline.inv $m.inv)"
done
echo
echo "--- what fix_opts changed in S10:"
diff <(sed -n '/^S10 /,/^S11 /p' baseline.inv) <(sed -n '/^S10 /,/^S11 /p' fix_opts.inv)

fix_wns changes: S1 S2 S3 S8 S10
fix_exit changes: S11
fix_opts changes: S10

--- what fix_opts changed in S10:
2,4c2,3
<   exit   : 1
<   stdout : ss_0p72v_125c    paths=3   viol=2   wns=-0.043 tns=-0.054
<            tt_0p80v_25c     paths=2   viol=0   wns=0.000 tns=0.000
---
>   exit   : 255
>   stdout : (nothing)
6,8c5,6
<   files  : created tmp.csv:
<                ss_0p72v_125c,2,-0.043
<                tt_0p80v_25c,,
---
>            usage: sta_summary.pl [-v] [dir]
>   files  : (nothing)

Each of the three "small fixes" changed what callers observe, and each in a different way.

fix_wns changed five of eleven scenarios, including tmp.csv. The clean corner's row became tt_0p80v_25c,,0.087, so post_wns.sh would stop printing clean for it. The fix is right, and it is still an interface change to the dashboard.

fix_exit changed exactly one scenario: S11. For every ordinary input, the old and new exit statuses agree (0 and 1). The difference appears only at 256 failing corners. Without that probe, this inventory would have reported the change as no change at all. That one result is the strongest argument in this chapter for boundary scenarios that look silly.

fix_opts turned a warning into a failure, and look at the details. The exit status is 255, not 2, because die uses errno when errno is set and GetOptions leaves it clear (Chapter 15.4's rule). And tmp.csv is no longer written at all. A careful reviewer looking only at the diff would see a better error message. The inventory shows a new exit code and a missing output file.

9. What an Inventory Cannot See

Be explicit about the limits, so that nobody reads the inventory as more complete than it is.

• Timing. The inventory says nothing about how long the script takes. If a replacement is ten times slower on a 50,000-path report, no row will show it. Measure it separately if it matters.
• Concurrency. Two runs in the same working directory both write tmp.csv, and the last one wins. A replacement that writes it differently (for example, through a temporary file and a rename, Chapter 13.13) changes what a reader sees during a run. The probe runs one scenario at a time, so it cannot show this.
• The environment. This inventory was taken with Perl 5.34.1 on macOS. Hash order (Chapter 6.6) does not affect this script because it sorts, but an unsorted script would give different output on every run, and the determinism check in run_probe.sh is what would catch it.
• Inputs nobody thought of. Eleven scenarios are eleven points. A report format from a newer tool version, a corner name with a space in it, a 2 GB report: none of these is in the inventory, so the inventory says nothing about them.
• Consumers outside the searched tree. Section 7 searched one flow. Another team may read tmp.csv from a cron job in a tree you cannot see.

Write those limits at the bottom of the inventory, in so many words. "Not probed" is a finding. Leaving the gap out implies that it was checked.

10. Mistakes That Make an Inventory Worthless

Probing only the happy path. S1 alone would have produced an inventory of one line: "prints a summary". Every interesting behaviour here came from an edge.

Recording one channel. Comparing STDOUT alone misses B5 through B8, B12 and most of B11. Those are exactly the channels a Makefile and a dashboard use.

Running probes in a real directory. The script writes tmp.csv wherever it runs. The harness gives every scenario a fresh directory for exactly this reason.

Writing intent instead of observation. "Exits non-zero on violations" is what the author meant. "Exit status = number of violating corners, so 256 reads as 0" is what the program does. Only the second can be checked against a replacement.

Fixing while inventorying. Each of the three mutants above is a reasonable change. Making any of them before the inventory exists destroys the baseline that would have shown what it changed.

Never testing the harness. Without the mutants, nobody would know the inventory could miss a change to the exit status.

11. Exercises

1. The stale-file consumer. Using S4's observation, write the two-line sequence of commands with which a nightly flow can publish yesterday's WNS numbers on today's dashboard while every step exits 0. Then add one scenario to probe.pl that would catch a replacement which deletes tmp.csv on an empty run, and say whether that replacement is better or worse for post_wns.sh.

2. A scenario for B10's other half. S7 and S8 drop a corner silently. Add a scenario where every report is unreadable. Predict the four channels before running it, then run it. Which inventory row does it extend, and does it reveal a new behaviour?

3. A mutant the inventory misses. Find a one-line change to sta_summary.pl that changes its behaviour for some input but changes none of the eleven scenarios. Then add the scenario that catches it. (Hint: look at what -v prints for an endpoint the script never saw.)

4. Inventory a script from your own flow. Choose a short inherited script. Write at least eight scenarios from its branches, defaults and exits, run them with a copy of probe.pl adapted to its arguments, and produce the twelve-column-style table. Mark each row's kind honestly, and list what you did not probe.

5. The consumer search, for real. In your own flow, search for one script's name and for every file it writes. How many consumers did the file search find that the name search missed?

12. What To Remember

• A migration reproduces what the script does, not what it was meant to do. The inventory is the written record of what it does.
• Reading gives predictions; probes give observations. Every inventory row should point at a probe that reproduces it.
• Record four channels: STDOUT, STDERR, exit status, and every file created, changed or deleted.
• Scenarios come from the code: one per branch, default, early exit and assumption in the reading record.
• Every scenario runs in a fresh scratch directory, and the captures live outside it.
• Measured: 11 scenarios gave 12 behaviours, most of them absent from the usage comment.
• Measured: an empty directory exits 0 and keeps yesterday's tmp.csv; a missing one exits 2; a directory named 0 is never read; 256 failing corners exit 0.
• Every row is REQUIRED until someone decides otherwise. "Accidental" describes how it got there, not whether it may change.
• Search for consumers by script name AND by output file. Here, three consumers each depended on a different channel.
• Prove the inventory can fail. Three one-line fixes all changed observed behaviour; one of them showed up in only one scenario.
• Say what was not probed: timing, concurrency, environment, unlisted inputs and consumers outside the tree.

Chapter 28.3 puts the inventory to its first use. It takes a rewrite written the usual way, from the usage comment and a good idea of what the script is for, and runs it through the same probes. The question is how much of the inventory a sensible rewrite reproduces without being told.