Skip to content
VLSI Mentor

Perl Part II · Module 28 · Legacy Perl and Python Migration

Function-by-Function Incremental Conversion

Incremental conversion means the script is never half-broken. It is half-converted, but every intermediate version gives the same answers as the original, so the flow keeps running while the migration takes as many weeks as it needs. The technique has three parts: a seam where one function can be swapped out, a Python twin of that function, and a shadow mode that runs both on real input and records every disagreement. This chapter applies it to the parser inside the module's case-study script. The first Python twin printed exactly the same summary as the Perl one, and the shadow log still showed that every one of its four records was different.

Advanced28 min readPerlPart IIEDA Flows

Part II · Module 28 · Chapter 28.5 · Function-by-Function Incremental Conversion

1. The Idea: A Seam, Two Implementations, One Switch

Chapter 28.4 sent two of its five scripts to incremental conversion: the ones that change too often to leave alone and are too large or too untested to rewrite in one go. Incremental conversion is how a script like that moves to Python without ever being broken.

The method has five steps, and a sixth that people forget.

Figure 28.5.1 — One function at a time

Six steps of incremental conversion: take the inventory; extract the seam in Perl and prove nothing moved; write the Python twin; run both in shadow mode with Perl's answer used and disagreements logged; route through Python with Perl kept as a fallback; delete the Perl twin on purpose.Converting one function1Take the inventorythe definition of working2Extract the seamin Perl; nothing may move3Write the Python twinagainst the seam's contract4Run in shadowPerl answers; diffs logged5Route to PythonPerl kept as the fallback6Delete the Perl twinlater, and on purpose
Each step leaves a working flow behind it. The inventory comes first because it is the definition of 'working'. The seam is cut in Perl and proved to change nothing before any Python exists. The twin is written against the seam's contract, then run in shadow, where the Perl answer is still the one used and every disagreement is logged. Only when the log stays empty is the switch moved. The Perl implementation stays available as an instant fallback, and is deleted later, deliberately.

A seam is a place in the program where one implementation can be swapped for another without editing the code around it. In a legacy script there usually is none: the logic is spread through the main loop and shares global variables with it. So the first job is to cut one, in Perl, and prove that cutting it changed nothing.

2. Step 1: Extract the Seam in Perl

The parser in sta_summary.pl is the obvious candidate. It is the part with the most logic, the part a new tool format would change, and it has a clear input (one report file) and a clear output (four numbers and a list of endpoints). Here is the original again, followed by the extracted version:

Azvya Education Pvt. Ltd.VLSI Mentor
sta_summary.pl
#!/usr/bin/perl
# sta_summary.pl - summarise per-corner timing reports
# usage: sta_summary.pl [-v] [report_dir]
#   prints one line per corner: paths, violations, WNS and TNS
#
# (A SAMPLE legacy script written for this module, in the style of
#  scripts that have run unchanged in a flow for a decade.)

use Getopt::Long;

$verbose = 0;
GetOptions("v" => \$verbose);

$dir = shift @ARGV;
$dir = "." unless $dir;

opendir(D, $dir) || die "cannot open $dir: $!\n";
@files = grep(/\.rpt$/, readdir(D));
closedir(D);

if (!@files) {
    print "No reports found in $dir\n";
    exit 0;
}

foreach $f (@files) {
    ($corner = $f) =~ s/\.rpt$//;
    open(F, "$dir/$f") || next;
    while (<F>) {
        if (/^Endpoint\s*:\s*(\S+)/) {
            $ep = $1;
        }
        if (/slack \((\w+)\)\s+(\S+)/) {
            $paths{$corner}++;
            if ($1 eq "VIOLATED") {
                $viol{$corner}++;
                $tns{$corner} += $2;
                push(@{$eps{$corner}}, $ep);
            }
            $wns{$corner} = $2 if $2 < $wns{$corner};
        }
    }
    close(F);
}

open(CSV, ">tmp.csv");
$bad = 0;
foreach $c (sort keys %paths) {
    printf("%-16s paths=%-3d viol=%-3d wns=%.3f tns=%.3f\n",
           $c, $paths{$c}, $viol{$c}, $wns{$c}, $tns{$c});
    print CSV "$c,$viol{$c},$wns{$c}\n";
    if ($verbose) {
        foreach $e (@{$eps{$c}}) { print "    $e\n"; }
    }
    $bad++ if $viol{$c};
}
close(CSV);

exit $bad;
Azvya Education Pvt. Ltd.VLSI Mentor
sta_summary_v3.pl
#!/usr/bin/perl
# sta_summary_v3.pl - sta_summary.pl with ONE function extracted, so that
# function can be converted to Python while the rest of the flow keeps running.
#   PARSE_IMPL=perl    (default) use the Perl parse_report
#   PARSE_IMPL=shadow  run BOTH, use Perl's answer, log disagreements to $SHADOW_LOG
#   PARSE_IMPL=python  use the Python twin
use strict;
use warnings;
no warnings 'uninitialized';      # the legacy output relies on undef printing as ''/0
use Getopt::Long;
use File::Basename qw(basename dirname);

my $verbose = 0;
GetOptions("v" => \$verbose);     # return value ignored, as before (B11)

my $dir = shift @ARGV;
$dir = "." unless $dir;           # "0" becomes ".", as before (B9)

opendir(my $dh, $dir) || die "cannot open $dir: $!\n";
my @files = grep { /\.rpt$/ } readdir($dh);
closedir($dh);

if (!@files) {
    print "No reports found in $dir\n";
    exit 0;
}

my %rec;                          # corner => record from parse_report
for my $f (@files) {
    (my $corner = $f) =~ s/\.rpt$//;
    my $r = parse_report("$dir/$f") or next;      # unreadable: skipped (B10)
    $rec{$corner} = $r if $r->{paths};            # no slack lines: absent (B10)
}

open(my $csv, '>', 'tmp.csv');
my $bad = 0;
for my $c (sort keys %rec) {
    my $r = $rec{$c};
    printf("%-16s paths=%-3d viol=%-3d wns=%.3f tns=%.3f\n",
           $c, $r->{paths}, $r->{viol}, $r->{wns}, $r->{tns});
    print {$csv} "$c,$r->{viol},$r->{wns}\n";
    if ($verbose) { print "    $_\n" for @{ $r->{eps} } }
    $bad++ if $r->{viol};
}
close($csv);
exit $bad;

# ---------------------------------------------------------------- the seam
# parse_report(PATH) -> { paths, viol, tns, wns, eps } or undef if unreadable.
# viol and wns stay UNDEF when the legacy script left them undef: the CSV
# prints undef as an empty field (B6), so undef-ness is part of the contract.
sub parse_report {
    my ($path) = @_;
    my $impl = $ENV{PARSE_IMPL} // 'perl';
    return parse_report_python($path) if $impl eq 'python';
    my $r = parse_report_perl($path);
    if ($impl eq 'shadow') {
        my $p = parse_report_python($path);
        my $diff = compare_records($r, $p);
        if ($diff && $ENV{SHADOW_LOG}) {
            open my $log, '>>', $ENV{SHADOW_LOG} or die "cannot append $ENV{SHADOW_LOG}: $!\n";
            print {$log} basename($path), ": $diff\n";
        }
    }
    return $r;
}

sub parse_report_perl {
    my ($path) = @_;
    open(my $fh, '<', $path) || return undef;
    my (%r, $ep);
    $r{paths} = 0;
    while (<$fh>) {
        $ep = $1 if /^Endpoint\s*:\s*(\S+)/;
        if (/slack \((\w+)\)\s+(\S+)/) {
            $r{paths}++;
            if ($1 eq "VIOLATED") {
                $r{viol}++;
                $r{tns} += $2;
                push @{ $r{eps} }, $ep;
            }
            $r{wns} = $2 if $2 < ($r{wns} // 0);   # undef compares as 0 (B4), now visible
        }
    }
    close($fh);
    return \%r;
}

sub parse_report_python {
    my ($path) = @_;
    my $helper = dirname(__FILE__) . '/parse_report.py';
    open(my $py, '-|', 'python3', $helper, $path) or die "cannot run $helper: $!\n";
    my $line = <$py>;
    close($py);
    return undef if $? >> 8 == 3;                 # the twin's "unreadable" status
    die "parse_report.py failed on $path\n" if $? || !defined $line;
    chomp $line;
    my ($paths, $viol, $tns, $wns, $eps) = split /\t/, $line, -1;
    return { paths => $paths,
             viol  => $viol eq '' ? undef : $viol,
             tns   => $tns  eq '' ? undef : $tns,
             wns   => $wns  eq '' ? undef : $wns,
             eps   => [ grep { length } split /,/, $eps ] };
}

sub compare_records {
    my ($p, $y) = @_;
    return 'python twin returned nothing' if defined $p && !defined $y;
    return '' if !defined $p && !defined $y;
    my @bad;
    for my $k (qw(paths viol tns wns)) {
        my ($a1, $b1) = map { defined $_->{$k} ? $_->{$k} : 'undef' } $p, $y;
        push @bad, "$k perl=$a1 python=$b1" if $a1 ne $b1;
    }
    my ($ea, $eb) = map { join ',', @{ $_->{eps} || [] } } $p, $y;
    push @bad, "eps perl=$ea python=$eb" if $ea ne $eb;
    return join '; ', @bad;
}

The extraction is deliberately conservative. Read it with Chapter 28.2's inventory beside you, and every behaviour is still there, now with a comment naming its inventory row:

  • GetOptions's return value is still ignored (B11), and "0" still becomes "." (B9).
  • An unreadable report still disappears (parse_report(...) or next) and so does one with no slack lines (if $r->{paths}), which is B10.
  • wns still compares against 0 when nothing has been recorded, now written as ($r{wns} // 0) so that the quirk (B4) is visible instead of hidden in an undefined comparison.
  • no warnings 'uninitialized' is there on purpose. The output relies on an undefined value printing as empty in the CSV and as zero in printf (B6), and the new code must not start printing warnings that the old code never printed, because STDERR is one of the four channels.

One thing did change, knowingly. $ep is now a my variable inside parse_report_perl, so an endpoint can no longer leak from one report into the next (the stale-endpoint assumption Chapter 28.1 noted). For well-formed reports that makes no difference. For a report whose first slack line comes before any Endpoint line, it does. None of the eleven scenarios contains such a report, so the inventory cannot see this change. Write it down as a decided change, the way Chapter 28.2 says to, rather than letting "11 of 11" imply there is none.

Before any Python exists, prove the seam: run the extracted script through the original's full inventory. That is the first line of route_check.sh in section 7, and it reports 11 of 11 scenarios identical. Until it does, nothing else in this chapter is worth doing.

3. The Record Is a Contract

The seam is the function parse_report, and its return value is the contract both implementations must meet. The comment above it says what that contract is, including the part that is easy to miss:

viol and wns stay UNDEF when the legacy script left them undef: the CSV prints undef as an empty field (B6), so undef-ness is part of the contract.

That sentence is the whole difficulty of this chapter in one line. A clean corner has no violations. In the original, $viol{$corner} was never incremented, so it is undefined, and the CSV row is tt_0p80v_25c,,. "Zero" and "undefined" print differently, so a twin that returns 0 returns a different answer, even though zero is arguably the correct number.

The two implementations live in different processes, so the contract also needs a wire format. The twin prints one tab-separated line, paths viol tns wns endpoints, where an empty field means undef. parse_report_python reads it back over a pipe (Chapter 15.11), and exit status 3 means "unreadable", mapped to undef the way open ... || return undef does in Perl. Every detail of that encoding is part of the contract too.

4. Step 2: The First Python Twin

Here is the twin as most people would first write it: a direct translation of parse_report_perl, readable and correct by any ordinary standard.

Azvya Education Pvt. Ltd.VLSI Mentor
parse_report_first.py
#!/usr/bin/env python3
"""parse_report.py - the Python twin of parse_report() in sta_summary_v3.pl.

usage: parse_report.py REPORT
prints ONE tab-separated record:  paths  viol  tns  wns  endpoints
An empty field means "undef" on the Perl side. Exit 3 if unreadable.
FIRST VERSION: a straightforward translation. (Saved as parse_report_first.py.)
"""
import re
import sys

ENDPOINT = re.compile(r"^Endpoint\s*:\s*(\S+)")
SLACK = re.compile(r"slack \((\w+)\)\s+(\S+)")


def parse(path):
    paths, viol, tns, wns, eps, ep = 0, 0, 0.0, 0.0, [], None
    with open(path) as fh:
        for line in fh:
            m = ENDPOINT.search(line)
            if m:
                ep = m.group(1)
            m = SLACK.search(line)
            if m:
                paths += 1
                if m.group(1) == "VIOLATED":
                    viol += 1
                    tns += float(m.group(2))
                    eps.append(ep)
                wns = min(wns, float(m.group(2)))
    return paths, viol, tns, wns, eps


def main():
    try:
        paths, viol, tns, wns, eps = parse(sys.argv[1])
    except OSError:
        return 3
    print(f"{paths}\t{viol}\t{tns}\t{wns}\t{','.join(eps)}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

It starts every counter at zero, parses the slack as a float, and keeps the minimum. Hand it the sample reports and compare the summaries, and you will find nothing wrong. That is exactly the trap.

5. Step 3: Shadow Mode

In shadow mode (PARSE_IMPL=shadow), parse_report calls both implementations on every report. It uses the Perl answer, so the flow's output cannot change, and it compares the two records field by field, appending any disagreement to the file named in SHADOW_LOG. The comparison is text equality (ne) on every field. That is deliberately strict, and section 6 explains why.

Three details of parse_report matter in production:

  • Perl's answer wins. Shadow mode is safe to turn on in a real nightly flow precisely because it cannot change what the flow does.
  • The log goes to a file, never to STDERR. STDERR is one of the four observable channels, and a migration aid that prints to it has changed the script's behaviour.
  • The log is opened only when there is something to write. An empty log file in the run directory would be a new side effect.

The corpus is the two sample reports plus two from Chapter 28.2's edges: a slack written with four decimals, and a report from a run that died before writing any paths.

Azvya Education Pvt. Ltd.VLSI Mentor
reports/ss_0p72v_125c.rpt
Report : timing (synthetic teaching format)
Design : chip_top
Corner : ss_0p72v_125c

Startpoint: u_core/r_reg[3]
Endpoint  : u_core/acc_reg[7]
  slack (VIOLATED)   -0.043

Startpoint: u_core/r_reg[1]
Endpoint  : u_core/acc_reg[2]
  slack (VIOLATED)   -0.011

Startpoint: u_core/cnt_reg[0]
Endpoint  : u_core/cnt_reg[5]
  slack (MET)         0.120

End of report
Azvya Education Pvt. Ltd.VLSI Mentor
reports/tt_0p80v_25c.rpt
Report : timing (synthetic teaching format)
Design : chip_top
Corner : tt_0p80v_25c

Startpoint: u_core/r_reg[3]
Endpoint  : u_core/acc_reg[7]
  slack (MET)         0.087

Startpoint: u_core/cnt_reg[0]
Endpoint  : u_core/cnt_reg[5]
  slack (MET)         0.240

End of report
Azvya Education Pvt. Ltd.VLSI Mentor
corpus/ss_4dp.rpt
Report : timing

Startpoint: u_io/a_reg
Endpoint  : u_io/q_reg
  slack (VIOLATED)   -0.0430

End of report
Azvya Education Pvt. Ltd.VLSI Mentor
corpus/ff_dead.rpt
Report : timing
Error: license unavailable

stage.sh installs one twin beside a copy of the script, so that the page's own files are never modified by a run:

Azvya Education Pvt. Ltd.VLSI Mentor
stage.sh
#!/bin/bash
# stage.sh TWIN : print a fresh directory holding sta_summary_v3.pl with the
# given Python twin installed as parse_report.py. The page's own files are
# never modified, so every run below starts from the same state.
stage=$(mktemp -d)
cp sta_summary_v3.pl "$stage/"
cp "$1" "$stage/parse_report.py"
echo "$stage"
Azvya Education Pvt. Ltd.VLSI Mentor
shadow_run.sh
#!/bin/bash
# Run the extracted script in all three modes over the same corpus, first
# with the FIRST Python twin, then with the corrected one.
here=$(pwd)
cp reports/*.rpt corpus/          # the two sample reports join the corpus
for twin in parse_report_first.py parse_report.py; do
    stage=$(bash stage.sh "$twin")
    echo "=== twin: $twin"
    for impl in perl shadow python; do
        work=$(mktemp -d)
        cd "$work" || exit 1
        PARSE_IMPL=$impl SHADOW_LOG="$work/shadow.log" \
            perl "$stage/sta_summary_v3.pl" "$here/corpus" > out.txt
        echo "  $impl: exit $?, STDOUT checksum $(cksum < out.txt | cut -d' ' -f1)"
        echo "    tmp.csv: $(tr '\n' ' ' < tmp.csv)"
        [ -s shadow.log ] && sort shadow.log | sed "s/^/    shadow: /"
        cd "$here"
    done
done
Azvya Education Pvt. Ltd.VLSI Mentor
Captured output
=== twin: parse_report_first.py
  perl: exit 2, STDOUT checksum 3659609560
    tmp.csv: ss_0p72v_125c,2,-0.043 ss_4dp,1,-0.0430 tt_0p80v_25c,, 
  shadow: exit 2, STDOUT checksum 3659609560
    tmp.csv: ss_0p72v_125c,2,-0.043 ss_4dp,1,-0.0430 tt_0p80v_25c,, 
    shadow: ff_dead.rpt: viol perl=undef python=0; tns perl=undef python=0.0; wns perl=undef python=0.0
    shadow: ss_0p72v_125c.rpt: tns perl=-0.054 python=-0.05399999999999999
    shadow: ss_4dp.rpt: wns perl=-0.0430 python=-0.043
    shadow: tt_0p80v_25c.rpt: viol perl=undef python=0; tns perl=undef python=0.0; wns perl=undef python=0.0
  python: exit 2, STDOUT checksum 3659609560
    tmp.csv: ss_0p72v_125c,2,-0.043 ss_4dp,1,-0.043 tt_0p80v_25c,0,0.0 
=== twin: parse_report.py
  perl: exit 2, STDOUT checksum 3659609560
    tmp.csv: ss_0p72v_125c,2,-0.043 ss_4dp,1,-0.0430 tt_0p80v_25c,, 
  shadow: exit 2, STDOUT checksum 3659609560
    tmp.csv: ss_0p72v_125c,2,-0.043 ss_4dp,1,-0.0430 tt_0p80v_25c,, 
  python: exit 2, STDOUT checksum 3659609560
    tmp.csv: ss_0p72v_125c,2,-0.043 ss_4dp,1,-0.0430 tt_0p80v_25c,,

6. What the Shadow Run Found

Look first at the column people usually check. With the first twin, the STDOUT checksum is identical in all three modes: Perl, shadow and Python. The summary a person reads is byte-for-byte the same. If the migration had been checked by eye, or by diffing STDOUT, the first twin would have been routed into production.

Now look at the shadow log. All four reports disagree, in four different ways:

  1. viol perl=undef python=0 on the clean corner and the dead report. The twin starts its counter at zero; the Perl hash element was never created. In STDOUT both print as viol=0, because printf "%d" formats undef as zero. In the CSV, one is an empty field and the other is 0.
  2. wns perl=undef python=0.0 on the same two. The same story, plus a second difference: Python writes the float zero as 0.0, where Perl would write 0.
  3. tns perl=-0.054 python=-0.05399999999999999. Both languages add the same two IEEE doubles and get the same bits. They write them differently: Perl prints a number with 15 significant digits, and Python prints the shortest string that converts back to the same float, which here needs 16 digits. Chapter 2.4 covers how Perl stores numbers; this is the other half, how it prints them. STDOUT hides this one because %.3f rounds both to -0.054.
  4. wns perl=-0.0430 python=-0.043. The Perl code stores the text it captured; the twin converts to float and back. -0.0430 and -0.043 are the same number and different bytes, and the CSV carries the bytes.

Then look at the routed run (python) with the first twin, where the twin's answers are actually used. STDOUT is still identical, but tmp.csv is not: tt_0p80v_25c,0,0.0 instead of tt_0p80v_25c,,, and ss_4dp,1,-0.043 instead of -0.0430. Chapter 28.2 found that post_wns.sh reads an empty WNS field as clean. Through the first twin, the dashboard would have shown the clean corner with a WNS of 0.0.

7. The Corrected Twin

Each disagreement turns into a rule, and the second version of the twin states all three in its docstring:

Azvya Education Pvt. Ltd.VLSI Mentor
parse_report.py
#!/usr/bin/env python3
"""parse_report.py - the Python twin of parse_report() in sta_summary_v3.pl.

usage: parse_report.py REPORT
prints ONE tab-separated record:  paths  viol  tns  wns  endpoints
An empty field means "undef" on the Perl side. Exit 3 if unreadable.

SECOND VERSION, after the shadow run. Three rules the first version broke:
  * a value the Perl code never set is EMPTY, not 0 (the CSV prints it empty)
  * wns is the TEXT captured from the report, never re-formatted
  * a computed number is written the way Perl writes it: %.15g
"""
import re
import sys

ENDPOINT = re.compile(r"^Endpoint\s*:\s*(\S+)")
SLACK = re.compile(r"slack \((\w+)\)\s+(\S+)")


def perl_num(x):
    """Stringify a float as Perl does for print: 15 significant digits."""
    return "%.15g" % x


def parse(path):
    paths, viol, tns, wns, eps, ep = 0, None, None, None, [], None
    with open(path) as fh:
        for line in fh:
            m = ENDPOINT.search(line)
            if m:
                ep = m.group(1)
            m = SLACK.search(line)
            if m:
                paths += 1
                text = m.group(2)
                if m.group(1) == "VIOLATED":
                    viol = (viol or 0) + 1
                    tns = (tns or 0.0) + float(text)
                    eps.append(ep)
                if float(text) < (float(wns) if wns is not None else 0.0):
                    wns = text                      # keep the report's own text
    return paths, viol, tns, wns, eps


def main():
    try:
        paths, viol, tns, wns, eps = parse(sys.argv[1])
    except OSError:
        return 3
    fields = [str(paths),
              "" if viol is None else str(viol),
              "" if tns is None else perl_num(tns),
              "" if wns is None else wns,
              ",".join(eps)]
    print("\t".join(fields))
    return 0


if __name__ == "__main__":
    sys.exit(main())
  • A value the Perl code never set is empty, not zero. viol and tns start as None and are written as empty fields.
  • wns is the captured text, compared numerically but stored and returned exactly as the report wrote it.
  • A computed number is written the way Perl writes it, with "%.15g". perl_num(-0.05399999999999999) is -0.054, the same string Perl prints.

In the second half of the shadow output above, the corrected twin produces no shadow log at all, and tmp.csv is byte-identical in every mode. Now run the stronger check: the whole eleven-scenario inventory from Chapter 28.2, with the extracted script routed through each twin. The probe harness and the comparison tool are repeated in section 11 so this page runs on its own.

Azvya Education Pvt. Ltd.VLSI Mentor
route_check.sh
#!/bin/bash
# The extracted script against the ORIGINAL's full eleven-scenario inventory:
# Perl seam, then routed through each Python twin.
perl probe.pl sta_summary.pl > original.inv
check() {      # check LABEL SCRIPT : inventory SCRIPT and compare
    perl probe.pl "$2" > candidate.inv
    printf '%-42s %s\n' "$1" "$(perl compare_inv.pl original.inv candidate.inv | tail -1)"
}
check "PARSE_IMPL=perl" sta_summary_v3.pl
for twin in parse_report_first.py parse_report.py; do
    stage=$(bash stage.sh "$twin")
    PARSE_IMPL=python check "PARSE_IMPL=python, $twin" "$stage/sta_summary_v3.pl"
done
Azvya Education Pvt. Ltd.VLSI Mentor
Captured output
PARSE_IMPL=perl                            scenarios identical on all four channels: 11 of 11
PARSE_IMPL=python, parse_report_first.py   scenarios identical on all four channels: 5 of 11
PARSE_IMPL=python, parse_report.py         scenarios identical on all four channels: 11 of 11

Three lines and the whole chapter in numbers: the Perl seam is identical in 11 of 11 scenarios; routing through the first twin drops to 5 of 11; routing through the corrected twin is back to 11 of 11. Two different checks, the shadow log and the inventory, agree about which twin is safe. That agreement is what makes it reasonable to move the switch.

8. Step 4: Route; Step 5: Delete, on Purpose

Routing is setting PARSE_IMPL=python in the flow's environment, for example in the Makefile recipe or the CI job. Two properties make this step safe.

It is a one-word change, and so is undoing it. If anything looks wrong in the first nights after the switch, PARSE_IMPL=perl puts the original parser back with no code change and no deployment. The Perl twin is the rollback plan, which is why it is not deleted at this step. Chapter 28.10 treats cutover and rollback for the whole script in the same way.

The rest of the script has not moved. Only the parser crossed the language boundary. The CSV writer, the exit logic and the option handling are still the Perl code that the inventory was taken from. The next function can be converted the same way, independently, with the same switch pattern.

Deleting the Perl twin is a separate, later decision: after a defined number of clean regression cycles, with nobody having used the fallback. Remove parse_report_perl and the shadow branch in one commit whose message says what was removed and why. "Later, and on purpose" is the sixth step of the figure for a reason. A fallback nobody ever removes becomes a second implementation that nobody tests, and the next person to set PARSE_IMPL=perl in an emergency gets code that has drifted.

9. What the Half-Converted State Costs

Incremental conversion is safe, not free. Be honest about the costs, because they are what a reviewer will ask about.

  • Two implementations of one function must be kept in step until the Perl one is deleted. Any fix to one needs the same fix to the other, and shadow mode is what tells you when you forgot.
  • A process per call. The routed script starts one python3 process per report. For four reports that is invisible. For the 256 corners of Chapter 28.2's S11 it is 256 interpreter start-ups, and in shadow mode every report is parsed twice. When the remaining Perl is small enough, the end state is to move the caller to Python too, and the process boundary disappears.
  • A wire format (the tab-separated record) that exists only for the migration, with its own rules: empty means undef, exit 3 means unreadable. It must be documented in both files, as it is here.
  • Mixed-language debugging. A failure might be in the Perl, in the Python, or in the encoding between them. Shadow mode reduces this, because a disagreement names the field and the file.

These costs are the price of a flow that never breaks. They are worth paying for the scripts Chapter 28.4 sent here, and not worth paying for a 40-line formatter, which is why the strategy is chosen per script.

10. Mistakes in Incremental Conversion

Writing the twin before proving the seam. If the Perl extraction itself changed behaviour, every later disagreement is ambiguous. Prove 11 of 11 first.

Comparing final output instead of the record. Measured: identical STDOUT, four disagreeing records, and a wrong tmp.csv once routed.

Translating "undefined" as zero. The most natural line of Python, viol = 0, was a behaviour change in the CSV.

Letting the number formatting drift. Python's float output and Perl's differ in the 16th digit, and the record carried that difference.

Logging disagreements to STDERR. A migration aid that writes to an observable channel has changed the script.

Deleting the fallback on the day of the switch. The Perl twin is the rollback; remove it after the cycles that prove it is not needed.

Modifying files in place during a comparison. An interrupted early version of shadow_run.sh copied one twin over the other and left the wrong one installed. stage.sh makes every run start from the page's files.

11. The Harness, Repeated

probe.pl and compare_inv.pl from Chapters 28.2 and 28.3, unchanged, so that route_check.sh runs on this page:

Azvya Education Pvt. Ltd.VLSI Mentor
probe.pl
use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Find;
use File::Copy qw(copy);
use File::Spec;
use Digest::MD5;

# probe.pl - record what a script DOES, one scenario at a time.
# usage: perl probe.pl SCRIPT      (a .py script is run with python3)
# Every scenario runs in its own fresh scratch directory. For each one we
# record the four channels a caller can observe: STDOUT, STDERR, the exit
# status, and every file created, changed or deleted in that directory.
my $script = File::Spec->rel2abs(shift // die "usage: probe.pl SCRIPT\n");
my @run = $script =~ /\.py\z/ ? ('python3', $script) : ($^X, $script);
my $fixtures = File::Spec->rel2abs('reports');     # the two sample reports

sub put {                                           # write a file, making dirs
    my ($path, $text) = @_;
    my ($vol, $dirs) = File::Spec->splitpath($path);
    mkdir $dirs unless -d $dirs;
    open my $fh, '>', $path or die "cannot write $path: $!\n";
    print {$fh} $text;
    close $fh;
}
sub sample_reports {                                # copy both samples into $to
    my ($to) = @_;
    mkdir $to;
    copy("$fixtures/$_", "$to/$_") or die "copy $_: $!\n"
        for qw(ss_0p72v_125c.rpt tt_0p80v_25c.rpt);
}
sub one_path {                                      # a minimal one-path report
    my ($slack, $status) = @_;
    return "Report : timing (synthetic teaching format)\n\n"
         . "Startpoint: u_core/a_reg\nEndpoint  : u_core/b_reg\n"
         . "  slack ($status)   $slack\n\nEnd of report\n";
}

# name, setup(sub receiving the scratch dir), arguments
my @scenarios = (
  [ 'S1  two reports',             sub { sample_reports("$_[0]/reports") }, 'reports' ],
  [ 'S2  two reports, -v',         sub { sample_reports("$_[0]/reports") }, '-v', 'reports' ],
  [ 'S3  no argument at all',      sub { sample_reports($_[0]) } ],
  [ 'S4  empty dir, old tmp.csv',  sub { mkdir "$_[0]/empty"; put("$_[0]/tmp.csv", "old,9,-9.999\n") }, 'empty' ],
  [ 'S5  missing dir',             sub { }, 'nosuch' ],
  [ 'S6  dir named 0',             sub { sample_reports("$_[0]/0") }, '0' ],
  [ 'S7  unreadable report',       sub { sample_reports("$_[0]/r"); chmod 0, "$_[0]/r/tt_0p80v_25c.rpt" }, 'r' ],
  [ 'S8  report with no slack',    sub { sample_reports("$_[0]/r");
                                         put("$_[0]/r/ff_0p88v_m40c.rpt", "Report : timing\nError: license unavailable\n") }, 'r' ],
  [ 'S9  slack with 4 decimals',   sub { put("$_[0]/r/ss_x.rpt", one_path('-0.0430', 'VIOLATED')) }, 'r' ],
  [ 'S10 unknown option -x',       sub { sample_reports("$_[0]/reports") }, '-x', 'reports' ],
  [ 'S11 256 failing corners',     sub { put(sprintf("%s/r/c%03d.rpt", $_[0], $_), one_path('-0.010', 'VIOLATED'))
                                             for 1 .. 256 }, 'r' ],
);

sub snapshot {                                      # relative path => md5
    my ($root) = @_;
    my %s;
    find({ no_chdir => 1, wanted => sub {
        return unless -f $_;
        my $rel = File::Spec->abs2rel($_, $root);
        if (open my $fh, '<', $_) { binmode $fh; $s{$rel} = Digest::MD5->new->addfile($fh)->hexdigest }
        else                      { $s{$rel} = 'unreadable' }
    } }, $root);
    return \%s;
}
sub slurp { open my $fh, '<', $_[0] or return ''; local $/; return scalar <$fh> }
sub show {                                          # print a channel, briefly
    my ($label, $text) = @_;
    my @l = split /\n/, $text;
    if (!@l) { printf "  %-7s: (nothing)\n", $label; return }
    my @shown = @l > 4 ? (@l[0 .. 1], sprintf "... %d lines in all", scalar @l) : @l;
    printf "  %-7s: %s\n", $label, shift @shown;
    printf "  %-7s  %s\n", '', $_ for @shown;
}

for my $sc (@scenarios) {
    my ($name, $setup, @argv) = @$sc;
    my $dir = tempdir(CLEANUP => 1);
    my $cap = tempdir(CLEANUP => 1);                # captures live OUTSIDE $dir
    $setup->($dir);
    my $before = snapshot($dir);

    my $pid = fork() // die "fork: $!\n";
    if ($pid == 0) {
        open STDOUT, '>', "$cap/out" or die;
        open STDERR, '>', "$cap/err" or die;
        chdir $dir or die "chdir: $!\n";
        exec @run, @argv or die "exec: $!\n";
    }
    waitpid $pid, 0;
    my ($code, $sig) = ($? >> 8, $? & 127);
    my $after = snapshot($dir);

    my @files;
    for my $f (sort keys %{{ %$before, %$after }}) {
        my ($was, $now) = ($before->{$f}, $after->{$f});
        my $what = !defined $was ? 'created'
                 : !defined $now ? 'deleted'
                 : $was ne $now  ? 'changed'
                 :                 '';
        next unless $what;
        my @body = $what eq 'deleted' ? () : split /\n/, slurp("$dir/$f");
        push @files, @body > 3 ? sprintf("%s %s: %d lines", $what, $f, scalar @body)
                               : ("$what $f:", map { "    $_" } @body);
    }
    push @files, "tmp.csv untouched" if exists $before->{'tmp.csv'} && !grep /tmp\.csv/, @files;

    print "$name    argv: ", (@argv ? "@argv" : '(none)'), "\n";
    printf "  %-7s: %d%s\n", 'exit', $code, $sig ? " (signal $sig)" : '';
    show('stdout', slurp("$cap/out"));
    show('stderr', slurp("$cap/err"));
    show('files',  join "\n", @files);
    chmod 0644, glob("$dir/*/*");                   # let CLEANUP remove S7's file
}
Azvya Education Pvt. Ltd.VLSI Mentor
compare_inv.pl
use strict;
use warnings;

# compare_inv.pl - channel-by-channel comparison of two inventories.
# usage: perl compare_inv.pl old.inv new.inv
sub load {
    my ($file) = @_;
    my (%inv, $id, $ch);
    open my $fh, '<', $file or die "cannot read $file: $!\n";
    while (my $line = <$fh>) {
        if ($line =~ /^(S\d+)\s+(.*?)\s+argv:/) { $id = $1; $inv{$id}{name} = $2; next }
        if ($line =~ /^  (\w+)\s*: (.*)$/)       { $ch = $1; $inv{$id}{$ch} = "$2\n"; next }
        $inv{$id}{$ch} .= $line if defined $ch;  # a continuation line
    }
    return \%inv;
}
my ($old, $new) = map { load($_) } @ARGV;
printf "%-30s %-9s %-7s %-7s %s\n", 'scenario', 'exit', 'stdout', 'stderr', 'files';
my $same_all = 0;
for my $id (sort { substr($a, 1) <=> substr($b, 1) } keys %$old) {
    my ($o, $n) = ($old->{$id}, $new->{$id});
    my @verdict = map { $o->{$_} eq $n->{$_} ? 'same' : 'DIFF' } qw(stdout stderr files);
    chomp(my $eo = $o->{exit}); chomp(my $en = $n->{exit});
    $same_all++ if $eo eq $en && !grep { $_ eq 'DIFF' } @verdict;
    printf "%-30s %-9s %-7s %-7s %s\n", sprintf("%-4s %s", $id, $o->{name}), "$eo -> $en", @verdict;
}
printf "\nscenarios identical on all four channels: %d of %d\n", $same_all, scalar keys %$old;

12. Exercises

1. The leaking endpoint. Write a report whose first slack line comes before any Endpoint line, and put it in a directory with one well-formed report that is read before it. Show that the original and sta_summary_v3.pl (Perl mode) print different -v output for it. Then add it as scenario S12 to probe.pl and confirm the inventory now sees the change that section 2 documented.

2. Shadow with tolerance. Change compare_records so that tns is compared after formatting both values with %.3f, the precision at which the script uses it. Run shadow_run.sh with the first twin. Which of the four disagreements disappear, and which channel would still have been wrong if you had routed on that basis?

3. Convert the second function. Extract the CSV writer into a seam of its own (write_csv(\%rec)), with its own CSV_IMPL switch, and write its Python twin. What contract does the twin need, and how does it receive %rec across the process boundary?

4. Count the processes. Instrument parse_report_python to count its calls, and run the S11 scenario (256 reports) in shadow mode. How many interpreter start-ups does one run cost? Measure the wall time of perl mode and shadow mode three times each and state the result as "slower in N of 3 rounds", not as a millisecond figure.

5. A twin for your own script. Choose one function in a script Chapter 28.4 would send to INCREMENTAL. Write down its contract, including every value that can be undefined, before writing a line of Python. Then write the twin and run it in shadow on a week of real inputs.

13. What To Remember

  • Incremental conversion keeps the flow working at every step: seam, twin, shadow, route, and delete later.
  • Cut the seam in Perl first and prove it: 11 of 11 scenarios identical before any Python exists.
  • Any knowing change in the extraction (here, $ep scoped per report) is written down, even when the inventory cannot see it.
  • The seam's return value is a contract, and undefined is part of it. Zero and undef printed differently in the CSV.
  • Shadow mode runs both, uses Perl's answer, and logs disagreements to a file, never to STDERR.
  • Measured: the first twin printed byte-identical STDOUT and disagreed on 4 of 4 records. Routed, it matched 5 of 11 scenarios.
  • Compare at the seam, as text, field by field. printf rounds away the differences that other channels carry.
  • Perl prints numbers with 15 significant digits; Python's shortest round-trip form can need more. "%.15g" makes a twin write what Perl writes.
  • Keep captured text as text when the original did. -0.0430 is not -0.043 in a CSV.
  • Routing is a one-word switch; the Perl twin is the rollback. Delete it later, deliberately, in its own commit.
  • The half-converted state costs two implementations, a process per call and a temporary wire format. Pay it for the scripts that need it.

Chapter 28.6 collects the translation hazards behind this chapter's four disagreements, and more. Each one is a piece of Perl behaviour that a line-by-line Python translation silently changes, with no error at all.

Where this fits

Part of the Perl curriculum.