Misconception: "Python Replaced All the Perl Code"

Part II · Module 16 · Chapter 16.7 · Misconception: "Python Replaced All the Perl Code"

1. Migrations Stall; They Do Not Fail

Nobody decides to stop a migration. What happens is smaller and more reasonable:

Somebody needs a change to the table formatter. It is forty lines, it reads one file and writes another, and rewriting it in Python takes an afternoon including tests. Done, and it was clearly the right call.

Six months later the same person needs a change to the regression driver. It is the script Chapter 16.2 measured — twenty-four lines, five environment variables, three external tools, no tests. Rewriting it means a week of work and a tapeout-week risk. They make the change in Perl, which takes twenty minutes.

Both decisions were correct. Repeat them for two years and you get a flow that is mostly Python, with Perl in exactly the places that matter most — and a team that genuinely believes the migration is nearly finished, because by file count it is.

2. What Actually Decides the Order

The scoring here is the Chapter 16.2 analyser applied across several scripts instead of to one: count what a replacement would have to reproduce.

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;
use File::Path qw(make_path);

# ---- four SAMPLE scripts spanning the real spectrum (not real artefacts)
my $dir = tempdir(CLEANUP => 1);
my $bin = File::Spec->catdir($dir, "bin");
make_path($bin);
my %script = (
"fmt_table.pl" => <<'E',
#!/usr/bin/perl
use strict; use warnings;
while (my $l = <>) { chomp $l; my @f = split /\s+/, $l; printf "%-10s %8s\n", $f[0], $f[1] }
E
"cmp_pinlist.pl" => <<'E',
#!/usr/bin/perl
use strict; use warnings;
my ($a, $b) = @ARGV;
open my $x, "<", $a or die "cannot read $a: $!\n";
open my $y, "<", $b or die "cannot read $b: $!\n";
my %A; while (<$x>) { chomp; $A{$_} = 1 }
my %B; while (<$y>) { chomp; $B{$_} = 1 }
for (sort keys %A) { print "only in A: $_\n" unless $B{$_} }
exit(0);
E
"parse_sim_log" => <<'E',
#!/usr/bin/perl
use strict; use warnings;
my $log = shift or die;
my $sev = $ENV{SEV_LEVEL} || "UVM_ERROR";
open my $f, "<", $log or die;
my $n = 0; while (<$f>) { $n++ if /^\Q$sev\E/ }
exit($n ? 1 : 0);
E
"run_regress" => <<'E',
#!/usr/bin/perl
my $ROOT=$ENV{PROJ_ROOT}||"/proj/chipA"; my $S=$ENV{SCRATCH}||"/tmp";
my $T=$ENV{SIM}||"vcs"; my $SEED=$ENV{SEED}||1; my $Q=$ENV{QUEUE}||"normal";
open(L,"<$ENV{LIST}") || die; my @t=<L>; close L;
foreach my $t (@t) { chomp $t; system("$T -f $ROOT/rtl/files.f +seed=$SEED $t");
  if ($?>>8) { warn "FAIL\n" } system("bsub -q $Q true"); }
system("rm -rf $S/*") unless $ENV{KEEP}; exit(0);
E
);
for my $n (sort keys %script) {
    open my $w, ">", File::Spec->catfile($bin, $n) or die "cannot write: $!\n";
    print $w $script{$n};
    close $w;
}

# ---- score each by what a replacement must reproduce
my @row;
for my $n (sort keys %script) {
    my $src = $script{$n};
    my $loc = grep { /\S/ && !/^\s*#/ } split /\n/, $src;
    my (%env, %cmd, %path);
    $env{$1}  = 1 while $src =~ /\$ENV\{(\w+)\}/g;
    $cmd{$1}  = 1 while $src =~ /(?:system|exec)\s*\(?\s*["']?\$?(\w+)/g;
    $path{$1} = 1 while $src =~ m{["'](/[\w/]{4,})}g;
    # weights: an env var or an external tool is 3 points of unknown,
    # a hard-coded path is 2. Lines of code are NOT scored at all.
    my $score = keys(%env) * 3 + keys(%cmd) * 3 + keys(%path) * 2;
    push @row, [$n, $loc, scalar keys %env, scalar keys %cmd, scalar keys %path, $score];
}

printf "%-16s %4s %4s %5s %6s %6s  %s\n",
    "SCRIPT", "loc", "env", "cmds", "paths", "SCORE", "order a team would do them";
my $order = 0;
# tie-break ALPHABETICALLY, not by line count -- using loc as a tie-break
# would smuggle the very assumption this table exists to test
for my $r (sort { $a->[5] <=> $b->[5] || $a->[0] cmp $b->[0] } @row) {
    $order++;
    my $note = $order == 1           ? "  <- goes first"
             : $order == scalar @row ? "  <- never gets done"
             :                          "";
    printf "%-16s %4d %4d %5d %6d %6d  %d%s\n", @$r, $order, $note;
}

# does line count predict the order?
my @by_score = map { $_->[0] } sort { $a->[5] <=> $b->[5] || $a->[0] cmp $b->[0] } @row;
my @by_loc   = map { $_->[0] } sort { $a->[1] <=> $b->[1] } @row;
printf "\n  order by interface score : %s\n", join(" -> ", @by_score);
printf "  order by line count      : %s\n", join(" -> ", @by_loc);
printf "  same order? %s\n", ("@by_score" eq "@by_loc" ? "yes" : "NO - line count does not predict it");

SCRIPT            loc  env  cmds  paths  SCORE  order a team would do them
cmp_pinlist.pl      8    0     0      0      0  1  <- goes first
fmt_table.pl        2    0     0      0      0  2
parse_sim_log       6    1     0      0      3  3
run_regress         6    7     3      1     32  4  <- never gets done

  order by interface score : cmp_pinlist.pl -> fmt_table.pl -> parse_sim_log -> run_regress
  order by line count      : fmt_table.pl -> parse_sim_log -> run_regress -> cmp_pinlist.pl
  same order? NO - line count does not predict it

Three things in that table are worth reading carefully.

The longest script goes first. cmp_pinlist.pl has the most lines of the four and an interface score of zero — two filenames in, text out, nothing else. It is the safest rewrite on the list precisely because there is nothing to find out.

Two scripts with the same line count are twenty-nine points apart. parse_sim_log and run_regress both have 6 lines of code. One scores 3; the other scores 32. The difference is five environment variables, three external tools and a hard-coded path — none of which shows up in a line count.

The two orderings disagree. Sorting by interface score and sorting by line count give different sequences, which is the measurable version of the chapter's claim.

3. The Four Categories That Never Go

From the shape of that table, the scripts that survive a migration indefinitely fall into four groups.

The regression driver. Top of the score for the reasons measured. Every test's verdict passes through it, its interface is environment variables nobody documented, and a subtle difference shows up as a regression that wrongly passes — Chapter 14.1's third kind of failure at project scale.

Anything that talks to a queueing system. bsub, qsub, or whatever the site uses. The script encodes years of accumulated knowledge about queue names, resource strings and retry behaviour, almost none of it written down, and a wrong guess wastes a night of farm time rather than failing visibly.

Report parsers for tool versions you still support. A parser that handles the current tool and the two older versions still in use on older projects. The Python rewrite handles the current one; the old projects keep calling the Perl. Now you have both, forever.

Anything whose output another team consumes. The moment a script's output is an interface to somebody else — a file format, a column order, a specific exit code — changing it requires their agreement, and that is a scheduling problem rather than a coding one.

Notice that none of these four is about Perl. They are about coupling. The same four would resist a Python-to-Python rewrite.

4. What a Stalled Migration Actually Costs

This is where the misconception does damage, because the cost is not "some old code remains".

You maintain two flows. Two ways to read a config, two logging conventions, two error-reporting styles, two sets of helper libraries. Every new engineer learns both.

The boundary is where bugs live. A Python script calling a Perl script and reading its exit status has to agree with it about what the status means — and Chapter 15.4 measured how easily that goes wrong, including die producing a status of 2 or 13 instead of 255, and exit 256 arriving as 0. Those boundaries are now part of the flow.

The remaining Perl is the hardest Perl. The measurement makes this concrete: what is left is what scored highest. The team's Perl knowledge decays exactly as the surviving Perl gets harder.

That last point is the practical argument of this whole module, and it is the opposite of what the misconception implies. A mostly-finished migration increases the value of being able to read Perl, because the easy examples are gone and what remains is the regression driver.

The way out is not a bigger migration push. It is Chapter 16.2's strangling: documented interfaces, a test harness that records the current behaviour, and one piece at a time — with the flow working at every commit. Module 28 is the full treatment.

5. What To Remember

• Migrations stall, they do not fail. Every individual decision is correct; the sequence produces the stall.
• By file count a migration looks nearly done while the important scripts are untouched.
• What decides the order is how little you need to know, not how much the script matters.
• Measured: the longest of four scripts goes first, because its interface score is zero.
• Measured: two scripts with the same 6 lines of code score 3 and 32.
• Measured: ordering by interface score and by line count disagree — line count does not predict difficulty.
• The scoring weights are a judgement, and the middle of the ordering moves if you change them; the two ends do not.
• Lines of code, regex count and cleverness are deliberately not scored. None of them is the risk.
• Four categories survive indefinitely: the regression driver, anything touching the queueing system, parsers for tool versions you still support, and anything another team consumes.
• None of the four is about Perl — they are about coupling, and would resist any rewrite.
• The real cost is two flows, not leftover code: two config styles, two logging conventions, two of everything.
• The boundary between them is where bugs live — exit-status agreement especially, per Chapter 15.4.
• What remains is the hardest Perl, so a mostly-finished migration makes reading Perl more valuable.
• The way out is strangling, not a bigger push — document, record the behaviour, replace one piece at a time.

Chapter 16.8 takes the misconception that is a technical claim rather than an organisational one, and therefore the one that can be settled outright. The belief is that regex-heavy work needs Perl. The regex engines can simply be compared, feature by feature, and they are.