When Engineers Encounter Perl Today

Part II · Module 16 · Chapter 16.3 · When Engineers Encounter Perl Today

1. The Three Places

The regression system. The thing that runs every test every night and tells you in the morning what broke. It builds a list of tests, launches the simulator once per test, reads each log, decides pass or fail, and writes a summary. It is usually the oldest code in the flow and the most central, because it was the first thing anybody automated. Chapter 16.2's sample driver is this.

The in-house utilities. A bin directory with twenty or thirty small programs: compare two netlists, pull the worst paths out of a timing report, convert a pin list between two formats, rename signals across a tree. Each one does a single job, each is between twenty and three hundred lines, and collectively they encode a great deal of what the team knows.

The report-processing steps. The glue between two tools. Synthesis writes a report; something has to turn it into the table that goes in the review. STA writes slack per path; something has to group it by clock domain and flag the violations. These rarely have names anyone remembers and they run inside a Makefile rule.

What the three have in common is that they are all text in, decision out, text out — the shape Chapter 16.1 started from. That is why they are the parts that are in Perl.

2. Finding It: The Extension Trap

The obvious way to find the Perl in a tree is wrong, and it is worth measuring how wrong.

use strict;
use warnings;
use File::Find;
use File::Temp qw(tempdir);
use File::Spec;

# ---- build a SAMPLE flow tree (not taken from any real project)
my $root = File::Spec->catdir(tempdir(CLEANUP => 1), "flow");
for my $d (qw(bin lib regress rtl scripts reports)) {
    my $p = File::Spec->catdir($root, $d);
    File::Path::make_path($p);
}
my %file = (
  # three real-world shebang spellings, and two with NO Perl extension
  "bin/run_regress"      => "#!/usr/bin/perl\nuse strict; print \"driver\\n\";\n",
  "bin/mk_filelist.pl"   => "#!/usr/bin/env perl\nuse strict; print \"filelist\\n\";\n",
  "scripts/scrape_sta"   => "#! /usr/bin/perl -w\nprint \"sta\\n\";\n",
  "lib/Report.pm"        => "package Report; 1;\n",
  "scripts/summarise.py" => "#!/usr/bin/env python3\nprint('summary')\n",
  "regress/nightly.sh"   => "#!/bin/sh\nperl \$FLOW/bin/run_regress \"\$\@\"\npython3 \$FLOW/scripts/summarise.py\n",
  "rtl/files.f"          => "-f ./sub.f\n./alu.v\n",
  "reports/setup.rpt"    => "U1/Q 0.44 setup\n",
  "Makefile"             => "regress:\n\tperl bin/run_regress lists/nightly.list\n"
                          . "\t\$(PERL) bin/mk_filelist.pl > rtl/files.f\n",
);
for my $rel (sort keys %file) {
    my $p = File::Spec->catfile($root, split m{/}, $rel);
    open my $w, ">", $p or die "cannot write $p: $!\n";
    print $w $file{$rel};
    close $w;
}

# ---- now look for Perl, two different ways
my (@by_ext, @by_shebang, @invoked);
find(sub {
    return unless -f $_;
    (my $rel = $File::Find::name) =~ s{^\Q$root\E/?}{};

    if (/\.(?:pl|pm|t)$/) { push @by_ext, $rel; return }

    # no Perl extension -- the first line is the only evidence
    open my $fh, "<", $_ or return;
    my $first = <$fh> // "";
    close $fh;
    push @by_shebang, $rel
        if $first =~ m{^\#!\s*\S*/perl\b} || $first =~ m{^\#!\s*\S*/env\s+perl\b};
}, $root);

# ---- and ask which ones something actually CALLS
find(sub {
    return unless -f $_ && (/\.(?:sh|mk)$/ || $_ eq "Makefile");
    (my $rel = $File::Find::name) =~ s{^\Q$root\E/?}{};
    open my $fh, "<", $_ or return;
    while (my $l = <$fh>) {
        while ($l =~ /(?:\bperl|\$\(PERL\)|\$PERL)\s+(\S+)/g) { push @invoked, "$rel -> $1" }
    }
    close $fh;
}, $root);

printf "found by .pl / .pm extension : %d\n", scalar @by_ext;
print  "   $_\n" for sort @by_ext;
printf "found ONLY by the shebang    : %d\n", scalar @by_shebang;
print  "   $_\n" for sort @by_shebang;
printf "invoked from a Makefile / sh : %d\n", scalar @invoked;
print  "   $_\n" for sort @invoked;

my $missed = @by_shebang;
my $total  = @by_ext + @by_shebang;
printf "\n   searching for '*.pl' would have missed %d of %d Perl files\n", $missed, $total;

BEGIN { require File::Path }

found by .pl / .pm extension : 2
   bin/mk_filelist.pl
   lib/Report.pm
found ONLY by the shebang    : 2
   bin/run_regress
   scripts/scrape_sta
invoked from a Makefile / sh : 3
   Makefile -> bin/mk_filelist.pl
   Makefile -> bin/run_regress
   regress/nightly.sh -> $FLOW/bin/run_regress

   searching for '*.pl' would have missed 2 of 4 Perl files

Half the Perl in the tree has no Perl extension. find . -name '*.pl' finds mk_filelist.pl and misses bin/run_regress and scripts/scrape_sta entirely.

And the ones it misses are the important ones. Look at the invocation list: the Makefile calls bin/run_regress — one of the two with no extension. The executables that run every night live in bin/ and are named like commands, because that is what they are. A program you type is called run_regress, not run_regress.pl.

So the rule is: find Perl by its shebang, not its filename. Three spellings all appear in real trees and all three must be recognised:

#!/usr/bin/perl
#!/usr/bin/env perl
#! /usr/bin/perl -w

Note the space after #! in the third and the trailing -w. A pattern that assumes neither will miss files.

3. Which One You Will Be Handed First

In practice the order is predictable, and it is the reverse of the order above.

First: a report-processing step. Something in a Makefile produces a table that is now wrong because the tool version changed and a column moved. This is the best possible introduction: the script is short, the input and output are both files you can look at, and you can run it as many times as you like without consequences.

Second: a utility. You need the worst paths grouped a way the existing script does not group them. Now you are modifying somebody's code rather than reading it, but the blast radius is still one report.

Third, and much later: the regression system. By the time anybody asks you to change this, you are expected to know the flow. Chapter 16.2's analysis is the preparation, and the honest answer in your first months is usually "I can read it, and I would rather someone else changed it."

What this ordering means for study: reading fluency matters far more than writing fluency. You will read Perl in your first month and write it, if ever, much later. Part I was deliberately a reading course as much as a writing one — the regex module, the references module and the chapter on reading OO Perl exist for exactly this.

4. Reading an Unfamiliar Script in Ten Minutes

A repeatable procedure, in the order that gets you most of the answer fastest.

1. Read the last few lines first. What does it print, and what does it exit with? That tells you what the script is for better than the top does. Chapter 15.4's point: the exit status is the script's actual output as far as the flow is concerned.

2. List the inputs. @ARGV at the top, every $ENV{...}, every file it opens. Chapter 16.2's analyser does this mechanically, and it is the same list.

3. Find the main loop. Almost every flow script has exactly one: while (<$fh>), for my $t (@tests), or an implicit loop under -n. Everything else is setup and reporting.

4. Inside the loop, find the one regex that matters. A report scraper is usually one pattern that recognises the interesting line, and everything else is bookkeeping. Read that pattern carefully — it is the specification of the input format, and often the only such specification that exists.

5. Check what it does with failure. Does it die, warn, or ignore? Does it check $? after system? This is where you will find the behaviour nobody documented, and it is what Chapter 14.3 and 15.10 prepared you to see.

6. Only then read it top to bottom. By now you know what you are looking at, and the parts that looked cryptic are mostly Part I idioms: $_, @F, $1, a slice, a hash of arrays.

One concrete tip for step 6. If a line genuinely will not resolve, ask Perl rather than guessing:

perl -MO=Deparse -e 'the confusing line here'

Chapter 15.7 used that to settle what -n and -p expand to. It works just as well on a dense expression, and it will show you precedence you had read wrongly.

5. What To Remember

• Three places: the regression system, the bin directory of utilities, and the report-processing steps inside Makefiles.
• All three are text in, decision out, text out — which is why they are the parts in Perl.
• Measured: searching for *.pl missed 2 of 4 Perl files in the sample tree.
• The missed ones are the important ones — the Makefile called bin/run_regress, which has no extension.
• An executable is named like a command, not like a source file.
• Find Perl by its shebang, and recognise all three spellings including #! /usr/bin/perl -w with a space and a flag.
• Follow what the Makefiles call, since a bin directory accumulates dead scripts.
• File::Find does not follow symlinks by default — usually right, because shared bin directories are linked in many places.
• The order you meet them is the reverse of their importance: a report step first, a utility second, the regression system much later.
• Reading fluency matters more than writing fluency in the first months.
• To read a script fast: last lines first, then the inputs, then the one main loop, then the one regex that matters, then the failure handling, and only then top to bottom.
• The one regex is usually the only specification of the input format that exists.
• perl -MO=Deparse settles a confusing line instead of guessing at precedence.

Chapter 16.4 is the chapter this module exists to make possible. With the footprint understood and the places named, the question becomes which language a given task belongs in — and it is answerable on evidence rather than preference.