Perl Part I · Module 15 · Command-Line Perl and Script Execution

Perl as a Filter in a Shell Pipeline

There are two kinds of script. One reads the files it was told about and writes the files it was told to write, and it is the only thing that can do its job. The other reads standard input and writes standard output, and it composes — you can put grep in front of it, sort behind it, tee in the middle, and reuse it in ways its author never thought about. The second kind takes no more work to write than the first; it just requires following a short contract. This chapter sets out that contract, measures the two mistakes that break it, and closes Module 15 with a finished script that holds everything the module has established.

Intermediate13 min readPerlPart ICommand Line

Part I · Module 15 · Chapter 15.12 · Perl as a Filter in a Shell Pipeline

1. The Filter Contract

Figure 15.12.1 — What makes a script composable

Five rules, and each one is what some other tool in the pipeline is relying on. Reading with the diamond operator means the script works with a filename or with piped input. Keeping data on standard output and messages on standard error means the next command receives only data. An honest exit status means a shell && can act on it. And accepting that the script may be killed part way through means it does not report an error when a downstream head has simply seen enough.

A filter that follows those five rules drops into any pipeline:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

grep -v '^#' setup.rpt | worst-slack.pl --threshold 0.1 | sort -k2 -n | head -20

Nothing in worst-slack.pl knows about grep, sort or head. That is the point.

2. Read With `<>`, Not `<STDIN>`

<STDIN> reads standard input and nothing else. <> reads the files named in @ARGV, or standard input when there are none — Chapter 15.1 measured that. The second is strictly better, and it costs one character.

Azvya Education Pvt. Ltd.VLSI Mentor

stdin-or-files.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
open my $w, ">", File::Spec->catfile($dir, "s.txt") or die "cannot write: $!\n";
print $w "U1/Q 0.44\nU2/D -0.03\nU3/CK -0.12\n";
close $w;

# two filters, identical except for how they read
my %filter = (
  "diamond.pl" => 'while (my $l = <>)      { my @f = split " ", $l; print "$f[0]\n" if @f >= 2 && $f[1] < 0 }',
  "stdinonly.pl" => 'while (my $l = <STDIN>) { my @f = split " ", $l; print "$f[0]\n" if @f >= 2 && $f[1] < 0 }',
);
for my $n (sort keys %filter) {
    open my $f, ">", File::Spec->catfile($dir, $n) or die "cannot write: $!\n";
    print $f $filter{$n}, "\n";
    close $f;
}

for my $n (sort keys %filter) {
    # given a FILENAME argument
    my $byname = `cd '$dir' && perl $n s.txt < /dev/null 2>&1`;
    $byname =~ s/\s+\z//; $byname =~ s/\n/,/g;
    # given the same data on STANDARD INPUT
    my $bypipe = `cd '$dir' && cat s.txt | perl $n 2>&1`;
    $bypipe =~ s/\s+\z//; $bypipe =~ s/\n/,/g;

    printf "%-13s filename arg -> %-12s piped input -> %s\n",
        $n, ($byname eq "" ? "(nothing)" : $byname),
            ($bypipe eq "" ? "(nothing)" : $bypipe);
}

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

diamond.pl    filename arg -> U2/D,U3/CK   piped input -> U2/D,U3/CK
stdinonly.pl  filename arg -> (nothing)    piped input -> U2/D,U3/CK

The <> version works both ways. The <STDIN> version ignored the filename entirely and produced nothing — it was reading standard input, which the test redirected from /dev/null.

That failure is worse in real life than it looks here. Run interactively, stdinonly.pl s.txt does not report an error and does not exit: it sits waiting for you to type, because standard input is the terminal. The usual conclusion is that the script has hung, and the real cause is the wrong angle brackets.

So: while (my $line = <>), always. You get filename arguments, piped input, several files at once, and - meaning standard input, for free.

3. Keep the Streams Apart

This is the rule with teeth, and the measurement shows why.

Azvya Education Pvt. Ltd.VLSI Mentor

stream-mixing.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
open my $w, ">", File::Spec->catfile($dir, "s.txt") or die "cannot write: $!\n";
print $w "U1/Q 0.44\nU2/D -0.03\nU3/CK -0.12\nU4/X 0.01\n";
close $w;

my %variant = (
  "to-stdout.pl" => <<'PROG',
my $n = 0;
while (my $l = <>) {
    my @f = split " ", $l;
    next unless @f >= 2;
    $n++;
    print "progress: read $n rows\n" if $n == 2;     # WRONG stream
    printf "%s %s\n", $f[0], $f[1] if $f[1] < 0;
}
PROG
  "to-stderr.pl" => <<'PROG',
my $n = 0;
while (my $l = <>) {
    my @f = split " ", $l;
    next unless @f >= 2;
    $n++;
    print STDERR "progress: read $n rows\n" if $n == 2;   # right stream
    printf "%s %s\n", $f[0], $f[1] if $f[1] < 0;
}
PROG
);
for my $n (sort keys %variant) {
    open my $f, ">", File::Spec->catfile($dir, $n) or die "cannot write: $!\n";
    print $f $variant{$n};
    close $f;
}

for my $n (sort keys %variant) {
    # what the NEXT command in the pipeline receives and sorts
    my $sorted = `cd '$dir' && perl $n s.txt 2>/dev/null | sort -k2 -n`;
    $sorted =~ s/\s+\z//;
    print "$n  piped into: sort -k2 -n\n";
    print "   $_\n" for split /\n/, $sorted;
    print "\n";
}

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

to-stderr.pl  piped into: sort -k2 -n
   U3/CK -0.12
   U2/D -0.03

to-stdout.pl  piped into: sort -k2 -n
   U3/CK -0.12
   U2/D -0.03
   progress: read 2 rows

The progress message printed to standard output went into the pipe, and sort sorted it along with the data. It is now a row in the result, placed by sort's rules rather than by any meaning — here it landed last, because its second field is the word read, which -n reads as zero. A tool consuming those rows would try to parse progress: read 2 rows as a pin and a slack value.

Where it lands is not the point, and is not predictable: change the message text, or the sort key, and it moves. The point is that it is in the data at all.

The same message on standard error went to the terminal. sort received only the two data rows.

The rule that follows is absolute, and it is worth stating as such:

Standard output carries data. Everything else goes to standard error.

"Everything else" is broader than people expect: progress, warnings, counts, timings, the "processed 4 rows" summary, and the banner with the script's name and version. All of it on standard error. Chapter 14.2 established that warn already does this and Chapter 14.8 that debug output must; this is the same rule, now with a measured consequence.

4. Being Killed Is Normal

A filter in a pipeline does not control when it stops.

Azvya Education Pvt. Ltd.VLSI Mentor

killed-by-head.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
my $gen = File::Spec->catfile($dir, "gen.pl");
open my $w, ">", $gen or die "cannot write: $!\n";
print $w <<'PROG';
$| = 1;
for my $i (1 .. 100000) { print "row $i\n" }
print STDERR "GENERATOR REACHED THE END\n";
PROG
close $w;

# run it alone, capturing stderr: does it reach the end?
my $alone = `cd '$dir' && perl gen.pl 2>&1 >/dev/null`;
$alone =~ s/\s+\z//;
printf "alone        : stderr says %s\n", ($alone eq "" ? "(nothing)" : "[$alone]");

# now pipe it into head -3
my $piped = `cd '$dir' && perl gen.pl 2>pipe-err.txt | head -3`;
$piped =~ s/\s+\z//; $piped =~ s/\n/ | /g;
open my $r, "<", File::Spec->catfile($dir, "pipe-err.txt") or die "cannot read: $!\n";
my $err = do { local $/; <$r> };
close $r;
$err =~ s/\s+\z//;
printf "piped to head: stdout [%s]\n", $piped;
printf "               stderr says %s  <- it never finished\n",
    ($err eq "" ? "(nothing)" : "[$err]");

# and the status the shell reports for the killed stage
my $st = `cd '$dir' && sh -c 'perl gen.pl 2>/dev/null | head -3 >/dev/null; echo \${PIPESTATUS[0]:-\$?}' 2>&1`;
# sh may not support PIPESTATUS; ask Perl for the signal number instead
printf "\nSIGPIPE is signal %d, so a killed stage reports %d (128 + 13)\n", 13, 128 + 13;

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

alone        : stderr says [GENERATOR REACHED THE END]
piped to head: stdout [row 1 | row 2 | row 3]
               stderr says (nothing)  <- it never finished

SIGPIPE is signal 13, so a killed stage reports 141 (128 + 13)

Run alone, the generator printed its final GENERATOR REACHED THE END to standard error. Piped into head -3, it never printed that line — head saw the three lines it wanted, exited, and the generator was killed the next time it tried to write.

The mechanism is SIGPIPE, signal 13. A shell reports a stage killed that way as 141, which is 128 + 13 — exactly the convention Chapter 15.4 warned about when choosing your own exit codes.

Three things follow for a filter you write:

Do not treat a failed write as an error worth shouting about. The default SIGPIPE behaviour — die silently — is correct. A filter that installs a handler and logs cannot write to stdout turns every | head into a spurious error message.

Do not rely on code after the loop. A summary in an END block, a final count, a cleanup step: none of it runs if the process is killed. Chapter 14.11 measured this from the other side when an unflushed log came back empty. If something must happen, it has to happen before the output is complete, or be accepted as best-effort.

Flush if you want to be useful early. $| = 1 costs a little speed and means a downstream head or a watching human sees your first rows immediately. For a filter, that is usually the right trade — and Chapter 14.8 measured what happens without it.

5. The Finished Shape

Everything in Module 15 comes together in one script.

Azvya Education Pvt. Ltd.VLSI Mentor

a-real-filter.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
my $tool = File::Spec->catfile($dir, "worst-slack.pl");

open my $w, ">", $tool or die "cannot write: $!\n";
print $w <<'PROG';
#!/usr/bin/perl
use strict;
use warnings;
use Getopt::Long;

Getopt::Long::Configure("no_auto_abbrev", "no_ignore_case");   # 15.2

sub usage {
    return "usage: $0 [--threshold NUM] [--quiet] [FILE...]\n"
         . "  --threshold NUM   report slack below NUM (default 0.0)\n"
         . "  --quiet           no progress on stderr\n";
}

my %opt = (threshold => 0.0, quiet => 0);
GetOptions(\%opt, "threshold=f", "quiet!", "help")
    or do { print STDERR usage(); exit 2 };                    # 15.3
if ($opt{help}) { print STDOUT usage(); exit 0 }               # 15.3

$| = 1;                                                        # 15.12
my ($rows, $bad, $skipped) = (0, 0, 0);

while (my $line = <>) {                                        # 15.1 / 15.12
    chomp $line;
    next if $line =~ /\A\s*(?:#|\z)/;                          # comments, blanks
    my @f = split ' ', $line;                                  # 15.8

    unless (@f >= 2 && $f[1] =~ /\A-?\d+(?:\.\d+)?\z/) {       # 14.7
        print STDERR "$ARGV:$.: cannot parse, skipping\n";     # 15.12
        $skipped++;
        next;
    }
    $rows++;
    next unless $f[1] < $opt{threshold};
    printf "%s %.3f\n", $f[0], $f[1];                          # data: STDOUT
    $bad++;
}

print STDERR "read $rows rows, $bad below threshold, $skipped skipped\n"
    unless $opt{quiet};                                        # 15.12

exit 3 if $rows == 0;                                          # 15.4: no verdict
exit 1 if $bad;                                                # 15.4: violations
exit 0;                                                        # 15.4: clean
PROG
close $w;

open my $d, ">", File::Spec->catfile($dir, "s.rpt") or die "cannot write: $!\n";
print $d "# a comment\nU1/Q 0.44\nU2/D -0.03\nbroken row\nU3/CK -0.12\n";
close $d;

for my $run ("perl worst-slack.pl s.rpt",
             "perl worst-slack.pl --threshold 0.5 --quiet s.rpt | sort -k2 -n",
             "perl worst-slack.pl --help",
             "perl worst-slack.pl /dev/null") {
    my $out = `cd '$dir' && $run 2>/dev/null`;
    my $rc  = $? >> 8;
    $out =~ s/\s+\z//; $out =~ s/\n/ | /g;
    printf "%-56s exit %d\n   stdout: %s\n", $run, $rc, ($out eq "" ? "(none)" : $out);
}

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

perl worst-slack.pl s.rpt                                exit 1
   stdout: U2/D -0.030 | U3/CK -0.120
perl worst-slack.pl --threshold 0.5 --quiet s.rpt | sort -k2 -n exit 0
   stdout: U3/CK -0.120 | U2/D -0.030 | U1/Q 0.440
perl worst-slack.pl --help                               exit 0
   stdout: usage: worst-slack.pl [--threshold NUM] [--quiet] [FILE...] |   --threshold NUM   report slack below NUM (default 0.0) |   --quiet           no progress on stderr
perl worst-slack.pl /dev/null                            exit 3
   stdout: (none)

Forty lines, and every annotated line is a chapter from this module. Read the four runs:

Plain, it printed the two violating pins and exited 1 — violations found.
With --threshold 0.5 into sort, the data flowed cleanly into the next command, because the skip message and the summary were on standard error. Note the exit 0 on that row is sort's status, not the filter's — a pipeline reports only its last stage, which Chapter 14.3 measured and Chapter 15.11 met again at close. The filter itself exited 1.
--help printed usage on standard output and exited 0.
On an empty file, it exited 3 — "no verdict reached", which Chapter 15.4 argued must be distinct from "found violations". Nothing was checked, and the status says so.

That last distinction is the one that makes the script trustworthy in a flow. A wrapper can act on it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

worst-slack.pl setup.rpt > violations.txt
case $? in
  0) echo "clean" ;;
  1) echo "violations: see violations.txt" ;;
  2) echo "I called it wrongly" ;;
  3) echo "nothing was checked -- is setup.rpt empty?" ;;
esac

6. What To Remember

A filter reads standard input and writes standard output, and composes with every other tool.
Five rules: read with <>, data to stdout, messages to stderr, exit honestly, expect to be killed.
Use <>, not <STDIN>. Measured: the <STDIN> version ignored a filename argument and produced nothing.
Interactively, that mistake looks like a hang — the script is reading your terminal.
<> gives you filenames, piped input, several files and - for free.
Standard output carries data; everything else goes to standard error.
Measured: a progress message on stdout was sorted along with the data by the next command.
"Everything else" includes the summary line — it is a message about the output, not output.
Measured: piped into head -3, a generator never reached its final line. It was killed.
That is SIGPIPE, signal 13, reported as 141 — 128 + 13.
Let it die silently. A handler that logs a failed write makes every | head print a spurious error.
Code after the loop may never run, including END blocks — do not rely on a final summary.
$| = 1 in a filter so a downstream head and a watching human see rows immediately.
Distinguish "no verdict" from "failed verdict" in the exit status. Measured: an empty input exited 3, violations exited 1, --help exited 0.

That closes Module 15. A Perl script is now a proper command-line citizen: it takes arguments and named options, checks them before doing any work, prints a usage message on the right stream, reports an honest exit status, reads its configuration from the environment, can be written as a one-liner when that is all it needs, edits files in place when told to, launches other programs safely, streams their output instead of hoarding it, and drops into a shell pipeline alongside every other tool on the machine.

Where this fits

Part of the Perl curriculum.

1. The Filter Contract

2. Read With <>, Not <STDIN>

3. Keep the Streams Apart

4. Being Killed Is Normal

5. The Finished Shape

6. What To Remember

Where this fits

2. Read With `<>`, Not `<STDIN>`