Pipes: Reading and Writing Another Process

Part I · Module 15 · Chapter 15.11 · Pipes: Reading and Writing Another Process

1. Reading From a Command

The three-argument open from Chapter 13.1 takes a mode of -|, meaning "run this and let me read what it prints".

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
my $gen = File::Spec->catfile($dir, "gen.pl");
open my $w, ">", $gen or die "cannot write: $!\n";
print $w 'for my $i (1 .. 20) { print "row $i\n" }' . "\n";
close $w;

# LIST form: the program and its arguments as separate items, no shell
open my $fh, "-|", $^X, $gen or die "cannot run: $!\n";
my @kept;
while (my $line = <$fh>) {
    chomp $line;
    push @kept, $line if $line =~ /[02468]$/;     # even rows only
}
close $fh;

printf "read the output as a filehandle, kept %d of 20 rows\n", scalar @kept;
print "   first four: ", join(", ", @kept[0 .. 3]), "\n";
printf "   close status: \$? >> 8 = %d\n", $? >> 8;

read the output as a filehandle, kept 10 of 20 rows
   first four: row 2, row 4, row 6, row 8
   close status: $? >> 8 = 0

The handle behaves like any other filehandle. <$fh> in a while loop reads a line at a time, $. counts lines, eof works. Everything Module 13 taught about reading files applies unchanged — which is the real appeal: there is nothing new to learn.

Note the list form: open my $fh, "-|", $^X, $gen. The program and each argument are separate items, so no shell is involved. That is Chapter 15.10's measured safety point, and here it is available for a capturing read, which backticks cannot do.

2. Why This Matters: The Streaming Difference

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
my $gen = File::Spec->catfile($dir, "gen.pl");
open my $w, ">", $gen or die "cannot write: $!\n";
print $w '$| = 1; for my $i (1 .. 200000) { print "row $i\n" }' . "\n";
close $w;

# A: a pipe -- read what you need and stop
open my $fh, "-|", $^X, $gen or die "cannot run: $!\n";
my ($n, @first) = (0);
while (my $line = <$fh>) {
    $n++;
    push @first, $line if $n <= 3;
    last if $n >= 3;
}
close $fh;
chomp @first;
printf "A. pipe       : read %d line(s) and stopped -> %s\n", $n, join(" | ", @first);

# B: backticks -- everything into memory first
my $all = `$^X '$gen'`;
my $lines = () = $all =~ /\n/g;
printf "B. backticks  : captured %d bytes (%d lines) before we could look at any\n",
    length $all, $lines;

printf "\nthe pipe looked at %d lines; the capture held %d. same command.\n", $n, $lines;

A. pipe       : read 3 line(s) and stopped -> row 1 | row 2 | row 3
B. backticks  : captured 2088895 bytes (200000 lines) before we could look at any

the pipe looked at 3 lines; the capture held 200000. same command.

The same command, two shapes. The pipe looked at three lines. The backticks pulled over two million bytes into a single Perl string before the script could examine one of them.

Three consequences, in order of how often they bite:

Memory. A two-gigabyte log captured with backticks needs two gigabytes of memory, plus whatever Perl's string overhead adds. The pipe needs one line's worth. This is the same argument Chapter 13.3 made about slurping a file versus reading it line by line, and it is the reason that chapter exists.

You can stop early. The pipe above read three lines and closed. Checking "does this report contain any hold violations" does not require reading the whole report, and with a pipe it does not.

Work starts immediately. With a pipe, your first line is available as soon as the command produces it. With backticks, nothing happens until the command has finished entirely. For a tool that takes ten minutes, that is ten minutes of your script doing nothing, and no possibility of printing progress.

3. Writing To a Command

The mirror image is |-: run a program and let you write to its standard input.

use strict;
use warnings;

# feed unsorted lines to sort(1) and let it do the work
open my $out, "|-", "sort" or die "cannot run sort: $!\n";
print $out "$_\n" for qw(delta alpha charlie bravo);
my $ok = close $out;          # close WAITS for sort to finish
printf "\nsort: close returned %s, status %d\n", ($ok ? "true" : "false"), $? >> 8;

# a more realistic use: write a summary straight into a compressor,
# so the uncompressed version never exists on disk
print "\nthe same shape, writing into a filter:\n";
open my $wc, "|-", "wc", "-l" or die "cannot run wc: $!\n";
print $wc "line $_\n" for 1 .. 7;
close $wc;
printf "wc -l reported the count above; status %d\n", $? >> 8;

alpha
bravo
charlie
delta

sort: close returned true, status 0

the same shape, writing into a filter:
       7
wc -l reported the count above; status 0

sort received four words and printed them in order; wc -l counted seven lines. In both cases the child's output went to your standard output, because that is what it inherited.

Two things to watch:

close waits. Closing a write pipe sends end-of-file to the child and then blocks until it exits. That is what you want — until you forget to close, in which case the child may still be running when your script ends and its output can appear after everything else.

Order with your own output is not guaranteed. The child writes to the same place you do, independently, so its lines can land before or after yours. Chapter 14.8 measured exactly this scrambling with stdout and stderr. If the order matters, capture instead of letting it write through.

The genuinely useful pattern here is writing into a compressor or a formatter so an intermediate file never exists:

open my $gz, "|-", "gzip", "-c" or die "cannot run gzip: $!\n";
# ... print thousands of rows ...
close $gz or warn "gzip failed: status " . ($? >> 8) . "\n";

4. The Status Arrives At close, Not At open

This is the part that catches everyone, and the answer has a wrinkle worth measuring.

use strict;
use warnings;

my @cases = (
    [ "list form, missing command", [ "nosuchtool-11111" ] ],
    [ "list form, with an arg",     [ "nosuchtool-11111", "x" ] ],
    [ "STRING form, missing",       [ "nosuchtool-11111 2>/dev/null" ] ],
    [ "list form, exits 5",         [ $^X, "-e", 'print "x\n"; exit 5' ] ],
    [ "list form, exits 0",         [ $^X, "-e", 'print "x\n"' ] ],
);

printf "%-26s %-7s %-6s %-7s %s\n", "CASE", "open", "lines", "close", '$? >> 8';
for my $c (@cases) {
    my ($label, $cmd) = @$c;

    my $fh;
    my $opened = open($fh, "-|", @$cmd);

    my ($lines, $closed, $status) = (0, "-", "-");
    if ($opened) {
        my @got = <$fh>;
        $lines  = scalar @got;
        $closed = close($fh) ? "true" : "FALSE";
        $status = $? >> 8;
    }
    printf "%-26s %-7s %-6s %-7s %s\n",
        $label, ($opened ? "true" : "FALSE"), $lines, $closed, $status;
}

CASE                       open    lines  close   $? >> 8
list form, missing command FALSE   0      -       -
list form, with an arg     FALSE   0      -       -
STRING form, missing       true    0      FALSE   127
list form, exits 5         true    1      FALSE   5
list form, exits 0         true    1      true    0

Three different things are happening in those five rows.

With the list form, a missing command makes open itself return false. Perl runs the program directly, so the failed exec is reported back and you find out immediately. Both list rows behave this way.

With the string form, open returns true even though the command does not exist — because what Perl started was a shell, and the shell started perfectly well. The failure only appears at close, as status 127, which is the shell's "command not found". That is the same measurement Chapter 14.3 made about system with one string, arriving here from a different direction.

For a command that exists and then fails, open is always true. Nothing is known at that point: the child was created, and whether the work succeeded is a question about the future. close returned FALSE and $? >> 8 held 5.

So the correct shape for a read pipe checks in two places, and the second one is the one almost everybody omits:

open my $fh, "-|", $prog, @args
    or die "cannot run $prog: $!\n";                      # list form: catches missing

while (my $line = <$fh>) { ... }

close $fh
    or die "$prog failed (status " . ($? >> 8) . ")\n";   # catches everything else

Without that second check, a tool that dies half way through gives you a partial read and a script that is entirely happy about it — Chapter 14.1's third kind of failure, one more time, and Chapter 15.3's point about a plausible partial result.

And note that this is another reason to prefer the list form, on top of Chapter 15.10's quoting argument: it tells you about a missing tool at the point of the call rather than at the point of cleanup.

5. What Pipes Cannot Do

Two limits worth knowing before you design around them.

A pipe is one-way. You cannot open a handle that both writes to and reads from the same command with open. The obvious attempt deadlocks: you write, the child fills its output buffer, it blocks waiting for you to read, and you are still blocked writing. For genuine two-way conversation use IPC::Open2 or IPC::Open3 — both core — and read their documentation on deadlock first, because the problem does not go away, it just becomes manageable.

Most of the time two-way is not needed. Writing the input to a temporary file and reading the output from a pipe is simpler, slower, and correct.

Standard error is not included. A read pipe gives you the child's standard output only, exactly as backticks do. To capture both:

open my $fh, "-|", "sh", "-c", 'mytool 2>&1' or die $!;   # a shell, deliberately

That reintroduces the shell, so quote carefully — or use IPC::Open3, which gives you all three handles without one.

And one practical note that follows from Chapter 14.8: a child writing to a pipe is not writing to a terminal, so its output is block-buffered. A tool that prints progress lines may appear to produce nothing for a long time and then everything at once. That is the tool's buffering, not your pipe, and you usually cannot fix it from this side — which is why the generator in section 2 sets $| = 1 on its own output.

6. What To Remember

• open my $fh, "-|", $prog, @args runs a command and gives you its output as a filehandle.
• It behaves like any filehandle — while (<$fh>), $., eof all work.
• Use the list form: the program and each argument separate, so no shell is involved.
• Avoid the two-argument "cmd |" form — it hands the whole string to a shell.
• Measured: the pipe read 3 lines and stopped; backticks captured 2,088,895 bytes from the same command.
• Three wins: memory, stopping early, and starting immediately — the same argument as reading a file line by line.
• open my $out, "|-", $prog, @args writes to a command's standard input.
• close on a write pipe waits for the child to finish.
• The child's output interleaves unpredictably with yours — capture it if the order matters.
• Writing into gzip -c avoids an intermediate file entirely.
• Measured: with the LIST form, open returns false for a missing command — the failed exec is reported straight back.
• Measured: with the STRING form, open returns true for a missing command, because a shell started fine; the failure shows at close as 127.
• For a command that exists and then fails, open is always true. Measured: close returned false and $? >> 8 was 5.
• So the list form also gives you earlier failure detection, on top of the quoting argument.
• Always check close, or a tool that dies half way gives you a partial read and no complaint.
• A false close after stopping early is normal — the child took SIGPIPE. Decode $? in branches.
• A pipe is one-way; two-way needs IPC::Open2/Open3 and a deadlock strategy.
• Standard error is not included — fold it in with a deliberate shell, or use IPC::Open3.
• The child's output is block-buffered because it is not writing to a terminal.

Chapter 15.12 closes the module by turning the pipe around. A script that reads standard input and writes standard output is not a program that happens to use files — it is a component that grep, sort and every other tool can be combined with.