Running External Commands: system, backticks and exec

Part I · Module 15 · Chapter 15.10 · Running External Commands: system, backticks and exec

1. Three Ways, One Question

• system — run it and wait. You get the exit status; the program's output goes wherever yours goes.
• Backticks, or qx// — run it and wait, capturing its standard output as a string. The status is in $?.
• exec — replace your process with it. Nothing after the exec runs, ever.

2. What Each One Gives Back

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
my $helper = File::Spec->catfile($dir, "helper.pl");
open my $w, ">", $helper or die "cannot write: $!\n";
print $w <<'PROG';
print "stdout line 1\n";
print STDERR "stderr line\n";
print "stdout line 2\n";
exit (shift(@ARGV) // 0);
PROG
close $w;

print "1. system -- status returned, output passes straight through\n";
my $r = system($^X, $helper, 3);
printf "   returned %d, \$? >> 8 = %d\n\n", $r, $? >> 8;

print "2. backticks -- STDOUT captured, stderr NOT\n";
my $out = `$^X '$helper' 4 2>/dev/null`;
printf "   captured %d chars: %s\n", length $out, join(" / ", split /\n/, $out);
printf "   \$? >> 8 = %d\n\n", $? >> 8;

print "3. backticks in LIST context -- one element per line\n";
my @lines = `$^X '$helper' 2>/dev/null`;
chomp @lines;
printf "   %d element(s): %s\n\n", scalar @lines, join(" | ", map { "[$_]" } @lines);

print "4. capturing stderr as well, with 2>&1\n";
my $both = `$^X '$helper' 2>&1`;
my @b = split /\n/, $both;
printf "   %d line(s): %s\n\n", scalar @b, join(" / ", @b);

print "5. exec -- run in a child so this script survives to report\n";
my $x = File::Spec->catfile($dir, "x.pl");
open my $w2, ">", $x or die "cannot write: $!\n";
print $w2 <<'PROG';
print "   before exec\n";
exec($^X, "-e", 'print "   I am the replacement\n"; exit 9');
print "   AFTER exec -- this line never runs\n";
PROG
close $w2;
print `cd '$dir' && perl x.pl 2>&1`;
printf "   the exec'd program's status became the script's: %d\n", $? >> 8;

1. system -- status returned, output passes straight through
stdout line 1
stdout line 2
   returned 768, $? >> 8 = 3

2. backticks -- STDOUT captured, stderr NOT
   captured 28 chars: stdout line 1 / stdout line 2
   $? >> 8 = 4

3. backticks in LIST context -- one element per line
   2 element(s): [stdout line 1] | [stdout line 2]

4. capturing stderr as well, with 2>&1
   3 line(s): stderr line / stdout line 1 / stdout line 2

5. exec -- run in a child so this script survives to report
   before exec
   I am the replacement
   the exec'd program's status became the script's: 9

Five measured results, and four of them are the ones people get wrong.

system returned 768, which is 3 << 8. It returns the same packed value it puts in $?, so $? >> 8 gives the real status — Chapter 14.3 took that apart. The helper's output appeared on the terminal because system does not capture anything.

Backticks captured standard output only. The stderr line is not in the 28 characters captured. This is the trap: a tool that fails and explains itself on standard error gives you an empty capture and no explanation, and a script that only looks at the captured text reports "no output" rather than the error.

In list context, backticks give one element per line, with newlines still attached — hence the chomp. In scalar context you get one string. Both are useful; pick deliberately.

2>&1 folds standard error into the capture. Use it when you want the diagnostics, and be aware the two streams are then interleaved in an order that depends on buffering — Chapter 14.8 measured exactly that scrambling.

exec never returned. before exec printed, the replacement ran, and the line after exec did not. The replacement's exit status became the script's.

3. The List Form Is The Safe One

Both system and backticks accept either one string or a list, and the difference is not stylistic.

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);

# three filenames a real project can produce
my @names = ("plain.rpt", "Rev A.rpt", "weird;name.rpt");
for my $n (@names) {
    open my $w, ">", File::Spec->catfile($dir, $n) or die "cannot write: $!\n";
    print $w "one line\n";
    close $w;
}

my $counter = File::Spec->catfile($dir, "count.pl");
open my $c, ">", $counter or die "cannot write: $!\n";
print $c 'printf "%s: %d arg(s): %s\n", "count.pl", scalar(@ARGV), join("][",@ARGV);' . "\n";
close $c;

for my $n (@names) {
    my $path = File::Spec->catfile($dir, $n);

    # STRING form: Perl hands the whole thing to a shell
    my $s = `$^X '$counter' $path 2>&1`;
    $s =~ s/\s+\z//;
    $s =~ s/\Q$dir\E\///g;

    # LIST form: no shell at all, the name is passed through untouched
    my @cmd = ($^X, $counter, $path);
    open my $fh, "-|", @cmd or die "cannot run: $!\n";
    my $l = do { local $/; <$fh> };
    close $fh;
    $l =~ s/\s+\z//;
    $l =~ s/\Q$dir\E\///g;

    printf "filename %s\n   string form -> %s\n   list form   -> %s\n\n", "[$n]", $s, $l;
}

filename [plain.rpt]
   string form -> count.pl: 1 arg(s): plain.rpt
   list form   -> count.pl: 1 arg(s): plain.rpt

filename [Rev A.rpt]
   string form -> count.pl: 2 arg(s): Rev][A.rpt
   list form   -> count.pl: 1 arg(s): Rev A.rpt

filename [weird;name.rpt]
   string form -> count.pl: 1 arg(s): weird
sh: name.rpt: command not found
   list form   -> count.pl: 1 arg(s): weird;name.rpt

Read the Rev A.rpt row. The string form gave the counter two arguments — the shell split the name at the space. The list form gave it one, the name intact.

And weird;name.rpt is worse. In the string form the shell sees a ; and treats everything after it as a second command. With a filename that is merely odd, the result is a confusing error. With a filename an attacker controls, it is arbitrary code execution.

So the rule has no exceptions worth learning:

system($prog, @args)                         # safe: no shell
system("$prog @args")                        # unsafe: a shell parses it

open my $fh, "-|", $prog, @args              # safe
my $out = `$prog @args`                      # unsafe

Three consequences worth knowing:

Backticks have no list form. That is a genuine gap. When you need to capture output and pass awkward arguments, use the pipe form from Chapter 15.11: open my $fh, "-|", @cmd.

The string form's exit status can mislead. Chapter 14.3 measured it: the same missing program gave -1 bare, 127 with a redirect, and 0 with ; true — because a compound command reports only its last stage.

You lose shell features, which is the point. No globs, no > redirection, no pipes. If you need a glob, do it in Perl with glob(); if you need a redirect, open the file yourself. Both are clearer than embedding shell syntax in a string.

4. Checking That It Worked

Launching is the easy half. Chapter 14.3 built the decoder; here is where it belongs.

use strict;
use warnings;

sub describe {
    my ($rc) = @_;
    return "could not start it ($!)"                        if $rc == -1;
    return "killed by signal " . ($rc & 127)                if $rc & 127;
    my $code = $rc >> 8;
    return $code == 0 ? "finished cleanly" : "exited $code";
}

my @cases = (
    [ "a clean tool",    [ $^X, "-e", "exit 0" ] ],
    [ "a failing tool",  [ $^X, "-e", "exit 2" ] ],
    [ "a killed tool",   [ $^X, "-e", 'kill "TERM", $$; sleep 5' ] ],
    [ "a missing tool",  [ "this-tool-does-not-exist-98765" ] ],
);

for my $c (@cases) {
    my ($label, $cmd) = @$c;
    system(@$cmd);
    my $rc = $?;
    printf "%-16s raw %-6d %s\n", $label, $rc, describe($rc);
}

print "\nthe short form, for when you only need yes or no:\n";
my $ok = system($^X, "-e", "exit 0") == 0;
printf "   system(...) == 0  ->  %s\n", ($ok ? "worked" : "failed");

a clean tool     raw 0      finished cleanly
a failing tool   raw 512    exited 2
a killed tool    raw 15     killed by signal 15
a missing tool   raw -1     could not start it (No such file or directory)

the short form, for when you only need yes or no:
   system(...) == 0  ->  worked

Four outcomes, and only the first means the work was done. The two that catch people:

A killed tool has $? >> 8 of zero. A synthesis run stopped by a time limit, or killed when the machine ran out of memory, reports success to anything that only shifts. Chapter 14.3 measured this; it matters here because a long tool run is exactly what gets killed.

A missing tool gives -1, with the reason in $! — there was no child, so there is no status.

For the common case where you just need to stop on failure:

system($prog, @args) == 0
    or die "$prog failed (status " . ($? >> 8) . ")\n";

And a point that closes the loop back to Chapter 15.4: the status you report should be yours, not the tool's. Passing a tool's exit code straight out of your script with exit($? >> 8) looks tidy and silently inherits its scheme — including the possibility of a 0 from a killed process.

5. What To Remember

• system runs a program and gives you the status; its output goes where yours goes.
• Backticks run a program and give you its standard output; the status is in $?.
• exec replaces your process and never returns.
• Measured: system returned 768 for an exit 3 — the packed value, so use $? >> 8.
• Measured: backticks capture stdout only. A tool that explains its failure on stderr gives you an empty string and no reason.
• Add 2>&1 to capture stderr too, accepting that the interleaving is buffering-dependent.
• Backticks in list context give one element per line, newlines attached.
• Measured: nothing after exec runs, and the replacement's exit status becomes the script's.
• END blocks do not run on exec — there is no process left to run them.
• exec returns only on failure, so exec(@cmd) or die "cannot exec: $!\n";.
• Use the LIST form. Measured: Rev A.rpt became two arguments through a shell and stayed one through a list.
• A ; in a filename becomes a second command in the string form.
• Backticks have no list form — use open my $fh, "-|", @cmd when you need both capture and safety.
• The string form's status can mislead — prog; true reports success for a missing program.
• Decode the status in three branches: -1 could not start, & 127 killed, else >> 8.
• A killed tool shifts to 0 — the case that matters most for long runs.
• system(...) == 0 or die is the right short form.
• Report your own exit status, not the tool's.

Chapter 15.11 takes the pipe form seriously. When a tool produces a hundred thousand lines, capturing all of it before looking at any of it is the wrong shape — and the measured difference is larger than you might expect.