Passing Data Across the Bash/Perl Boundary

Part II · Module 26 · Chapter 26.2 · Passing Data Across the Bash/Perl Boundary

1. Four Ways In, Two Ways Back

Chapter 26.1 forwarded arguments with "$@" and stopped there. A real entry point has more to pass: a project root, a corner name, a list of reports, a seed. There are exactly four channels for it, and each one has a natural kind of value.

Coming back, there are only two: Perl's exit status (Chapter 26.1) and Perl's output, which Bash captures with $(...) or redirects to a file. Section 7 measures what $(...) does to it.

One idea runs through every section of this chapter. A value is safe when it travels as data and unsafe when it travels as code. Every failure below is a value that was supposed to be data and ended up being parsed — by Perl's compiler, by Perl's own switch parser, by Getopt::Long, or by the shell.

2. Arguments: Never Splice a Value Into Perl Code

The most common mistake is not in a script at all. It is in a one-liner inside a Bash wrapper, where a shell variable is pasted into the Perl code between double quotes:

perl -e "my \$c = '$corner'; print \"corner=\$c\\n\""     # the value becomes CODE
perl -e 'my $c = shift; print "corner=$c\n"' -- "$corner"  # the value stays DATA

The first line builds the Perl program by text substitution. Whatever is in $corner is now part of the program's source. Three corner values show what that means:

#!/bin/bash
for corner in ss_0p72v_125c "tt_owner's" "x'; print \"INJECTED\\n\"; '"; do
    echo "--- value: $corner"
    printf '  spliced:  '
    perl -e "my \$c = '$corner'; print \"corner=\$c\\n\"" 2>&1
    echo "            exit $?"
    printf '  argument: '
    perl -e 'my $c = shift; print "corner=$c\n"' -- "$corner"
    echo "            exit $?"
done

--- value: ss_0p72v_125c
  spliced:  corner=ss_0p72v_125c
            exit 0
  argument: corner=ss_0p72v_125c
            exit 0
--- value: tt_owner's
  spliced:  Substitution pattern not terminated at -e line 1.
            exit 255
  argument: corner=tt_owner's
            exit 0
--- value: x'; print "INJECTED\n"; '
  spliced:  INJECTED
corner=x
            exit 0
  argument: corner=x'; print "INJECTED\n"; '
            exit 0

An ordinary value works both ways, which is why the spliced form passes every test its author writes.

A value containing a quote is a syntax error in the spliced form. The program Perl actually received was my $c = 'tt_owner's'; .... The apostrophe closed the string after tt_owner, and Perl then read s'; print ... as the start of a substitution — s with ' as its delimiter — which never finds its closing delimiter. Hence the baffling message about a substitution pattern, in a program that contains no substitution at all. The exit status is non-zero, so at least this failure is loud; the message, though, describes code the author never wrote.

A value that happens to be valid Perl is executed. The third value closes the string, adds a statement, and reopens a string so the rest still compiles. The program ran the injected print — the INJECTED line, followed by corner=x on a line of its own — and exited 0. Nobody is likely to name a corner like that on purpose; the point is that a test name read from a spreadsheet, a path from a user, or a field copied out of a report is text from somewhere else, and the spliced form gives that text the power to change the program.

The argument form gives the same answer for all three, including the hostile one, because the value never passes through any parser. It arrives in @ARGV exactly as Bash held it. This is the same rule as Chapter 15.10's list-form system, seen from the other side of the boundary: keep the code fixed and pass the values separately.

3. A Dash at the Start: Two Parsers, Two Places for --

A filename that starts with a dash is rare and real — a generated report named after a negative corner temperature, -40c.rpt, is the classic case. Such a name meets two separate option parsers on its way into a Perl script, and each needs its own protection.

The first parser is Perl's own. When Perl is started with -e, it keeps reading its own switches after the code, so a dash-leading argument is taken as a Perl switch. The second is Getopt::Long inside the script, which treats anything starting with a dash as one of the script's options.

use strict;
use warnings;
use Getopt::Long;

my $corner = "tt";
GetOptions("corner=s" => \$corner) or do { print "  GetOptions failed\n"; exit 2 };
print "  corner=$corner files=[@ARGV]\n";

#!/bin/bash
echo "--- Perl's own switch parser (perl -e)"
perl -e 'print "  ARGV=[@ARGV]\n"' -n.rpt 2>&1 || echo "  exit non-zero"
perl -e 'print "  ARGV=[@ARGV]\n"' -- -n.rpt;       echo "  exit $?"

echo "--- the script's parser (Getopt::Long)"
perl opt.pl --corner ss -n.rpt 2>&1 | sed 's/^Unknown/  Unknown/'
echo "  exit ${PIPESTATUS[0]}"
perl opt.pl --corner ss -- -n.rpt;                   echo "  exit $?"

echo "--- an option whose value is missing"
perl opt.pl --corner -n.rpt a.rpt;                   echo "  exit $?"

--- Perl's own switch parser (perl -e)
Unrecognized switch: -.rpt  (-h will show valid options).
  exit non-zero
  ARGV=[-n.rpt]
  exit 0
--- the script's parser (Getopt::Long)
  Unknown option: n.rpt
  GetOptions failed
  exit 2
  corner=ss files=[-n.rpt]
  exit 0
--- an option whose value is missing
  corner=-n.rpt files=[a.rpt]
  exit 0

Perl's switch parser rejected -n.rpt — it read -n as the -n loop switch (Chapter 15.7), then tried the rest, .rpt, as another switch, which is why the message says -.rpt. A -- after the code ends Perl's switch parsing, and the name arrives as an ordinary argument.

The driver prints "non-zero" rather than the number on purpose. The status Perl exits with here is not a fixed code: it was 29 every time inside this page's runner and at a terminal, and 25 every time its stderr was redirected to /dev/null, because it is whatever errno was left behind — the same behaviour Chapter 15.4 measured for die. A caller can rely on it being non-zero and on nothing else.

Getopt::Long rejected it again, as Unknown option: n.rpt. Note that passing a script name instead of -e does protect against the first parser — Perl stops reading its own switches at the script name — but not against the second. A -- after the script's options tells Getopt::Long that everything that follows is a file.

The last case is the dangerous one, and it exits 0. --corner expects a value, the corner was forgotten, and Getopt::Long took the next word — the filename — as the corner name. The script then checked a corner called -n.rpt against a file list missing one file. There is no error message because, from the parser's point of view, nothing was wrong.

The defence for that one is not --; it is the validation Chapter 15.3 teaches. A corner name has a known shape, so check it:

die "corner '$corner' does not look like a corner name\n"
    unless $corner =~ /^[a-z]{2}_[0-9a-z_]+$/;

A wrapper that always forwards with "$@" and puts -- before any file list closes both parsers' doors. The validation closes the third.

4. The Environment: Shared Settings, With Three States

The environment is the right channel for a value that every process in the job needs — the project root, a tool installation directory, a licence server. It is the wrong channel for a value that describes this call, because nothing in a command line shows that it was passed.

Chapter 15.5 covered the Perl side: %ENV, the difference between set, empty and unset, and why a child cannot change its parent's environment. The Bash side adds two rules of its own, and the combination produces one specific trap.

#!/bin/bash
check='my $r = $ENV{PROJ_ROOT} // die "  PROJ_ROOT is not set\n";
       printf "  got [%s] (length %d)\n", $r, length $r;'

echo "--- a shell variable that was never exported"
PROJ_ROOT=/proj/chip_a
perl -e "$check" 2>&1;                       echo "  exit $?"

echo "--- exported"
export PROJ_ROOT
perl -e "$check" 2>&1;                       echo "  exit $?"

echo "--- set for one command only"
PROJ_ROOT=/proj/chip_b perl -e "$check" 2>&1; echo "  exit $?"
echo "  and afterwards the shell still has: $PROJ_ROOT"

echo "--- exported but EMPTY"
PROJ_ROOT= perl -e "$check" 2>&1;            echo "  exit $?"

echo "--- removed"
unset PROJ_ROOT
perl -e "$check" 2>&1;                       echo "  exit $?"

--- a shell variable that was never exported
  PROJ_ROOT is not set
  exit 255
--- exported
  got [/proj/chip_a] (length 12)
  exit 0
--- set for one command only
  got [/proj/chip_b] (length 12)
  exit 0
  and afterwards the shell still has: /proj/chip_a
--- exported but EMPTY
  got [] (length 0)
  exit 0
--- removed
  PROJ_ROOT is not set
  exit 255

That driver puts Perl code in a shell variable and splices it into -e — exactly what section 2 warned against. Here it is safe for one reason only: the spliced text is a fixed string written by the script's author, and no outside value is pasted into it. The value under test travels through the environment, as data.

Reading the five cases:

A shell variable is not in the environment until it is exported. The first case set PROJ_ROOT in Bash and Perl saw nothing. This is the source of the classic "it works when I type it" report: the variable was exported in the engineer's login shell and only assigned in the script.

NAME=value command sets it for one command only, and the shell's own value is unchanged afterwards. That makes it the cleanest way to give one tool a different setting without disturbing anything else.

Exported but empty passes // die. The defined-or operator tests for defined, and an empty string is defined. So the check that reads correctly — "die if PROJ_ROOT is missing" — lets an empty root through, and every path the script builds from it becomes relative to wherever the job happens to be running. That is how a cleanup script ends up operating on the wrong directory. The check that says what is meant is:

my $root = $ENV{PROJ_ROOT};
die "PROJ_ROOT is not set or is empty\n" unless defined $root && length $root;
die "PROJ_ROOT=$root is not a directory\n" unless -d $root;

The second line matters as much as the first. A root that is set, non-empty and wrong is the more common failure, and -d (Chapter 13.6) catches it at the top of the script instead of thirty lines later.

5. When the List Is Long: A File or STDIN

Arguments have a size limit. It is set by the operating system, not by Bash or Perl, and it covers the total size of all arguments plus the environment. A regression manager that passes every test's log path, or a filelist generator that passes every source file, can reach it.

use strict;
use warnings;

# How many paths can be passed to a child as arguments? The limit is the
# operating system's, so this measures THIS machine.
for my $n (1_000, 10_000, 50_000) {
    my @paths = map { sprintf "rtl/core/block_%06d.sv", $_ } 1 .. $n;
    my $rc = system($^X, "-e", "exit 0", @paths);
    printf "%6d paths as arguments: %s\n", $n,
        $rc == -1 ? "could not start the child -- $!" : "ok (exit " . ($rc >> 8) . ")";
}

  1000 paths as arguments: ok (exit 0)
 10000 paths as arguments: ok (exit 0)
 50000 paths as arguments: could not start the child -- Argument list too long

At fifty thousand paths the child was never started — system returned -1, and $! says why. Under use warnings, Perl also printed Can't exec "perl": Argument list too long on stderr, which is not shown above because the captured block is standard output only; in a batch job that warning lands in a log nobody reads, so the -1 check is the one that matters. That is Chapter 14.3's "no child at all" case, and it is a different failure from any exit status the child could have produced. The exact limit varies: this machine reports getconf ARG_MAX as 1,048,576 bytes, and Linux systems are typically larger and also limit the size of a single argument. Do not design to the number. Design so that the number does not matter.

The two channels with no size limit are a file and standard input. A file is the better default for anything that is also evidence — the list of tests a regression ran is worth keeping in the run directory, not just passing through a pipe. Standard input is right when another program is producing the list right now.

6. A Newline in a Filename

Standard input has its own failure. The usual way to send a list of paths through a pipe is one path per line. That works for every path that does not contain a newline — and a newline is a legal character in a Unix filename.

Filenames with newlines are not common in a well-run project. They appear when a script builds a name from a field it did not clean — a test description copied out of a log, a report title — and then the first tool to read a newline-separated list quietly loses the file.

#!/bin/bash
mkdir -p reports
touch "reports/plain.rpt" "reports/Rev A.rpt" "reports/two
lines.rpt"

echo "--- one path per line"
find reports -name '*.rpt' |
    perl -ne 'chomp; printf "  %-22s %s\n", "[$_]", (-e $_ ? "exists" : "MISSING")' | sort

echo "--- NUL-separated"
find reports -name '*.rpt' -print0 |
    perl -0 -ne 'chomp; (my $show = $_) =~ s/\n/\\n/g;
                 printf "  %-22s %s\n", "[$show]", (-e $_ ? "exists" : "MISSING")' | sort

--- one path per line
  [lines.rpt]            MISSING
  [reports/Rev A.rpt]    exists
  [reports/plain.rpt]    exists
  [reports/two]          MISSING
--- NUL-separated
  [reports/Rev A.rpt]    exists
  [reports/plain.rpt]    exists
  [reports/two\nlines.rpt] exists

One path per line turned one file into two names, and both are missing. Perl did nothing wrong — it read lines, as asked. The list format could not represent the data.

NUL-separated, all three exist. The NUL byte is the one character that cannot appear in a Unix path, so a NUL-separated list can carry any filename. find -print0 writes that format, and Perl's -0 switch (with no digits) sets the input record separator $/ to the NUL byte, so chomp removes the NUL rather than a newline. From Bash, printf '%s\0' "${files[@]}" writes the same format from an array.

Both outputs end in sort because find returns names in directory order, which is not guaranteed to be the same on every filesystem — the same argument Chapter 17.8 makes for any output a person or a diff will read.

7. Coming Back: What $(...) Does to Perl's Output

The way back from Perl to Bash is command substitution: result=$(perl ...). Two of its properties cause real bugs.

#!/bin/bash
echo "--- trailing newlines"
v=$(perl -e 'print "worst=-0.043\n\n\n"')
printf '  captured [%s]\n' "$v"
v=$(perl -e 'print "\n"')
printf '  a single newline captured as [%s] -- length %d\n' "$v" "${#v}"

echo "--- the status of a capture"
plain()  { v=$(perl -e 'exit 3');       echo "  plain assignment:    \$? = $?"; }
local1() { local v=$(perl -e 'exit 3'); echo "  local v=\$(...):      \$? = $?"; }
local2() { local v; v=$(perl -e 'exit 3'); echo "  local v; v=\$(...):   \$? = $?"; }
plain; local1; local2

echo "--- only stdout is captured"
v=$(perl -e 'print "data\n"; print STDERR "  a message on stderr\n"' 2>/dev/null)
printf '  captured [%s]\n' "$v"

--- trailing newlines
  captured [worst=-0.043]
  a single newline captured as [] -- length 0
--- the status of a capture
  plain assignment:    $? = 3
  local v=$(...):      $? = 0
  local v; v=$(...):   $? = 3
--- only stdout is captured
  captured [data]

Every trailing newline is removed, not just one. Three newlines became none, and a value that was only a newline became the empty string. For one-line results that is convenient. For a value where trailing blank lines carry meaning — the exact text of a report section — capture to a file instead.

local v=$(...) reports status 0. This is the boundary trap that catches experienced Bash authors. local is a command, and $? after it is the status of local — which succeeded in declaring a variable — not of the Perl program inside the substitution. The plain assignment and the split form both report Perl's 3. So inside a function, declare first, then assign: local v; v=$(perl ...).

Only standard output is captured. That is the behaviour you want, and it is why Chapter 15.12's rule — data on stdout, every message on stderr — matters most at exactly this boundary. A Perl tool that prints its progress messages on stdout puts them into the variable.

8. Perl Writing Values That Bash Will eval

Sometimes a Perl program has to hand back several values — a corner, a run directory, a pattern. A common answer is for Perl to print NAME=value lines and for Bash to eval them. That is section 2's problem in reverse: Perl's output becomes Bash code, and every value must be quoted for the shell.

In a POSIX shell, the one quoting form with no special characters inside it is the single-quoted string. The only character that cannot appear inside it is the single quote itself, which is written by closing the string, adding an escaped quote, and reopening it: '\''.

use strict;
use warnings;

# Quote a value so a POSIX shell reads it back unchanged.
sub shell_quote {
    my ($s) = @_;
    $s =~ s/'/'\\''/g;
    return "'$s'";
}

my %settings = (
    CORNER  => "tt_owner's",
    RUN_DIR => "/proj/run A",
    PATTERN => "*.rpt",
    NOTE    => 'costs $5; see `date`',
);
for my $name (sort keys %settings) {
    die "bad variable name: $name\n" unless $name =~ /^[A-Z_][A-Z0-9_]*$/;
    print "$name=", shell_quote($settings{$name}), "\n";
}

#!/bin/bash
echo "--- what Perl printed"
perl emit_settings.pl | sed 's/^/  /'
echo "--- what Bash holds after eval"
eval "$(perl emit_settings.pl)"
printf '  CORNER=[%s]\n  NOTE=[%s]\n  PATTERN=[%s]\n  RUN_DIR=[%s]\n' \
    "$CORNER" "$NOTE" "$PATTERN" "$RUN_DIR"

--- what Perl printed
  CORNER='tt_owner'\''s'
  NOTE='costs $5; see `date`'
  PATTERN='*.rpt'
  RUN_DIR='/proj/run A'
--- what Bash holds after eval
  CORNER=[tt_owner's]
  NOTE=[costs $5; see `date`]
  PATTERN=[*.rpt]
  RUN_DIR=[/proj/run A]

Four values that each break something in an unquoted form — an apostrophe, a space, a glob, and a $ with backticks — all came back unchanged. Note that the variable names are checked too: a name is code in the generated line, so it must be a valid shell identifier, and the script refuses anything else rather than quoting it.

When the values are more than a handful, or nested, stop doing this. Write a file in a defined format and have the consumer parse it — which is exactly what Chapter 26.3 does.

9. Exercises

1. Classify the channel. For each of these values in a nightly regression, choose a channel and justify it in one sentence: the project root; the random seed for one test; the list of 3,000 test names; the path to the simulator installation; the output of a test-list generator that is still running; the run directory for this job.

2. Break the one-liner. A wrapper contains perl -ne "print if /$pattern/" "$log". Find two values of $pattern that make it misbehave in different ways — one a compile error, one a silently wrong result — and rewrite it so $pattern travels as data. What changes about the regex if the user's pattern is meant to be a literal string, and which Part I construct handles that?

3. Prove the empty-root bug. Write a ten-line Perl script that builds "$ENV{PROJ_ROOT}/runs" and lists its contents, guarded only by // die. Run it with PROJ_ROOT exported but empty from a directory that happens to contain a runs folder, and describe what the output would mislead a reader into believing. Then fix the guard as in section 4.

4. Round-trip test. Extend emit_settings.pl with a value containing a newline and one containing a single quote followed by a backslash. Predict the output of read_settings.sh, run it, and explain why shell_quote needs no special case for either.

5. The 50,000-path job. A regression script calls system("summarise.pl", @log_paths) and works on every block-level run, then fails on the full-chip regression with no output from summarise.pl at all. Using sections 5 and 14.3, write down exactly what $? and $! held, and give two different fixes — one that changes the channel and one that keeps arguments but batches them. Which would you choose for a list that is also evidence, and why?

10. What To Remember

• Four channels in — arguments, environment, a file, STDIN — and two back: exit status and stdout.
• A value is safe as data and unsafe as code. Every failure in this chapter is a value that got parsed.
• Never splice a shell variable into perl -e code. Measured: a quote is a syntax error; a crafted value runs, and exits 0.
• Pass values as arguments after --: perl -e '...' -- "$value".
• A dash-leading name meets two parsers — Perl's own (with -e) and Getopt::Long. Each needs a --.
• An option with a missing value swallows the next word, silently. Validate the value's shape (15.3).
• A shell variable reaches Perl only once exported. NAME=value cmd sets it for one command.
• Exported-but-empty passes // die. Check defined && length, then -d.
• Arguments have an OS size limit. Measured: 50,000 paths gave -1, Argument list too long, and no child. Use a file or STDIN.
• A newline in a filename breaks a one-per-line list (measured: one file became two missing names). Use NUL separation: find -print0, perl -0.
• $(...) strips every trailing newline, and captures stdout only.
• local v=$(cmd) hides cmd's status. Declare, then assign.
• Perl output that Bash will eval is code. Single-quote every value (' becomes '\'') and validate every name.

Chapter 26.3 replaces section 8's handful of shell assignments with something a whole toolchain can rely on: a report file with a documented schema, written by Perl and read by a consumer in another language.