Environment, Modules and Tool Setup

Part II · Module 26 · Chapter 26.8 · Environment, Modules and Tool Setup

1. The Environment Is an Input

A tool's result depends on the design, the constraints, the scripts — and on the environment it was started in. The environment decides which executable runs when a script says iverilog or pt_shell, which licence server it asks for a licence, which libraries and settings its own start-up files load. None of that appears in the design database or the run script, and all of it changes results.

So a flow that is meant to be reproducible has two jobs:

• Set the environment deliberately, for each tool, rather than inheriting whatever the person or the queue happened to have.
• Record what was actually used — the resolved executable, its version, the variables that mattered — next to the results, so that when two runs disagree, the difference can be found by reading two files rather than by guessing.

Chapter 15.5 covered %ENV itself: reading it safely, set versus empty versus unset, local for a temporary change, and the rule that the environment flows downhill only — a child can never change its parent's. This chapter applies those rules to tool setup.

2. PATH Chooses the Tool

Every command run by name is looked up in PATH, left to right, and the first match wins. Two installations of a tool on the same machine — a project-pinned version and a newer system one — are separated by nothing but their order in PATH.

To show it without a second simulator, this chapter uses a stub: a two-line script called iverilog that prints a line saying it is a stub. It is not a simulator; it exists only so that "which iverilog ran?" has a visible answer.

#!/bin/sh
echo "STUB iverilog from old_tools -- not a simulator"

use strict;
use warnings;
use Cwd qw(getcwd);

my $stub_dir = getcwd() . "/old_tools/bin";

sub which_ran {
    my $out = `iverilog -V 2>&1`;
    return (split /\n/, $out)[0];
}

print "inherited PATH : ", which_ran(), "\n";
{
    local $ENV{PATH} = "$stub_dir:$ENV{PATH}";
    print "stub dir first : ", which_ran(), "\n";
}
print "after the block: ", which_ran(), "\n";

inherited PATH : Icarus Verilog version 13.0 (stable) (v13_0)
stub dir first : STUB iverilog from old_tools -- not a simulator
after the block: Icarus Verilog version 13.0 (stable) (v13_0)

The same command, iverilog -V, ran two different programs depending only on PATH. And local did exactly what Chapter 15.5 promised: the change applied to every child started inside the block, and the moment the block ended PATH was restored, so the third call found the real simulator again.

That is the right shape for tool setup in Perl. Set the environment for the tool in the smallest scope that contains its launch, with local, rather than assigning to $ENV{PATH} at the top of the script. A global assignment leaks into every later child — including tools that were never meant to see the change — and makes the script's behaviour depend on the order of its own statements.

3. Resolve Once, Run by Absolute Path

PATH lookup happens at launch, every launch. If anything changes PATH between two steps — a local block, a sourced setup file, a helper that "fixes" the environment — the second step may run a different binary from the first without any message.

The stronger habit is to resolve the tool once, to an absolute path, and then run and record that path. Perl has no built-in which, but File::Spec->path returns PATH split into directories, which is all it takes:

use File::Spec;

sub find_tool {
    my ($name) = @_;
    for my $dir (File::Spec->path) {
        my $p = File::Spec->catfile($dir eq "" ? "." : $dir, $name);
        return $p if -f $p && -x _;
    }
    return undef;
}

The $dir eq "" case in that function is not a curiosity. An empty entry in PATH means the current directory. It appears whenever PATH is built by string concatenation from a variable that turns out to be empty — PATH="$EXTRA_BIN:$PATH" with EXTRA_BIN empty gives a leading :, and PATH="$PATH:$EXTRA_BIN" gives a trailing one. And the current directory of a flow step is usually a run directory full of generated files.

#!/bin/sh
echo "this is ./simtool, a file in the current directory"

use strict;
use warnings;

for my $path ("/usr/bin:/bin", "/usr/bin:/bin:") {
    local $ENV{PATH} = $path;
    my $rc = system("simtool");
    printf "PATH=%-15s -> %s\n", $path,
        $rc == -1 ? "simtool not found ($!)" : "ran something, exit " . ($rc >> 8);
}

PATH=/usr/bin:/bin   -> simtool not found (No such file or directory)
this is ./simtool, a file in the current directory
PATH=/usr/bin:/bin:  -> ran something, exit 0

The only difference between the two PATHs is one trailing colon, and with it a file sitting in the working directory was executed in place of a tool. In a run directory, that file could be anything a previous step wrote. A tool-setup routine should therefore refuse a PATH with an empty entry rather than quietly allow it:

die "PATH has an empty entry (the current directory): $ENV{PATH}\n"
    if grep { $_ eq "" } split /:/, $ENV{PATH}, -1;

The -1 limit matters: as Chapter 4.7 measured, without it split discards trailing empty fields — exactly the trailing-colon case this check exists to catch.

4. Licence and Tool Variables: For One Child, Not Globally

Commercial EDA tools find their licence server through environment variables. The names are tool-specific — LM_LICENSE_FILE is the generic one used by FlexLM-licensed tools, and some vendors read their own as well, such as the SNPSLMD_LICENSE_FILE in this chapter's canonical example. The value is usually port@host, or a list of them. Tools also commonly read a variable naming their installation directory.

There is no licence server on the machine these examples were run on, so this section cannot show a licence checkout. It can show the mechanism that matters for a Perl flow — giving one tool its settings without changing anyone else's:

use strict;
use warnings;

my $show = q{printf "  child sees LM_LICENSE_FILE=%s\n", $ENV{LM_LICENSE_FILE} // "(unset)"};

print "before:\n";
system($^X, "-e", $show);

{
    local $ENV{LM_LICENSE_FILE} = '27000@lic-a';
    print "inside local block, launching the tool:\n";
    system($^X, "-e", $show);
}

print "after:\n";
system($^X, "-e", $show);

before:
  child sees LM_LICENSE_FILE=(unset)
inside local block, launching the tool:
  child sees LM_LICENSE_FILE=27000@lic-a
after:
  child sees LM_LICENSE_FILE=(unset)

The child launched inside the block saw the setting; the children before and after did not. Two flow steps that need different licence servers — or one step that must not see a variable the user's login shell exports — are handled the same way.

The value 27000@lic-a is an example in the usual port@host shape, not a real server. Licence server addresses are site configuration rather than secrets, but credentials never belong in the environment of a flow step that logs its environment — which, after Section 7, every step should.

5. Why module load Cannot Work From Perl the Obvious Way

Many sites set tools up with environment modules — Environment Modules or Lmod — where module load pt/2023.03 adjusts PATH, licence variables and tool variables in one command. The obvious way to do that inside a Perl flow is system("module load ..."). It cannot work, for two separate reasons, and both can be measured.

use strict;
use warnings;

# 1. `module` is a shell FUNCTION defined in the user's login shell, not a
#    program on PATH -- so there is nothing for system() to execute.
my $rc = system("module", "load", "sim/13.0");
printf "system(module load)     -> %d (%s)\n", $rc, $rc == -1 ? $! : "ran";

# 2. Even through a shell, a child's environment changes die with the child.
system("sh", "-c", "export SIM_HOME=/tools/sim/13.0");
printf "after a child exported  -> SIM_HOME is %s in Perl\n",
    defined $ENV{SIM_HOME} ? $ENV{SIM_HOME} : "not set";

system(module load)     -> -1 (No such file or directory)
after a child exported  -> SIM_HOME is not set in Perl

The first failure is about what module is. In both common implementations it is defined as a shell function when the login shell starts, and it calls the real implementation behind the scenes. A shell function exists only inside a shell that defined it. Perl's list-form system runs a program directly, finds no program called module, and returns -1 — Chapter 14.3's "no child at all". (On this machine no module system is installed, so the message is the same either way; on a site that has one, it is still -1 for this reason.)

The second failure applies to every approach that runs setup in a child. The child's environment is its own. When the child exits, whatever it exported is gone. This is Chapter 15.5's "downhill only" rule, and no amount of quoting gets around it.

Three approaches do work.

Load modules before Perl starts. The Bash entry point from Chapter 26.1 runs module load ... and then execs Perl. Perl inherits the environment, and every tool Perl starts inherits it from Perl. This is the simplest and the most common, and it puts the environment decision in one visible place.

Run setup and tool in the same child shell. When one step needs a setup the others must not see, start a shell that does the setup and then execs the tool, so the setup and the tool share one process. The stand-in for a site's setup below is a small file that does what module load does — export variables and prepend to PATH; at a real site it would contain the module load line itself.

# stand-in for a site's tool setup (e.g. a `module load` line)
export SIM_HOME=/tools/sim/13.0
export PATH="$SIM_HOME/bin:$PATH"

use strict;
use warnings;

# Everything after 'bash' is passed as $1, $2, ... -- untouched, no re-quoting.
my @tool = ($^X, "-e", 'printf "  tool sees SIM_HOME=%s, args: %s\n", $ENV{SIM_HOME}, join(" | ", @ARGV)',
            "--", "--corner", "ss 0p72v");

system("bash", "-c", 'source ./sim_setup.sh && exec "$@"', "bash", @tool) == 0
    or die "setup or tool failed: $?\n";
printf "  and back in Perl, SIM_HOME is %s\n", $ENV{SIM_HOME} // "not set";

  tool sees SIM_HOME=/tools/sim/13.0, args: --corner | ss 0p72v
  and back in Perl, SIM_HOME is not set

The tool saw the setup, its arguments arrived intact — ss 0p72v is still one argument — and Perl's own environment was untouched. The pattern is worth reading slowly: the script passed to bash -c is a fixed string with no values pasted into it (Chapter 26.2's rule), the word bash after it becomes the shell's $0, and the tool's command line follows as separate arguments that "$@" forwards exactly (Chapter 26.1's rule). exec replaces the shell with the tool, so the tool's exit status and any signal reach Perl directly. And the -- after the -e code is Chapter 26.2's rule at work: without it, the first version of this example failed with Unrecognized switch: --corner, because Perl read the tool's own option as one of Perl's.

Ask the module system for Perl code. Environment Modules can print the environment changes for a load as statements in a chosen language, including Perl, which a script then evaluates. This avoids a child shell entirely, but how it is invoked depends on the installation and version, and no module system exists on this machine to verify it against. If your site uses it, the site's own documentation is the reference.

6. A Clean Environment Shows the Real Dependencies

A script that works in your shell may depend on things your shell happens to provide: a directory on PATH, a variable set in a start-up file, a module loaded weeks ago. The fastest way to find those dependencies is to remove them and see what breaks.

Perl can run a child with an almost empty environment by localising the whole of %ENV:

use strict;
use warnings;

my $minimal = "/usr/bin:/bin";

sub try {
    my ($label, @cmd) = @_;
    local %ENV = (PATH => $minimal);
    my $out = `@cmd -V 2>&1`;
    my $rc = $?;
    printf "%-34s %s\n", $label,
        $rc == -1 || ($rc >> 8) == 127 ? "NOT FOUND"
      : $rc == 0  ? "runs: " . (split /\n/, $out)[0]
      :             "exit " . ($rc >> 8);
}

try("minimal env, by name:",          "iverilog");
try("minimal env, by absolute path:", "/opt/homebrew/bin/iverilog");
{
    local %ENV = (PATH => $minimal);
    my @vars = `env`;
    printf "%-34s %d: %s", "variables the child received:", scalar @vars, join "", @vars;
}

minimal env, by name:              NOT FOUND
minimal env, by absolute path:     runs: Icarus Verilog version 13.0 (stable) (v13_0)
variables the child received:      1: PATH=/usr/bin:/bin

With only /usr/bin:/bin on PATH, iverilog is not found — on this machine it is installed under /opt/homebrew/bin, and the inherited PATH was the only reason the earlier examples worked. By absolute path it runs, so the tool itself needs nothing else from the environment to print its version.

That is the diagnostic value of a clean run: every failure in it names a dependency that was invisible before. Real tools typically fail in a clean environment on their installation variable or their licence variable first, and each failure tells you one line that belongs in the flow's explicit setup. Running a flow step this way once — not in production, as a check — converts "it works on my machine" into a list.

7. Record What Was Used

All of the above sets the environment deliberately. The last step makes it visible after the fact: before launching a tool, write down exactly what is about to run, in a file next to its results. When a result is questioned a week later, this file answers the first four questions anyone asks — which executable, which version, which arguments, which environment.

use strict;
use warnings;
use File::Spec;
use JSON::PP;
use Sys::Hostname qw(hostname);

sub find_tool {
    my ($name) = @_;
    for my $dir (File::Spec->path) {
        my $p = File::Spec->catfile($dir eq "" ? "." : $dir, $name);
        return $p if -f $p && -x _;
    }
    return undef;
}

sub tool_version {
    my ($path) = @_;
    open my $fh, "-|", $path, "-V" or return "unknown (cannot run: $!)";
    my $first = "";
    while (my $l = <$fh>) { chomp $l; if ($l =~ /\S/) { $first = $l; last } }
    close $fh;
    return length $first ? $first : "unknown (no output)";
}

# A fixed PATH so this listing is reproducible. On your machine, list the
# directory your simulator is installed in first.
local $ENV{PATH} = "/opt/homebrew/bin:/usr/bin:/bin";

my $name = "iverilog";
my $path = find_tool($name) // die "$name: not found on PATH\n";
my @argv = ("-g2012", "-o", "sim.vvp", "tb.v");
my @RECORD_VARS = qw(PATH LM_LICENSE_FILE SIM_HOME);

my %meta = (
    tool       => $name,
    resolved   => $path,
    version    => tool_version($path),
    argv       => [ $path, @argv ],
    env        => { map { $_ => $ENV{$_} // undef } @RECORD_VARS },
    perl       => sprintf("%vd", $^V),
    host       => hostname(),
    started    => scalar gmtime(),
);

open my $out, ">", "run_meta.json" or die "run_meta.json: $!\n";
print {$out} JSON::PP->new->canonical(1)->pretty->encode(\%meta);
close $out or die "run_meta.json: $!\n";

# Show the file, minus the two fields that differ on every run and machine.
my $shown = { %meta };
delete @{$shown}{qw(host started)};
print JSON::PP->new->canonical(1)->pretty->encode($shown);

{
   "argv" : [
      "/opt/homebrew/bin/iverilog",
      "-g2012",
      "-o",
      "sim.vvp",
      "tb.v"
   ],
   "env" : {
      "LM_LICENSE_FILE" : null,
      "PATH" : "/opt/homebrew/bin:/usr/bin:/bin",
      "SIM_HOME" : null
   },
   "perl" : "5.34.1",
   "resolved" : "/opt/homebrew/bin/iverilog",
   "tool" : "iverilog",
   "version" : "Icarus Verilog version 13.0 (stable) (v13_0)"
}

The listing above leaves out host and started only because they differ on every machine and every run; the file itself keeps both.

The design decisions in it are each deliberate:

resolved is an absolute path, found once, and argv starts with that same path — so the record describes the binary that the launch will use, not a name that might resolve differently later. Launching with $meta{argv} makes that true by construction.

version comes from the tool itself, read through a list-form pipe (Chapter 15.11), taking the first non-blank line of standard output. On this machine iverilog -V also prints a note about its preprocessor's version on stderr, which the pipe does not read and which reaches the terminal instead — one more reason to read a specific stream on purpose rather than merging them. Tools differ — some print their version on stderr, some need a different flag, some print a banner first — so the function returns unknown (...) rather than dying when it cannot tell. A missing version is worth recording as missing.

env lists named variables, including the ones that are unset. An unset licence variable is a fact about the run, and null in the JSON records it — Chapter 26.3's rule that "not known" and "empty" are different. The list is chosen, not the whole of %ENV: the whole environment contains personal settings and occasionally credentials, and a record that is sometimes unsafe to share is a record people stop keeping.

The file is canonical JSON (Chapter 26.3), so two runs' records can be compared with diff, and a difference in the tool version or PATH stands out as a changed line.

8. Exercises

1. Two installations. Put a second stub named iverilog in another directory and write a function that lists every iverilog on PATH in order, marking the one that wins. Run it with three different PATH orders. Why is "every match" more useful in a run record than "the winning match"?

2. A setup that fails. Change sim_setup.sh so that its last command fails. Predict what setup_then_exec.pl does, then run it. Now remove the && and use ; instead, and explain why that is worse even though the tool still runs.

3. The clean-run report. Extend clean_env.pl so that, given a command, it runs it in the minimal environment, then adds back one inherited variable at a time until the command works, and reports the smallest set of variables it needed. What are the limits of this method for a tool whose failure is a licence checkout that times out after two minutes?

4. Compare two records. Write meta_diff.pl that reads two run_meta.json files and prints only the fields that differ, treating a reordered PATH with the same entries as different. Then decide which differences should make a regression comparison stop with an error and which should only warn.

5. Audit a flow. In a flow you have access to, find every place PATH or a licence variable is assigned. For each one, note its scope (whole script, one block, one child), whether the value it builds can contain an empty entry, and whether the run directory records what was used.

9. What To Remember

• The environment is an input to every result. Set it deliberately; record what was used.
• PATH order chooses the tool. Measured: the same iverilog -V ran a stub or the real simulator depending only on order.
• Use local $ENV{...} around the launch, so the change reaches that child and is undone afterwards (15.5).
• Resolve the tool once to an absolute path, and run and record that path.
• An empty PATH entry is the current directory. Measured: one trailing colon executed a file in the working directory. Refuse such a PATH; split with limit -1.
• Licence and tool variables: set them for one child with local. Never put credentials where the environment is logged.
• system("module load ...") cannot work. Measured: -1, because module is a shell function, and a child's exports die with the child anyway.
• Load modules in the Bash entry point, or run setup and tool in one shell: bash -c 'source setup && exec "$@"' bash @cmd.
• A clean environment reveals dependencies. Measured: by name, iverilog was not found; by absolute path, it ran.
• Write run_meta.json before launching: resolved path, version, argv, chosen variables (unset ones as null), Perl version, host, time — canonical JSON, so records diff.

Chapter 26.9 closes the module by stepping back from mechanisms to decisions: which stage of a flow belongs to Bash, which to Tcl, which to Perl and which to Python — and the two common mistakes that come from choosing by habit.