shell=False, and When Shell Behaviour Actually Matters

Part II · Module 23 · Chapter 23.3 · shell=False, and When Shell Behaviour Actually Matters

1. What Arrives Without a Shell

import sys

for index, arg in enumerate(sys.argv[1:], 1):
    print(f"  argv[{index}] = {arg!r}")

import os
import subprocess
import sys

os.environ["SIM"] = "xcelium"
os.makedirs("rtl", exist_ok=True)
for name in ["alu.sv", "fifo.sv", "arb.v"]:
    open(os.path.join("rtl", name), "w").close()

for label, extra in [("a wildcard", ["rtl/*.sv"]),
                     ("a variable", ["$SIM"]),
                     ("a redirection", ["-o", ">", "out.log"])]:
    cp = subprocess.run([sys.executable, "echoargs.py"] + extra,
                        capture_output=True, text=True)
    print(label)
    print(cp.stdout.rstrip())

a wildcard
  argv[1] = 'rtl/*.sv'
a variable
  argv[1] = '$SIM'
a redirection
  argv[1] = '-o'
  argv[2] = '>'
  argv[3] = 'out.log'

All three arrived as literal text. The wildcard was not expanded, the variable was not substituted, and the > is an argument the tool will be confused by rather than a redirection.

That is the canonical annotation's first line:

subprocess.run(cmd)                         # no shell: no globbing

It is the same property Chapter 23.2 praised, seen from the other side. Nothing re-interprets your arguments — including the parts you wanted re-interpreted.

Four things a shell was doing, and the rest of this chapter is one section each.

2. Globbing

import os
from pathlib import Path

os.makedirs("rtl/cpu", exist_ok=True)
for name in ["rtl/alu.sv", "rtl/fifo.sv", "rtl/arb.v", "rtl/cpu/core.sv"]:
    Path(name).parent.mkdir(parents=True, exist_ok=True)
    Path(name).touch()

flat = sorted(str(p) for p in Path("rtl").glob("*.sv"))
deep = sorted(str(p) for p in Path("rtl").rglob("*.sv"))

print("Path.glob  :", flat)
print("Path.rglob :", deep)
print()
cmd = ["xrun", "-f", "files.f"] + flat
print("in a command:", cmd)

Path.glob  : ['rtl/alu.sv', 'rtl/fifo.sv']
Path.rglob : ['rtl/alu.sv', 'rtl/cpu/core.sv', 'rtl/fifo.sv']

in a command: ['xrun', '-f', 'files.f', 'rtl/alu.sv', 'rtl/fifo.sv']

Chapter 18.5's Path.glob does the expansion, and the result is a list — which is already the shape subprocess.run wants, so it goes straight into the command.

Two things it does better than a shell.

sorted() makes it deterministic. A shell's glob order depends on the locale and the filesystem; Chapter 18.5 Section 4 established that an unsorted glob gives a different command every run, which Chapter 21.3 Section 6 showed produces a spurious diff. One sorted() removes it.

You can see what matched before running anything. print(flat) before the command is a debugging step a shell command has no equivalent of — the expansion happens inside the shell, and by the time anything goes wrong the list is gone.

The third difference matters in a flow: a glob matching nothing is visible. A shell passes an unmatched rtl/*.sv through literally, so the tool receives a nonexistent filename; Path.glob returns an empty list and if not files: is one line.

3. Variable Expansion

import os
import subprocess
import sys

os.environ["SIM_ROOT"] = "/tools/xcelium/2024.09"

print("Python already has it:", os.environ["SIM_ROOT"])
print("with a default       :", os.environ.get("NOT_SET", "<unset>"))

cmd = ["xrun", "-f", os.path.join(os.environ["SIM_ROOT"], "share", "files.f")]
print("built into a command :", cmd[2])

cp = subprocess.run([sys.executable, "-c",
                     "import os; print('child sees:', os.environ.get('SIM_ROOT'))"],
                    capture_output=True, text=True)
print(cp.stdout.strip())

Python already has it: /tools/xcelium/2024.09
with a default       : <unset>
built into a command : /tools/xcelium/2024.09/share/files.f
child sees: /tools/xcelium/2024.09

This is the loss that costs nothing. $SIM_ROOT in a shell command is a lookup in the shell's environment; os.environ["SIM_ROOT"] is the same lookup in yours, and your script is already holding the value before the command is built.

It is also strictly better in one respect: a missing variable is visible. In a shell, $NOT_SET expands to an empty string, so xrun -f $NOT_SET/files.f becomes xrun -f /files.f and the tool reports a missing file at the root of the filesystem. In Python, os.environ["NOT_SET"] raises KeyError at the point of the mistake, and .get() with a default is the deliberate alternative — Chapter 8.3's distinction, deciding whether a missing setting is an error.

4. Redirection and Pipes

import sys

print("tool: compiling")
print("tool: warning on line 88", file=sys.stderr)
print("tool: done")

import subprocess
import sys
from pathlib import Path

with open("compile.log", "w", encoding="utf-8") as handle:
    cp = subprocess.run([sys.executable, "noisy.py"],
                        stdout=handle, stderr=subprocess.STDOUT)

print("returncode   :", cp.returncode)
print("stdout attr  :", cp.stdout)
print("compile.log  :")
for line in Path("compile.log").read_text(encoding="utf-8").strip().splitlines():
    print("   " + line)

returncode   : 0
stdout attr  : None
compile.log  :
   tool: warning on line 88
   tool: compiling
   tool: done

stdout=handle is > compile.log and stderr=subprocess.STDOUT is 2>&1, with no shell involved. Chapter 23.5 is about this choice properly; the point here is that redirection is an argument to subprocess.run, not a character in a command.

Look at the order in the log, though: the warning the tool printed second is in the file first. That is Chapter 22.5 Section 5's buffering again — merging the two streams into a file gives the child a block-buffered stdout and an unbuffered stderr, so the diagnostics land ahead of the output. A merged tool log is in an order the run did not happen in, and nothing about 2>&1 warns you.

And it is better than the shell form for the reason Chapter 22.4 Section 6 gave: tool | tee log discards the tool's exit status, while stdout=handle keeps cp.returncode intact. The thing people reach for shell=True to get is the thing that breaks their error reporting.

A genuine pipe — one tool's output feeding another's input — is rarer in a flow than it looks, because the second stage is usually something Python does better:

import subprocess
import sys

cp = subprocess.run([sys.executable, "-c",
                     "print('UVM_ERROR a')\nprint('UVM_INFO b')\nprint('UVM_ERROR c')"],
                    capture_output=True, text=True)

errors = [line for line in cp.stdout.splitlines() if line.startswith("UVM_ERROR")]
print("shell would be : tool | grep UVM_ERROR | wc -l")
print("errors found   :", len(errors))
print("and you kept   :", errors[0])
print("tool's status  :", cp.returncode)

shell would be : tool | grep UVM_ERROR | wc -l
errors found   : 2
and you kept   : UVM_ERROR a
tool's status  : 0

The pipeline's count is replaced by a list comprehension, and you keep both the matching lines and the tool's exit status — neither of which survives | grep | wc -l. Module 19 and Module 20 are entirely about doing this better than grep.

5. Sourcing a Setup Script

The one with no one-line answer. EDA flows are configured by shell scripts that set a dozen variables and extend PATH, and the instinct is to run one and then run the tool:

export SIM_ROOT=/tools/xcelium/2024.09
export SIM_LICENSE=5280@licsrv
export PATH="$SIM_ROOT/bin:$PATH"

import subprocess
import sys

subprocess.run(["/bin/sh", "-c", ". ./setup.sh"], check=True)

cp = subprocess.run([sys.executable, "-c",
                     "import os; print('SIM_ROOT set:', 'SIM_ROOT' in os.environ)"],
                    capture_output=True, text=True)
print("after sourcing in a separate child:")
print("  " + cp.stdout.strip())

after sourcing in a separate child:
  SIM_ROOT set: False

Nothing happened. The first child sourced the script, set its variables, and exited taking them with it — Chapter 23.1 Section 4's one-way boundary, in the form that costs the most time.

Two things do work.

Source and run in the same shell, which is one child doing both:

export SIM_ROOT=/tools/xcelium/2024.09
export SIM_LICENSE=5280@licsrv
export PATH="$SIM_ROOT/bin:$PATH"

import subprocess

cp = subprocess.run(["/bin/sh", "-c", '. ./setup.sh && echo "SIM_ROOT is $SIM_ROOT"'],
                    capture_output=True, text=True)
print(cp.stdout.strip())

SIM_ROOT is /tools/xcelium/2024.09

Simple, and it has a cost: every tool invocation is now a shell command string, which gives up everything Chapter 23.2 argued for. For a flow running one tool it is fine; for a flow running forty it reintroduces the quoting problem forty times.

Capture the environment the setup produces, once, and reuse it:

export SIM_ROOT=/tools/xcelium/2024.09
export SIM_LICENSE=5280@licsrv
export PATH="$SIM_ROOT/bin:$PATH"

import json
import os
import subprocess
import sys


def environment_after(script):
    """The environment a shell has after sourcing `script`."""
    dump = f'{sys.executable} -c "import json, os; print(json.dumps(dict(os.environ)))"'
    cp = subprocess.run(["/bin/sh", "-c", f". ./{script} && {dump}"],
                        capture_output=True, text=True, check=True)
    return json.loads(cp.stdout)


env = environment_after("setup.sh")

print("SIM_ROOT      :", env["SIM_ROOT"])
print("SIM_LICENSE   :", env["SIM_LICENSE"])
print("PATH extended :", len(env["PATH"]) > len(os.environ["PATH"]))
print("added to PATH :", env["PATH"].split(":")[0])

print()
cp = subprocess.run([sys.executable, "-c", "import os; print(os.environ['SIM_ROOT'])"],
                    env=env, capture_output=True, text=True)
print("a later child :", cp.stdout.strip())

SIM_ROOT      : /tools/xcelium/2024.09
SIM_LICENSE   : 5280@licsrv
PATH extended : True
added to PATH : /tools/xcelium/2024.09/bin

a later child : /tools/xcelium/2024.09

One shell runs the setup and immediately dumps its environment as JSON; Python parses it and passes it to every later call with env=. The shell is used once, for the one thing only a shell can do, and every tool invocation afterwards is a plain list.

Three details make it work in practice. The dump uses sys.executable so it is the same interpreter — Chapter 23.1 Section 2. check=True means a setup script that fails stops the flow rather than silently producing a partial environment. And the result is an ordinary dict, so a flow can log it, diff two of them, or add its own variables before use.

6. env= Replaces; It Does Not Extend

The mistake that undoes Section 5:

import os
import subprocess
import sys

PROBE = "import os; print('SIM:', os.environ.get('SIM'), '| PATH:', 'PATH' in os.environ)"

print("env={'SIM': 'vcs'}            ->", end=" ")
cp = subprocess.run([sys.executable, "-c", PROBE], env={"SIM": "vcs"},
                    capture_output=True, text=True)
print(cp.stdout.strip())

print("env={**os.environ, 'SIM': ...} ->", end=" ")
cp = subprocess.run([sys.executable, "-c", PROBE], env={**os.environ, "SIM": "vcs"},
                    capture_output=True, text=True)
print(cp.stdout.strip())

print("no env= at all                 ->", end=" ")
cp = subprocess.run([sys.executable, "-c", PROBE], capture_output=True, text=True)
print(cp.stdout.strip())

env={'SIM': 'vcs'}            -> SIM: vcs | PATH: False
env={**os.environ, 'SIM': ...} -> SIM: vcs | PATH: True
no env= at all                 -> SIM: None | PATH: True

env={"SIM": "vcs"} gave the child an environment containing exactly one variable. PATH is gone, and with it the ability to find any tool by name — so the next subprocess.run(["xrun", ...]) fails with a message about xrun not existing, on a machine where xrun is installed and on the path.

That error is maximally misleading. It names the tool, so the investigation starts with the tool setup, and the actual cause is a dict literal three functions away.

env={**os.environ, "SIM": "vcs"} is the form that extends — Chapter 21.6's merge, now deciding whether a child can find its tools. It is almost always what you meant.

A completely empty environment is occasionally what you want, for a reproducibility test that proves a tool needs nothing beyond what you pass it. That is a deliberate experiment, not a default.

7. When shell=True Is the Right Answer

The canonical annotation's second line:

subprocess.run("xrun ... | tee sim.log", shell=True)   # only when meant

Only when meant is the whole guidance. Three cases where it is genuinely right:

You are running a shell construct, deliberately — sourcing a setup script, as in Section 5, or a command whose whole point is a shell feature.

The command came from a human or a config as a line, and reproducing it exactly matters more than inspecting it. A cmd.sh from a previous run re-executed verbatim is the honest case.

A one-off script you are writing for yourself, where the inputs are fixed and known.

In every one of them the rule is the same: shell=True takes a string, and the string must be a constant or come from shlex.join — never from interpolating data.

import shlex
import subprocess

TEST = "axi_smoke"
cmd = ["echo", "running", TEST]

print("safe   :", shlex.join(cmd))
done = subprocess.run(shlex.join(cmd), shell=True, capture_output=True, text=True)
print("output :", done.stdout.strip())

safe   : echo running axi_smoke
output : running axi_smoke

Build the list, then serialise it — which keeps Chapter 23.2's correctness and adds the shell only at the last step.

8. The Argument Against Interpolation

import subprocess

SAFE = "axi_smoke"
ODD = "axi_smoke; echo INJECTED"

for name in [SAFE, ODD]:
    done = subprocess.run(f"echo running {name}", shell=True, capture_output=True, text=True)
    print(f"{name!r:<28} -> {done.stdout.strip()!r}")

print()
for name in [SAFE, ODD]:
    done = subprocess.run(["echo", "running", name], capture_output=True, text=True)
    print(f"{name!r:<28} -> {done.stdout.strip()!r}")

'axi_smoke'                  -> 'running axi_smoke'
'axi_smoke; echo INJECTED'   -> 'running axi_smoke\nINJECTED'

'axi_smoke'                  -> 'running axi_smoke'
'axi_smoke; echo INJECTED'   -> 'running axi_smoke; echo INJECTED'

With shell=True and an interpolated value, the data became a second command. With a list, the same value is one argument containing a semicolon.

In a web application this is a security story. In a VLSI flow it is almost always an accident story, and that is the version worth internalising: a test name with a semicolon is unusual, but a path containing a space, a define containing $, a filelist containing &, or a Verilog literal containing a backtick are all ordinary — and every one of them means something to a shell.

So the honest framing is not someone might attack your regression. It is: with shell=True and interpolated data, your command's behaviour depends on characters in your data, and you will not find out which ones until the day a block is named something new.

The list form has no such dependency, which is the real reason Chapter 23.2 is Critical and this chapter is not about getting the shell back for everyday use.

9. Predict, Run, Observe

import os
import subprocess
import sys
from pathlib import Path

os.makedirs("src", exist_ok=True)
Path("src/a.sv").touch()
Path("src/b.sv").touch()
os.environ["X"] = "xval"

PROBE = "import sys; print(sys.argv[1:])"

# A
print("A:", subprocess.run([sys.executable, "-c", PROBE, "src/*.sv"],
                           capture_output=True, text=True).stdout.strip())

# B
print("B:", sorted(str(p) for p in Path("src").glob("*.sv")))

# C
print("C:", subprocess.run([sys.executable, "-c", PROBE, "$X"],
                           capture_output=True, text=True).stdout.strip())

# D
print("D:", os.environ.get("X"), os.environ.get("NOPE", "<unset>"))

# E
out = subprocess.run([sys.executable, "-c", "import os; print('PATH' in os.environ)"],
                     env={"ONLY": "1"}, capture_output=True, text=True)
print("E:", out.stdout.strip())

# F
out = subprocess.run([sys.executable, "-c", "import os; print('PATH' in os.environ)"],
                     env={**os.environ, "ONLY": "1"}, capture_output=True, text=True)
print("F:", out.stdout.strip())

# G
print("G:", sorted(str(p) for p in Path("src").glob("*.v")))

The verified output:

A: ['src/*.sv']
B: ['src/a.sv', 'src/b.sv']
C: ['$X']
D: xval <unset>
E: False
F: True
G: []

A — the wildcard arrived literally. No shell, no expansion.

B — Path.glob expands it, sorted, as a list ready for a command.

C — the variable arrived literally too.

D — Python already had the value, and a missing one is visible rather than empty.

E — env= with one key left the child with no PATH. Section 6.

F — merging over os.environ keeps it.

G — a glob matching nothing is an empty list, where a shell would have passed src/*.v through to the tool as a filename.

10. Exercises

Exercise 1 — three literals. Pass a wildcard, a $VAR and a > to a stand-in child with no shell. Confirm all three arrive as text.

Exercise 2 — glob in Python. Replace a shell wildcard with Path.glob and feed the result into a command list.

Exercise 3 — sorted matters. Build the same command twice from an unsorted glob and say what Chapter 21.3 predicts about the two cmd.sh files.

Exercise 4 — nothing matched. Glob a pattern that matches no files, both in a shell and with Path.glob. Say what the tool receives in each case.

Exercise 5 — the missing variable. Reference an unset variable in a shell command and in Python. Compare what the tool is given.

Exercise 6 — redirection as an argument. Redirect a tool's output to a file with stdout= and confirm cp.returncode still holds the tool's status.

Exercise 7 — the pipeline you did not need. Replace tool | grep X | wc -l with a capture and a comprehension. List two things you gained.

Exercise 8 — sourcing does nothing. Source a setup script in one child and look for its variables in the next. Record the result and name the chapter that predicted it.

Exercise 9 — one shell, both jobs. Source and run in a single sh -c. Say what it costs for a flow with forty invocations.

Exercise 10 — capture the environment. Write environment_after and confirm a later child sees the sourced variables. Then make the setup script print a banner and record what breaks.

Exercise 11 — env= replaces. Pass env={"SIM": "vcs"} and check for PATH. Then describe the error the next tool launch would produce.

Exercise 12 — extend instead. Fix it with a merge and confirm both variables are present.

Exercise 13 — when shell=True. Name one place in your own flow where shell=True is genuinely right, and say what makes it so.

Exercise 14 — the accident. Put a semicolon, then a space, then a $ into a test name and run it both ways. Say which characters in your real data would be affected.

11. Summary

• Without a shell, a wildcard, a $VAR and a > all arrive at the tool as literal text.
• Globbing becomes Path.glob, which returns a list ready for a command — and sorted() makes it deterministic where a shell's order is not.
• A glob matching nothing is an empty list, where a shell hands the tool a filename that does not exist.
• Variable expansion costs nothing: Python already holds the value, and a missing variable raises instead of expanding to empty.
• Redirection is an argument, not a character. stdout=handle and stderr=subprocess.STDOUT are > and 2>&1, and they keep the tool's exit status where | tee discards it.
• A pipeline's second stage is usually something Python does better, and doing it in Python keeps both the matched lines and the status.
• Sourcing a setup script in a separate child does nothing — the variables leave with that child.
• Two things work: source and run in one shell, or capture the environment the setup produces and pass it with env= to every later call.
• Capture it by having the same shell dump os.environ as JSON, with check=True so a broken setup stops the flow.
• A setup script that prints a banner breaks a naive JSON parse — take the last line or use a sentinel.
• env= replaces the whole environment. env={"SIM": "vcs"} leaves the child with no PATH, and the next launch fails naming the tool rather than the cause.
• env={**os.environ, "SIM": "vcs"} extends, and is almost always what you meant.
• shell=True only when meant, with a constant string or one from shlex.join — never interpolated data.
• With shell=True, your command's behaviour depends on characters in your data — a space, a $, an & or a backtick are all ordinary in a flow and all mean something to a shell.
• shell=True also makes returncode the shell's, so a pipeline reports the last command's status and a missing tool gives 127 rather than an exception.

The next chapter takes the number that comes back and decides what to do with it — which is almost never to let it raise.