subprocess.run and the Command as a List

Part II · Module 23 · Chapter 23.2 · subprocess.run and the Command as a List

1. One Element per Argument

cmd = ["vcs", "-full64", "-f", "files.f", "-o", "simv"]
cp = subprocess.run(cmd)

Six strings, and each one is exactly one argument the tool receives. The program name is the first element; everything after it arrives as the tool's own argv[1:], in order, unchanged.

That last word is the whole chapter. Nothing between your list and the tool's arguments interprets, splits, expands or unquotes anything — which Chapter 23.1 Section 3 showed from the child's side, and which Section 2 is about to push on.

import sys

for index, arg in enumerate(sys.argv[1:], 1):
    print(f"  argv[{index}] = {arg!r}")
print("  count   =", len(sys.argv) - 1)

import subprocess
import sys

cmd = [sys.executable, "echoargs.py", "-full64", "-f", "files.f", "-o", "simv"]
cp = subprocess.run(cmd, capture_output=True, text=True)
print(cp.stdout.rstrip())

  argv[1] = '-full64'
  argv[2] = '-f'
  argv[3] = 'files.f'
  argv[4] = '-o'
  argv[5] = 'simv'
  count   = 5

echoargs.py stands in for the tool and does the one useful thing a stand-in can: it reports exactly what it was given, with repr so nothing is hidden.

Note that -f files.f is two elements, not one. A flag and its value are separate arguments to the tool, because that is how the tool's own argument parser will read them — Chapter 22.1 Section 4's while loop consuming a value, seen from the other end.

2. Three Hazards It Survives

The canonical purpose names three things a real command contains, and all three are the reason the list form exists:

import sys

for index, arg in enumerate(sys.argv[1:], 1):
    print(f"  argv[{index}] = {arg!r}")

import subprocess
import sys

HAZARDS = [
    ("a path with a space", ["-f", "/proj/my design/files.f"]),
    ("a plusarg with quotes", ["+UVM_TESTNAME=axi_smoke",
                               '+uvm_set_config_string=*,"msg","hi"']),
    ("defines with parentheses", ["-define", "WIDTH=(8+1)", "-define", "MODE='b101"]),
]

for label, extra in HAZARDS:
    print(label)
    cp = subprocess.run([sys.executable, "echoargs.py"] + extra,
                        capture_output=True, text=True)
    print(cp.stdout.rstrip())

a path with a space
  argv[1] = '-f'
  argv[2] = '/proj/my design/files.f'
a plusarg with quotes
  argv[1] = '+UVM_TESTNAME=axi_smoke'
  argv[2] = '+uvm_set_config_string=*,"msg","hi"'
defines with parentheses
  argv[1] = '-define'
  argv[2] = 'WIDTH=(8+1)'
  argv[3] = '-define'
  argv[4] = "MODE='b101"

Every character arrived. The space did not split anything, the double quotes were not consumed, the parentheses were not treated as a subshell, and 'b101 — a Verilog sized literal containing a single quote — came through whole.

You wrote no escaping and no quoting, because there was nothing to escape for. With no shell involved, the list goes to the operating system as a list, and the tool receives it as a list. Quoting is a shell's way of describing a list using a single line of text; when you already have the list, the problem does not arise.

Each hazard is worth naming because each has bitten somebody:

The path with a space. Project directories get named by people. /proj/my design/ is unremarkable, and it turns a command string into a command with an extra argument.

The plusarg with quotes. +uvm_set_config_string takes a comma-separated value that frequently contains quotes of its own, and getting it through two levels of shell quoting is genuinely hard.

The define with parentheses. -define WIDTH=(8+1) is ordinary in RTL, and bare parentheses in a shell command are a syntax error or a subshell depending on the shell.

3. A String Without a Shell Is One Program Name

import subprocess

try:
    subprocess.run("fake_tool -version", capture_output=True, text=True)
except FileNotFoundError as exc:
    print("FileNotFoundError")
    print("  looked for:", repr(exc.filename))

FileNotFoundError
  looked for: 'fake_tool -version'

Read the second line. Python looked for a program whose name is the fourteen characters fake_tool -version, flag and space included, and reasonably did not find one.

That is not a quirk — it is the direct consequence of Section 1. subprocess.run is given the program and its arguments; with a string and no shell, the whole string is the program name, because nothing was asked to split it.

The error message is unusually good here, which is lucky: it quotes the filename, so the space is visible. A more common version of this mistake uses a program that does exist with no arguments, which works, and then stops working the day somebody adds a flag.

4. .split() Is Not a Parser

The obvious repair is to split the string yourself:

import shlex

WANTED = ["xrun", "-f", "/proj/my design/files.f", "+UVM_TESTNAME=axi_smoke"]

plain = "xrun -f /proj/my design/files.f +UVM_TESTNAME=axi_smoke"
quoted = 'xrun -f "/proj/my design/files.f" +UVM_TESTNAME=axi_smoke'

print("wanted        :", len(WANTED), "arguments")
print(".split()      :", plain.split())
print("  gave        :", len(plain.split()), "arguments")
print("shlex.split() :", shlex.split(quoted))
print("  correct     :", shlex.split(quoted) == WANTED)

wanted        : 4 arguments
.split()      : ['xrun', '-f', '/proj/my', 'design/files.f', '+UVM_TESTNAME=axi_smoke']
  gave        : 5 arguments
shlex.split() : ['xrun', '-f', '/proj/my design/files.f', '+UVM_TESTNAME=axi_smoke']
  correct     : True

.split() produced five arguments where the command has four, and the damage is specific: the tool is handed -f /proj/my, which is a filelist that does not exist, and then design/files.f as a stray positional argument it will probably reject with a confusing message.

shlex.split knows the quoting rules a shell uses, so given a correctly quoted string it recovers the real list. That is the right tool when a command arrives as a string — from a config file, a legacy Makefile, a cmd.sh somebody committed — and it is the wrong tool for a command you are building, because you would have to add the quoting first in order to parse it back off.

5. shlex.join for Showing a Human

The other direction, and it is the one Chapter 23.10 is built on:

import shlex

CMD = ["xrun", "-64bit", "-f", "/proj/my design/files.f",
       "-define", "WIDTH=(8+1)", "-define", "MODE='b101",
       "+UVM_TESTNAME=axi_smoke"]

line = shlex.join(CMD)
print("as a shell line:")
print("  " + line)
print()
print("round trip      :", shlex.split(line) == CMD)
print("naive join      :", " ".join(CMD) == line)

as a shell line:
  xrun -64bit -f '/proj/my design/files.f' -define 'WIDTH=(8+1)' -define 'MODE='"'"'b101' +UVM_TESTNAME=axi_smoke

round trip      : True
naive join      : False

shlex.join adds exactly the quoting a shell needs and no more — the plain arguments are untouched, the path with a space gets single quotes, and MODE='b101 becomes 'MODE='"'"'b101'.

That last one is worth staring at. A single quote cannot appear inside single quotes, so the standard trick is to close the quoting, emit an escaped quote, and reopen it. Nobody writes that correctly by hand, and shlex.join does it every time.

" ".join(CMD) produces a line that looks right and is not — it is the string .split() would mangle in Section 4, with the quoting removed. If a command is being written somewhere a human will copy it, it has to be shlex.join.

6. Building a Command From Variables

Real commands are assembled, not typed:

import sys

for index, arg in enumerate(sys.argv[1:], 1):
    print(f"  argv[{index}] = {arg!r}")

import shlex
import subprocess
import sys


def compile_cmd(filelist, top, defines, waves=False, incdirs=()):
    """The compile command for one design, as a list."""
    cmd = ["xrun", "-64bit", "-f", filelist, "-top", top]
    for directory in incdirs:
        cmd += ["-incdir", directory]
    for name in sorted(defines):
        cmd += ["-define", f"{name}={defines[name]}"]
    if waves:
        cmd += ["-access", "+rwc"]
    return cmd


cmd = compile_cmd(
    filelist="/proj/my design/files.f",
    top="tb_top",
    defines={"WIDTH": "(8+1)", "MODE": "'b101"},
    waves=True,
    incdirs=["/proj/rtl/include"],
)

for part in cmd:
    print(" ", repr(part))
print()
print("shell line:", shlex.join(cmd))
print()
print("as the tool sees it:")
run = subprocess.run([sys.executable, "echoargs.py"] + cmd[1:],
                     capture_output=True, text=True)
print(run.stdout.rstrip())

  'xrun'
  '-64bit'
  '-f'
  '/proj/my design/files.f'
  '-top'
  'tb_top'
  '-incdir'
  '/proj/rtl/include'
  '-define'
  "MODE='b101"
  '-define'
  'WIDTH=(8+1)'
  '-access'
  '+rwc'

shell line: xrun -64bit -f '/proj/my design/files.f' -top tb_top -incdir /proj/rtl/include -define 'MODE='"'"'b101' -define 'WIDTH=(8+1)' -access +rwc

as the tool sees it:
  argv[1] = '-64bit'
  argv[2] = '-f'
  argv[3] = '/proj/my design/files.f'
  argv[4] = '-top'
  argv[5] = 'tb_top'
  argv[6] = '-incdir'
  argv[7] = '/proj/rtl/include'
  argv[8] = '-define'
  argv[9] = "MODE='b101"
  argv[10] = '-define'
  argv[11] = 'WIDTH=(8+1)'
  argv[12] = '-access'
  argv[13] = '+rwc'

Four properties of that function are worth copying.

cmd += ["-define", value] adds two elements, which is what keeps a flag and its value separate. Writing cmd.append(f"-define {value}") would add one element containing a space, and the tool would receive a single argument it does not recognise.

sorted(defines) makes the command deterministic. Dict order is insertion order, so two runs assembling the same defines in different orders would produce two different cmd.sh files and a spurious diff — Chapter 21.3 Section 6's argument, applied to a command.

Optional parts are if statements, not string fragments. waves=True adds two elements and waves=False adds nothing, with no trailing space or empty argument to worry about.

incdirs=() is an empty tuple, not an empty list. Chapter 11.5's mutable-default rule: a default list would be shared between every call to the function.

7. The Form That Works on Your Machine

What this chapter is really arguing against:

FILELIST = "/proj/my design/files.f"
TOP = "tb_top"
DEFINES = {"WIDTH": "(8+1)"}

as_string = f"xrun -64bit -f {FILELIST} -top {TOP}"
for name, value in DEFINES.items():
    as_string += f" -define {name}={value}"

print("looks fine  :", as_string)
print("but splits to:")
for part in as_string.split():
    print("   ", repr(part))
print("arguments   :", len(as_string.split()), "— the tool wanted 8")

looks fine  : xrun -64bit -f /proj/my design/files.f -top tb_top -define WIDTH=(8+1)
but splits to:
    'xrun'
    '-64bit'
    '-f'
    '/proj/my'
    'design/files.f'
    '-top'
    'tb_top'
    '-define'
    'WIDTH=(8+1)'
arguments   : 9 — the tool wanted 8

The printed line looks correct. It is the line you would paste into a terminal to check, and in a terminal with the right quoting it would work. Built this way and handed to a shell, the filelist argument is wrong and there is one extra positional.

This form has a particular failure pattern worth recognising: it works until the data changes. On a project at /proj/mydesign/ it is fine for a year. The day somebody creates /proj/my design/, every script built this way breaks, and the error comes from the tool rather than from the script — so the first guess is that the filelist is corrupt.

The parenthesised define is the same story. Passed through a shell, -define WIDTH=(8+1) is a syntax error in sh and a subshell in some others, and the list form never meets the question.

8. When You Were Given a String

Sometimes the command is not yours to build — it arrives from a config file, or a cmd.sh from a previous run:

xrun -64bit -f '/proj/my design/files.f' -top tb_top -define 'WIDTH=(8+1)'

import shlex
from pathlib import Path

line = Path("cmd.sh").read_text(encoding="utf-8").strip()
print("the line  :", line)

cmd = shlex.split(line)
print("as a list :")
for part in cmd:
    print("   ", repr(part))
print("elements  :", len(cmd))

cmd += ["+UVM_TESTNAME=axi_smoke"]
print("extended  :", shlex.join(cmd))

the line  : xrun -64bit -f '/proj/my design/files.f' -top tb_top -define 'WIDTH=(8+1)'
as a list :
    'xrun'
    '-64bit'
    '-f'
    '/proj/my design/files.f'
    '-top'
    'tb_top'
    '-define'
    'WIDTH=(8+1)'
elements  : 8
extended  : xrun -64bit -f '/proj/my design/files.f' -top tb_top -define 'WIDTH=(8+1)' +UVM_TESTNAME=axi_smoke

Parse once, into a list; work with the list; serialise once at the end. Appending +UVM_TESTNAME=axi_smoke to the string would have been safe here and is not safe in general — a test name is data, and data in a command string is the quoting problem again.

The one real caution: shlex.split implements POSIX shell quoting. A line written for csh or tcsh, which several EDA flows still use, has different rules — so a command recovered from a .csh wrapper may not round-trip. Where that matters, keep the list as the source of truth and generate the shell line, never the other way round.

9. Predict, Run, Observe

import shlex

A = ["tool", "-f", "a b.f"]
B = "tool -f a b.f"
C = "tool -f 'a b.f'"

# A
print("A:", len(A), len(B.split()), len(shlex.split(C)))

# B
print("B:", B.split()[2:4])

# C
print("C:", shlex.split(C) == A)

# D
print("D:", shlex.join(A))

# E
print("E:", " ".join(A))

# F
print("F:", shlex.split(" ".join(A)) == A)

# G
print("G:", shlex.split(shlex.join(A)) == A)

# H
print("H:", shlex.join(["x", "WIDTH=(8+1)"]))

The verified output:

A: 3 4 3
B: ['a', 'b.f']
C: True
D: tool -f 'a b.f'
E: tool -f a b.f
F: False
G: True
H: x 'WIDTH=(8+1)'

A — three elements, four from a naive split, three again once the string is quoted and parsed properly.

B — the two halves of the filename, which is what the tool would receive as two arguments.

C — shlex.split on a correctly quoted string reconstructs the list exactly.

D — shlex.join adds the quotes that make that possible.

E — " ".join does not, and the result is B.

F — so a naive join followed by a proper split does not round-trip. The information was destroyed at the join.

G — shlex in both directions does.

H — parentheses are quoted too, because a shell would otherwise treat them as syntax.

10. Exercises

Exercise 1 — one element each. Build a six-element command and confirm with a stand-in child that the tool receives five arguments.

Exercise 2 — flag and value. Write -f files.f as one element and as two. Say what the tool receives in each case.

Exercise 3 — the three hazards. Pass a path with a space, a plusarg containing quotes and a define containing parentheses. Confirm all three arrive unchanged.

Exercise 4 — no escaping needed. Explain in two sentences why none of Exercise 3 required quoting.

Exercise 5 — the string. Call subprocess.run with a command string and no shell. Record the exception and what it says it looked for.

Exercise 6 — split the difference. Take a command whose path contains a space and compare .split() with shlex.split on a quoted version. Report both lengths.

Exercise 7 — what the tool gets. For the .split() version, say exactly which two arguments the tool receives and what it will do with each.

Exercise 8 — join for a human. Use shlex.join on a command containing a space, parentheses and a single quote. Paste the result into a shell and confirm it runs.

Exercise 9 — the naive join. Compare " ".join with shlex.join for the same command, then try to shlex.split each back.

Exercise 10 — build from variables. Write a compile_cmd function for your own flow with optional parts and sorted defines. Say why sorting matters.

Exercise 11 — the two-element habit. Replace a cmd.append(f"-define {x}") with the two-element form and say what changed for the tool.

Exercise 12 — the mutable default. Give your function a default list argument instead of a tuple and say what Chapter 11.5 predicts.

Exercise 13 — it works until the data changes. Take an f-string command that works, change one input to contain a space, and describe the error the tool would report.

Exercise 14 — a string you were given. Read a command from a file, parse it, add an argument, and write it back out. Confirm the round trip.

11. Summary

• One list element per argument. The first is the program; the rest become the tool's argv[1:], in order, unchanged.
• A flag and its value are two elements, because that is how the tool's own parser reads them.
• The list form survives a path with a space, a plusarg with quotes and a define with parentheses — all three verified arriving byte for byte.
• You write no quoting, because there is no shell to quote for. Quoting is a shell's way of describing a list in one line of text.
• A string with no shell is one program name — subprocess.run("fake_tool -version") looks for a program called fake_tool -version.
• .split() is not a parser. One space in a filelist path turns four arguments into five, and the tool gets a path that does not exist plus a stray positional.
• shlex.split recovers a list from a correctly quoted string, which is right for a command that arrived as one.
• shlex.join produces the shell line, including the '"'"' dance for a single quote that nobody writes correctly by hand.
• " ".join looks right and destroys information — a naive join followed by a proper split does not round-trip.
• Build commands from variables with cmd += [flag, value], optional parts as if statements, and sorted() over any dict so two runs produce the same command.
• An empty tuple, not an empty list, as a default argument.
• An f-string command works until the data changes, and then the error comes from the tool, so the script is the last place anybody looks.
• If you find yourself adding quotes to a command in Python, you are in the wrong form.
• Keep the list as the source of truth and generate the shell line from it, not the other way round.

The next chapter is about the one thing the list form deliberately gives up — a shell — and how to get back each of the things a shell was doing for you.