The Command a Human Can Paste and Re-Run

Part II · Module 23 · Chapter 23.10 · The Command a Human Can Paste and Re-Run

1. A List Is Not Something You Can Paste

CMD = ["xrun", "-64bit", "-f", "files.f", "+UVM_TESTNAME=axi_smoke",
       "+ntb_random_seed=1839274", "-define", "DUMP_WAVES=1",
       "-l", "sim.log", "+arg with spaces", "-top", "tb_top"]

print(CMD)
print()
print("tokens:", len(CMD))

['xrun', '-64bit', '-f', 'files.f', '+UVM_TESTNAME=axi_smoke', '+ntb_random_seed=1839274', '-define', 'DUMP_WAVES=1', '-l', 'sim.log', '+arg with spaces', '-top', 'tb_top']

tokens: 13

That is correct, unambiguous and useless to a person at a shell prompt. Chapter 23.2 chose the list form precisely because it is unambiguous — and the price is that it is not a command any shell will accept.

So the obvious thing:

CMD = ["xrun", "-64bit", "-f", "files.f", "+UVM_TESTNAME=axi_smoke",
       "+ntb_random_seed=1839274", "-define", "DUMP_WAVES=1",
       "-l", "sim.log", "+arg with spaces", "-top", "tb_top"]

print(" ".join(CMD))

xrun -64bit -f files.f +UVM_TESTNAME=axi_smoke +ntb_random_seed=1839274 -define DUMP_WAVES=1 -l sim.log +arg with spaces -top tb_top

Pasteable, readable, and wrong.

2. The Naive Join Records a Different Command

import shlex

CMD = ["xrun", "-64bit", "-f", "files.f", "+UVM_TESTNAME=axi_smoke",
       "+ntb_random_seed=1839274", "-define", "DUMP_WAVES=1",
       "-l", "sim.log", "+arg with spaces", "-top", "tb_top"]

naive = " ".join(CMD)
read_back = shlex.split(naive)

print("the command that ran had :", len(CMD), "arguments")
print("the recorded line reads as:", len(read_back), "arguments")
print("round-trips               :", read_back == CMD)
print()
print("where they diverge:")
for a, b in zip(CMD[9:12], read_back[9:12]):
    print(f"   ran {a!r:22} recorded {b!r}")

the command that ran had : 13 arguments
the recorded line reads as: 15 arguments
round-trips               : False

where they diverge:
   ran 'sim.log'              recorded 'sim.log'
   ran '+arg with spaces'     recorded '+arg'
   ran '-top'                 recorded 'with'

Thirteen arguments were recorded as fifteen. One argument containing spaces was split into three, and everything after it shifted — so -top is now a value and tb_top is an argument on its own.

This is Chapter 23.3 Section 2's word-splitting, met from the other direction. There, the hazard was a string becoming the wrong list on the way in. Here it is a list becoming the wrong string on the way out, and it is harder to catch because nothing fails: the run was correct, only the record of it is wrong. Somebody pastes the recorded line, gets a different failure, and spends an hour on the difference.

3. shlex.join

import shlex

CMD = ["xrun", "-64bit", "-f", "files.f", "+UVM_TESTNAME=axi_smoke",
       "+ntb_random_seed=1839274", "-define", "DUMP_WAVES=1",
       "-l", "sim.log", "+arg with spaces", "-top", "tb_top"]

line = shlex.join(CMD)
print(line)
print()
print("round-trips:", shlex.split(line) == CMD)
print("tokens     :", len(shlex.split(line)), "of", len(CMD))

xrun -64bit -f files.f +UVM_TESTNAME=axi_smoke +ntb_random_seed=1839274 -define DUMP_WAVES=1 -l sim.log '+arg with spaces' -top tb_top

round-trips: True
tokens     : 13 of 13

shlex.join is the exact inverse of the shlex.split from Chapter 23.3: it produces a string that a shell — and shlex.split — reads back as the identical list.

That is the canonical annotation:

(run_dir / "cmd.sh").write_text(shlex.join(cmd))

And it stays readable, because it quotes only what needs quoting:

import shlex

CMD = ["xrun", "-64bit", "-f", "files.f", "+UVM_TESTNAME=axi_smoke",
       "-define", "DUMP_WAVES=1", "+arg with spaces", "-top", "tb_top"]

quoted = [token for token in CMD if shlex.quote(token) != token]
print("tokens            :", len(CMD))
print("tokens it quoted  :", len(quoted))
print("which             :", quoted)

tokens            : 10
tokens it quoted  : 1
which             : ['+arg with spaces']

One token out of ten. A tool that quoted everything defensively would produce a line nobody reads; shlex.join produces one that looks like what you would have typed, with quotes exactly where they change the meaning.

4. Why You Should Never Hand-Quote

import shlex

tricky = ["xrun", "-define", "MSG=it's here"]

line = shlex.join(tricky)
print("joined     :", line)
print("round-trips:", shlex.split(line) == tricky)
print()
print("an empty argument survives too:")
print("  quote('') :", shlex.quote(""))
print("  round-trip:", shlex.split(shlex.join(["a", "", "b"])) == ["a", "", "b"])
print()
print("and something that looks like an attack is just an argument:")
print(" ", shlex.join(["xrun", "-f", "a;rm -rf /"]))

joined     : xrun -define 'MSG=it'"'"'s here'
round-trips: True

an empty argument survives too:
  quote('') : ''
  round-trip: True

and something that looks like an attack is just an argument:
  xrun -f 'a;rm -rf /'

Look at that first line of output. 'MSG=it'"'"'s here' is correct shell quoting for an argument containing a single quote — close one quote, emit an escaped quote, open another — and almost nobody writes it right by hand. It is the standard idiom and it is unreadable, which is exactly why a library should produce it.

The empty argument is the quieter case. " ".join(["a", "", "b"]) produces a b, and the empty argument vanishes — the tool receives two arguments where three were sent. shlex.join emits '' and it survives.

And the last line connects to Chapter 23.3 Section 4: a token containing ; is quoted, so pasting the recorded command runs the tool with a strange filename rather than executing anything. The record inherits the list form's safety, which a hand-assembled string would not.

5. A cmd.sh That Actually Runs

import os
import shlex
import subprocess
import sys
from pathlib import Path

run_dir = Path("runs/axi_smoke_1839274")
run_dir.mkdir(parents=True, exist_ok=True)

cmd = [sys.executable, "-c", "import sys; print('argv:', sys.argv[1:])",
       "+arg with spaces", "+UVM_TESTNAME=axi_smoke"]

script = run_dir / "cmd.sh"
script.write_text("#!/bin/sh\n# written by the flow — paste-able, runnable\nexec "
                  + shlex.join(cmd) + "\n", encoding="utf-8")
os.chmod(script, 0o755)

print("the file:")
for line in script.read_text(encoding="utf-8").splitlines():
    print("   " + (line if not line.startswith("exec") else "exec <interpreter> " +
                   shlex.join(cmd[1:])))

cp = subprocess.run([f"./{script.name}"], cwd=run_dir, capture_output=True, text=True)
print()
print("running it :", cp.stdout.strip())
print("exit status:", cp.returncode)
print("executable :", os.access(script, os.X_OK))

the file:
   #!/bin/sh
   # written by the flow — paste-able, runnable
   exec <interpreter> -c 'import sys; print('"'"'argv:'"'"', sys.argv[1:])' '+arg with spaces' +UVM_TESTNAME=axi_smoke

running it : argv: ['+arg with spaces', '+UVM_TESTNAME=axi_smoke']
exit status: 0
executable : True

The arguments came back exactly as they went in — +arg with spaces is one argument on the other side of a file, a shell and a process boundary.

The display substitutes <interpreter> for the real path only because that path differs on every machine; the file on disk carries the real one, which is the point of recording it.

Three details in that script earn their place.

The shebang makes the file runnable rather than something you have to remember to pass to sh. With os.chmod(..., 0o755) it is a command — and Chapter 23.9 Section 4's shutil.which would now find it, where without the mode bits it would not.

exec replaces the shell with the tool rather than leaving a shell waiting around it. The practical consequence is signals and exit status: with exec, the tool's return code is the script's, and a SIGTERM from Chapter 23.8 Section 8's scheduler reaches the tool rather than its wrapper.

The comment says where the file came from. A file somebody finds in a run directory six months later should not be a mystery, and a cmd.sh with no provenance gets treated as something a human edited.

6. Two Records, Two Readers

import json
import shlex
import sys
from pathlib import Path


def record_run(run_dir, test, seed, cmd, status, rc):
    """cmd.sh for a person tonight; record.json for a program next month."""
    run_dir = Path(run_dir)
    run_dir.mkdir(parents=True, exist_ok=True)

    (run_dir / "cmd.sh").write_text(
        "#!/bin/sh\n# written by the flow\nexec " + shlex.join(cmd) + "\n",
        encoding="utf-8")

    record = {"test": test, "seed": seed, "status": status, "rc": rc,
              "cmd": cmd, "run_dir": str(run_dir),
              "log": str(run_dir / "sim.log")}
    (run_dir / "record.json").write_text(json.dumps(record, indent=2) + "\n",
                                         encoding="utf-8")
    return record


record = record_run("runs/axi_smoke_1839274", "axi_smoke", 1839274,
                    ["xrun", "-64bit", "+UVM_TESTNAME=axi_smoke",
                     "+ntb_random_seed=1839274", "-l", "sim.log"],
                    "FAIL", 1)

print("record.json keys :", sorted(record))
print("cmd is a LIST    :", isinstance(record["cmd"], list), "of", len(record["cmd"]))
print()
loaded = json.loads(Path("runs/axi_smoke_1839274/record.json").read_text(encoding="utf-8"))
print("round-trips      :", loaded["cmd"] == record["cmd"])
print("re-runnable line :", shlex.join(loaded["cmd"]))

record.json keys : ['cmd', 'log', 'rc', 'run_dir', 'seed', 'status', 'test']
cmd is a LIST    : True of 6

round-trips      : True
re-runnable line : xrun -64bit +UVM_TESTNAME=axi_smoke +ntb_random_seed=1839274 -l sim.log

Both files, every run, and they are not duplicates — they have different readers.

cmd.sh is for a person at a prompt tonight. It needs no tooling, no parsing and no explanation; it is a command.

record.json is for a program next month comparing four hundred runs — Chapter 21.4's structured record. It stores cmd as a list, which matters: JSON round-trips a list exactly, so the stored command can be fed straight back to subprocess.run with no re-parsing, and shlex.join regenerates the human form whenever it is wanted.

Store the list, derive the string. The reverse — storing the joined string and splitting it back — works too, because shlex round-trips, but it puts a parsing step between the record and its use for no gain.

7. What cmd.sh Does Not Record

One honest limit, and it is the one that catches people who trust the file completely.

import os
import shutil
import sys

ENV_KEYS = ["PATH", "LD_LIBRARY_PATH", "VCS_HOME", "LM_LICENSE_FILE", "UVM_HOME"]

print("environment keys that decide which tool a command finds:")
for key in ENV_KEYS:
    print(f"   {key:<18} set: {key in os.environ}")
print()
print("the same cmd.sh, run with a different PATH, finds a different tool.")
print("so a record needs the RESOLVED path, not just the name:")
print("   which('python3') resolved to something:", shutil.which("python3") is not None)
print("   which('xrun')    resolved to          :", shutil.which("xrun"))

environment keys that decide which tool a command finds:
   PATH               set: True
   LD_LIBRARY_PATH    set: False
   VCS_HOME           set: False
   LM_LICENSE_FILE    set: False
   UVM_HOME           set: False

Only PATH is set on this machine, because no EDA tools are installed here — the other four are the names a real flow would find populated, and whether each is set is the question, not what it contains.

cmd.sh records what was run, not what it ran with. Chapter 23.1 Section 3 established that the child inherits the parent's environment, and that inheritance is invisible in the command: xrun -64bit top.sv means a different binary, a different library and possibly a different version depending on PATH and LD_LIBRARY_PATH.

Three additions close most of the gap:

import json
import shutil
from pathlib import Path


def provenance(tool, env_keys=("PATH", "LD_LIBRARY_PATH", "VCS_HOME")):
    """What decided WHICH tool ran — alongside the command that ran it."""
    import os
    resolved = shutil.which(tool)
    return {"tool": tool,
            "resolved": resolved,
            "on_path": resolved is not None,
            "env_present": {k: (k in os.environ) for k in env_keys}}


info = provenance("python3")
print("keys captured :", sorted(info))
print("on_path       :", info["on_path"])
print("env_present   :", info["env_present"])
print("resolved recorded as a string:", isinstance(info["resolved"], str))

keys captured : ['env_present', 'on_path', 'resolved', 'tool']
on_path       : True
env_present   : {'PATH': True, 'LD_LIBRARY_PATH': False, 'VCS_HOME': False}
resolved recorded as a string: True

The resolved path itself is not printed because it differs per machine — but it is in the record, and that is the field that makes a six-month-old run interpretable.

The resolved path from Chapter 23.9's shutil.which, so the record says which binary, not just which name.

The relevant environment keys, by name and presence. Dumping the whole environment is the wrong answer: it is enormous, it changes constantly, and — the reason this module keeps returning to it — it can contain credentials. A named allow-list of the keys that affect tool selection is the version that is safe to write into a shared run directory.

The tool's own version, which only the tool can tell you — typically xrun -version captured with Chapter 23.5's capture_output. No simulator exists on this machine, so that call is named rather than demonstrated; the mechanism is exactly Chapter 23.5 Section 2's.

8. Predict, Run, Observe

import shlex

CMD = ["xrun", "-l", "sim.log", "+arg with spaces", "-top", "tb_top"]

# A
print("A:", len(CMD), len(shlex.split(" ".join(CMD))))

# B
print("B:", shlex.split(shlex.join(CMD)) == CMD)

# C
print("C:", shlex.join(CMD))

# D
print("D:", sum(1 for t in CMD if shlex.quote(t) != t))

# E
print("E:", repr(shlex.quote("")))

# F
print("F:", shlex.split(shlex.join(["a", "", "b"])) == ["a", "", "b"])

# G
print("G:", shlex.join(["xrun", "-f", "a;b"]))

# H
print("H:", shlex.split(shlex.join(["x", "it's"])) == ["x", "it's"])

The verified output:

A: 6 8
B: True
C: xrun -l sim.log '+arg with spaces' -top tb_top
D: 1
E: "''"
F: True
G: xrun -f 'a;b'
H: True

A — six arguments recorded by a naive join read back as eight. Section 2.

B — shlex.join round-trips exactly.

C — and quotes only the token that needs it.

D — one of six, so the line stays readable.

E — an empty argument becomes '' rather than disappearing.

F — so it survives the round trip.

G — a shell metacharacter is quoted into an ordinary argument, inheriting the list form's safety.

H — even a single quote inside an argument round-trips.

9. Exercises

Exercise 1 — unpasteable. Print a real command list and say why it cannot be pasted at a prompt.

Exercise 2 — count the divergence. Build a command with one space-containing argument, join it naively, split it back, and report both lengths.

Exercise 3 — the shifted argument. In that split-back list, find the argument that is now in the wrong position. Say what a tool would do with it.

Exercise 4 — nothing fails. Explain in two sentences why this bug is harder to notice than Chapter 23.3's.

Exercise 5 — worse than nothing. Argue in three sentences why an inaccurate cmd.sh costs more than having none.

Exercise 6 — the inverse. Show shlex.split(shlex.join(cmd)) == cmd for a command of your own with at least three awkward arguments.

Exercise 7 — how much it quotes. Count how many of your tokens shlex.quote changes. Say what the ratio buys a reader.

Exercise 8 — hand-quote it. Try to hand-write the shell quoting for an argument containing a single quote, then compare with shlex.join.

Exercise 9 — the empty argument. Show a naive join losing an empty argument and shlex.join keeping it.

Exercise 10 — runnable. Write a cmd.sh with a shebang, exec and mode 0o755. Run it and confirm the arguments survived.

Exercise 11 — why exec. Say what changes about exit status and signals when exec is there.

Exercise 12 — provenance. Add the comment line and say what a reader six months later concludes without it.

Exercise 13 — two readers. For cmd.sh and record.json, name the reader and the question each answers.

Exercise 14 — list, not string. Say why record.json should store cmd as a list.

Exercise 15 — different PATH. Describe a case where the same cmd.sh runs a different binary, and name the field that would have revealed it.

Exercise 16 — the allow-list. List the environment keys your flow should record and say why dumping the whole environment is wrong — including the one reason that is not about size.

Exercise 17 — what is still not captured. Name two things about a run that none of this records, and say what would be needed.

10. Summary

• A command list cannot be pasted at a prompt, which is the price Chapter 23.2 paid for being unambiguous.
• " ".join(cmd) records a different command. Thirteen arguments became fifteen, and an argument after the space-containing one became a value.
• Nothing fails when it happens — the run was right, only the record is wrong, so the cost lands on whoever trusts it.
• An inaccurate cmd.sh is worse than none, because its whole value is being believable.
• shlex.join is the exact inverse of Chapter 23.3's shlex.split and round-trips every time.
• It quotes only what needs quoting — one token in ten here — so the recorded line reads like something you would have typed.
• Never hand-quote. The correct form for an embedded single quote is 'it'"'"'s', which is standard, unreadable and almost never written right by hand.
• An empty argument survives shlex.join as '' where a naive join loses it silently.
• A quoted metacharacter stays an argument, so the record inherits the list form's safety.
• Give cmd.sh a shebang, exec and mode 0o755 — exec makes the tool's exit status and signals the script's own.
• Write a provenance comment, so nobody mistakes the file for something a human edited.
• Keep both cmd.sh and record.json — a person tonight and a program next month are different readers.
• Store cmd as a list in the record and derive the string, never the reverse.
• cmd.sh records what was run, not what it ran with. A different PATH means a different binary under the same command.
• Record the resolved path, a named allow-list of environment keys, and the tool's own version — never the whole environment, which is large, volatile and can contain credentials.
• Be exact about which part is exact. The command can be captured; the RTL moving underneath it cannot.

The last chapter of this module puts the whole of it behind one function that does not care which simulator you have.