How Python Starts Another Program

Part II · Module 23 · Chapter 23.1 · How Python Starts Another Program

1. A Note on the Examples

There is no simulator on the machine this curriculum is verified on, so every example in this module runs a small Python script standing in for the EDA tool. That substitution is honest in the way that matters: Python does not know or care what the child program is. subprocess.run(["xrun", "-version"]) and subprocess.run(["python3", "fake_xrun.py", "-version"]) go through exactly the same machinery, inherit the same things and hand back the same object.

The command shapes shown here are the real ones. What is not claimed anywhere in this module is any simulator's own behaviour — its flags, its output format, or what its exit codes mean. Those are in your tool's documentation, and Chapter 23.11 is about keeping them in one place.

2. What Comes Back

import sys

print("FakeSim version 2024.09")
sys.exit(0)

import subprocess
import sys

cp = subprocess.run([sys.executable, "fake_tool.py", "-version"],
                    capture_output=True, text=True)

print("type       :", type(cp).__name__)
print("returncode :", cp.returncode)
print("stdout     :", repr(cp.stdout))
print("stderr     :", repr(cp.stderr))
print("args       :", [a.split("/")[-1] for a in cp.args])

type       : CompletedProcess
returncode : 0
stdout     : 'FakeSim version 2024.09\n'
stderr     : ''
args       : ['python3', 'fake_tool.py', '-version']

That is the canonical annotation:

cp = subprocess.run(["xrun", "-version"])
cp.returncode

subprocess.run starts the program, waits for it to finish, and returns a CompletedProcess describing what happened. Four attributes matter:

returncode — the child's exit status, which is Chapter 22.4's number arriving from the other side. Your script wrote one in Module 22; now it reads one.

stdout and stderr — the child's two streams, as strings, if you asked for them with capture_output=True. Without that they are None, which Section 6 covers.

args — the command as it was actually run. Worth more than it looks: a command assembled from variables is reconstructible from the result, which is what Chapter 23.10 writes into the run directory.

sys.executable is the Python running this script. Using it rather than the string "python3" means the child is the same interpreter as the parent — which on a farm with three Pythons installed is the difference between a reproducible run and a mystery.

3. The Parent and the Child

The child starts as a copy of the parent's situation. Three things cross that boundary, and all three are set at the instant of launch:

import os
import sys

print("argv        :", sys.argv[1:])
print("sees SIM    :", os.environ.get("SIM", "<unset>"))
print("sees PATH   :", "PATH" in os.environ)
print("same cwd    :", os.getcwd() == sys.argv[1])

import os
import subprocess
import sys

os.environ["SIM"] = "xcelium"

cp = subprocess.run([sys.executable, "show_env.py", os.getcwd()],
                    capture_output=True, text=True)

for line in cp.stdout.strip().splitlines():
    print("  " + line.replace(os.getcwd(), "<the parent's cwd>"))

  argv        : ['<the parent's cwd>']
  sees SIM    : xcelium
  sees PATH   : True
  same cwd    : True

The working directory. The child starts wherever the parent was standing. That is why a tool writing simv and csrc/ drops them beside your script unless you say otherwise — Chapter 23.7.

The environment. Every variable the parent has, the child gets a copy of, including PATH — which is how the child is found at all, and why a missing licence variable is a launch failure rather than a tool bug. Chapter 23.9.

The arguments. The list you passed, which the child sees as its own sys.argv. Chapter 23.2 is about building that list correctly.

Note the .replace(os.getcwd(), ...) in the parent. The directory's real name is different on every machine, so the example compares paths rather than printing one — which is a habit worth keeping in any output a person will diff.

4. The Boundary Is One-Way

import os
import sys

os.chdir("/")
os.environ["CHILD_SET"] = "yes"

print("child moved to root  :", os.getcwd() == "/")
print("child set a variable :", os.environ.get("CHILD_SET"))

import os
import subprocess
import sys

before = os.getcwd()
os.environ["SIM"] = "xcelium"

cp = subprocess.run([sys.executable, "wanderer.py"], capture_output=True, text=True)
for line in cp.stdout.strip().splitlines():
    print("  " + line)

print()
print("parent cwd unchanged :", os.getcwd() == before)
print("parent sees CHILD_SET:", os.environ.get("CHILD_SET", "<unset>"))
print("child's exit status  :", cp.returncode)

  child moved to root  : True
  child set a variable : yes

parent cwd unchanged : True
parent sees CHILD_SET: <unset>
child's exit status  : 0

The child moved to the filesystem root and set a variable. The parent did neither.

This is the fact the rest of the module is built on, and it is worth stating in the form that makes the consequences obvious: a child process cannot change its parent, so everything the child needs must be arranged before it starts.

Three consequences, each a later chapter:

You cannot cd for a tool from inside Python by calling os.chdir in the child. The child is the tool; you do not control its code. The directory has to be chosen at launch — cwd= in Chapter 23.7.

You cannot source a setup script and then run the tool as two separate subprocess.run calls. The first child sets variables, exits, and takes them with it. Chapter 23.3 Section 5 is about what to do instead.

You cannot fix PATH after discovering the tool is not on it. By then the launch has already failed — Chapter 23.9.

5. The Exit Status Is the Child's

import sys

code = int(sys.argv[1])
print(f"doing work, will exit {code}")
sys.exit(code)

import subprocess
import sys

for code in ["0", "1", "2", "3"]:
    cp = subprocess.run([sys.executable, "exits.py", code], capture_output=True, text=True)
    print(f"child exited {code}  ->  cp.returncode = {cp.returncode}  "
          f"(parent still running: True)")

child exited 0  ->  cp.returncode = 0  (parent still running: True)
child exited 1  ->  cp.returncode = 1  (parent still running: True)
child exited 2  ->  cp.returncode = 2  (parent still running: True)
child exited 3  ->  cp.returncode = 3  (parent still running: True)

A child exiting non-zero does not stop the parent. subprocess.run reports the status and returns normally, and what to do about it is your decision — which is Chapter 23.4's subject, and the reason check=True exists as an opt-in rather than as the default.

That is the right default for a flow. A simulator exiting 1 is not an emergency; it is the result, and your script almost always wants to save the log and classify the run before deciding anything. Chapter 20.11's four outcomes are exactly that decision.

The numbers also mean whatever the tool says they mean. Module 22 established 0/1/2 as a contract for tools you write; a third-party simulator keeps its own scheme, and Chapter 23.11 is where that gets translated once per tool rather than everywhere.

6. Streams Go to the Parent's by Default

import sys

print("tool: starting")
print("tool: warning about something", file=sys.stderr)
print("tool: done")

import subprocess
import sys

print("--- captured")
cp = subprocess.run([sys.executable, "noisy.py"], capture_output=True, text=True)
print("stdout:", repr(cp.stdout))
print("stderr:", repr(cp.stderr))

print("--- not captured (the child writes where the parent would)")
cp2 = subprocess.run([sys.executable, "noisy.py"])
print("stdout attribute:", cp2.stdout)
print("stderr attribute:", cp2.stderr)
print("returncode      :", cp2.returncode)

--- captured
stdout: 'tool: starting\ntool: done\n'
stderr: 'tool: warning about something\n'
--- not captured (the child writes where the parent would)
tool: warning about something
tool: starting
tool: done
stdout attribute: None
stderr attribute: None
returncode      : 0

Without capture_output=True, the child inherits the parent's streams — its output goes wherever the parent's would go, interleaved with the parent's own. That is the right behaviour for a tool whose output a human is watching, and the stdout attribute is None because nothing was captured to put there.

Two things about the ordering in that uncaptured block. The tool's three lines appear before the parent's two, because subprocess.run waited for the child to finish before returning.

And the child's stderr line appears first, ahead of both its stdout lines, even though the child printed it second. That is Chapter 22.5 Section 5's buffering, now crossing a process boundary: this very example was itself run with its output captured, so the child inherited a pipe as its stdout and block-buffered it, while its stderr went straight through. Run the same thing in a terminal and the three lines appear in the order written. The interleaving of an uncaptured child's streams is a property of where the output is going, not of the child.

text=True is why cp.stdout is a string. Without it you get bytes, which is occasionally what you want and usually not; Chapter 23.5 covers the choice.

7. What the Rest of the Module Is

Every remaining chapter is one consequence of Sections 3 to 6:

The order is not arbitrary. 23.2 to 23.4 are the three things every single invocation needs; the rest are about doing it well on a tool that takes an hour, writes a gigabyte, and sometimes hangs.

8. Predict, Run, Observe

import os
import sys

print("args:", sys.argv[1:])
print("env :", os.environ.get("PROBE", "<unset>"))
sys.exit(len(sys.argv) - 1)

import os
import subprocess
import sys

# A
cp = subprocess.run([sys.executable, "po.py", "a", "b"], capture_output=True, text=True)
print("A:", cp.returncode)

# B
print("B:", cp.stdout.strip().splitlines()[0])

# C
print("C:", repr(cp.stderr))

# D
plain = subprocess.run([sys.executable, "po.py"], capture_output=True)
print("D:", type(plain.stdout).__name__)

# E
os.environ["PROBE"] = "set-by-parent"
cp2 = subprocess.run([sys.executable, "po.py"], capture_output=True, text=True)
print("E:", cp2.stdout.strip().splitlines()[1])

# F
print("F:", cp2.returncode)

# G
print("G:", [a.split("/")[-1] for a in cp.args])

The verified output:

A: 2
B: args: ['a', 'b']
C: ''
D: bytes
E: env : set-by-parent
F: 0
G: ['python3', 'po.py', 'a', 'b']

A — the child's sys.exit(2) became cp.returncode, and the parent carried on.

B — the two arguments arrived as the child's sys.argv[1:], unchanged.

C — an empty string, not None, because capture_output=True was given and the child wrote nothing there.

D — without text=True the captured output is bytes.

E — the variable set in the parent before launching was visible to the child.

F — zero arguments, so len(sys.argv) - 1 is 0 and the child exited 0.

G — cp.args is the list as run, which is what Chapter 23.10 records.

9. Exercises

Exercise 1 — the object. Run a program with capture_output=True and print all four attributes of the result.

Exercise 2 — sys.executable. Say what subprocess.run(["python3", ...]) would do differently on a machine with three Pythons installed.

Exercise 3 — three inheritances. Write a child that reports its working directory, one environment variable and its arguments. Confirm all three came from the parent.

Exercise 4 — compare, do not print. Change your child to compare paths rather than print them, and say why that matters for an output somebody will diff.

Exercise 5 — one-way. Have a child change directory and set a variable. Confirm the parent is unaffected by both.

Exercise 6 — the three things it does to. For each of cd, export and source, say in one sentence why running it as a child achieves nothing.

Exercise 7 — the status does not raise. Run a child that exits 3 and confirm the parent continues. Say why that is the right default for a flow tool.

Exercise 8 — whose meaning. Name a tool in your flow and say what its exit 1 means, from its documentation rather than from assumption.

Exercise 9 — captured and not. Run the same noisy child both ways. Record what the stdout attribute holds in each case.

Exercise 10 — bytes or text. Capture without text=True and print the type. Say when bytes is what you want.

Exercise 11 — read the map. For a tool invocation in your own flow, name which of Chapters 23.2 to 23.11 you already get wrong.

10. Summary

• Every tool invocation is a child process, and Python's machinery is identical whether the child is a simulator or a small script.
• subprocess.run starts the program, waits, and returns a CompletedProcess with returncode, stdout, stderr and args.
• sys.executable is the interpreter running your script — the right way to launch Python on a machine with several installed.
• The child inherits three things at launch: the working directory, a copy of the environment including PATH, and the argument list.
• The boundary is one-way. A child that changes directory or sets a variable changes neither for the parent.
• So everything the child needs must be arranged before it starts — which is why cwd=, the environment argument and PATH checks all exist.
• subprocess.run(["cd", run_dir]) does nothing, and neither does running export or source as a child.
• A non-zero exit status does not stop the parent. subprocess.run reports it and returns, which is the right default because a failing simulation is a result rather than an emergency.
• The numbers mean whatever the tool says they mean — Module 22's 0/1/2 contract applies to tools you write.
• Without capture_output=True the child inherits the parent's streams, and the attributes are None because nothing was captured.
• Without text=True captured output is bytes.
• Print comparisons, not paths, in anything a person will diff.

The next chapter is the one thing every invocation needs first: building the command as a list.