Python Part II · Module 22 · Building Command-Line VLSI Tools
Arguments: How a Script Learns What to Do
Chapter 20.5 gave a log checker one argument and said that one positional argument and a usage line is a complete interface. It is — and the moment a script needs two of them, or one of them is optional, the arrangement starts failing in ways that produce no error. This chapter takes the next two steps by hand: two positionals, then flags, writing the parser rather than importing one. The point is not that hand-rolling is wrong. It is that by the end you will have met every problem argparse solves, so the next chapter is a set of answers rather than a set of incantations.
Intermediate17 min readPythonPart IIAutomationVerification FlowTooling
Part II · Module 22 · Chapter 22.1 · Arguments: How a Script Learns What to Do
1. The List, and What It Is Made Of
Chapter 20.5 established the shape: sys.argv is the command line as a list of strings, argv[0] is the script, and your arguments start at index 1.
import sys
print("argv :", sys.argv)
print("count:", len(sys.argv))
print("types:", sorted({type(a).__name__ for a in sys.argv}))import subprocess
import sys
for extra in ([], ["axi_smoke", "1839274", "--waves"]):
done = subprocess.run([sys.executable, "show.py"] + extra, capture_output=True, text=True)
print(f"$ python3 show.py {' '.join(extra)}".rstrip())
for line in done.stdout.strip().splitlines():
print(" " + line)$ python3 show.py
argv : ['show.py']
count: 1
types: ['str']
$ python3 show.py axi_smoke 1839274 --waves
argv : ['show.py', 'axi_smoke', '1839274', '--waves']
count: 4
types: ['str']The thing worth noticing is the last line of each. --waves is not special. The shell split the command on spaces and handed Python four strings; nothing in sys.argv distinguishes a flag from a value from a filename. Every bit of meaning in a command line is something your code puts there.
2. Two Positional Arguments
import sys
test, seed = sys.argv[1], int(sys.argv[2])
print(f"running {test} with seed {seed}")
print(f"types: {type(test).__name__}, {type(seed).__name__}")import subprocess
import sys
for extra in (["axi_smoke", "1839274"], ["axi_smoke"], ["axi_smoke", "abc"]):
done = subprocess.run([sys.executable, "run_sim.py"] + extra, capture_output=True, text=True)
shown = done.stdout.strip() or done.stderr.strip().splitlines()[-1]
print(f"$ python3 run_sim.py {' '.join(extra)}")
print(f" rc={done.returncode} {shown}")$ python3 run_sim.py axi_smoke 1839274
rc=0 running axi_smoke with seed 1839274
types: str, int
$ python3 run_sim.py axi_smoke
rc=1 IndexError: list index out of range
$ python3 run_sim.py axi_smoke abc
rc=1 ValueError: invalid literal for int() with base 10: 'abc'That is the canonical annotation:
import sys
test, seed = sys.argv[1], int(sys.argv[2])Two unpackings and a conversion, and both failure modes raise rather than producing a wrong answer — which looks like the arrangement working. Section 3 is why it is not.
Note that int() is doing real work. Chapter 20.5 Section 2 established that every element of argv is a string, so a seed used in arithmetic, stored in a results table, or compared against a previous run has to be converted at the boundary — the same rule Chapter 19.8 Section 3 gave for a regex group and Chapter 21.2 Section 5 for a CSV field.
3. Position Is the Only Thing Identifying Them
The previous section's second failure looks like type checking. It is not — it is a coincidence:
import sys
test, sim = sys.argv[1], sys.argv[2]
print(f"running {test} on {sim}")import subprocess
import sys
for extra in (["axi_smoke", "xcelium"], ["xcelium", "axi_smoke"]):
done = subprocess.run([sys.executable, "two_str.py"] + extra, capture_output=True, text=True)
print(f"$ python3 two_str.py {' '.join(extra)}")
print(f" rc={done.returncode} {done.stdout.strip()}")$ python3 two_str.py axi_smoke xcelium
rc=0 running axi_smoke on xcelium
$ python3 two_str.py xcelium axi_smoke
rc=0 running xcelium on axi_smokeBoth succeed. The second one asks a simulator called axi_smoke to run a test called xcelium, reports no problem, and exits 0.
Section 2's version caught the swap only because a test name is not a valid integer. Two string arguments — a test and a simulator, a source and a destination, a golden log and a new one — have nothing distinguishing them, and the script cannot tell a mistake from an instruction.
That is the first real limit of positional arguments, and it gets worse with every one you add. With two, a user has to remember an order; with four, they have to remember an order they will get wrong, and the cost of getting it wrong is a run that completes and is about the wrong thing.
4. Flags, by Hand
A flag names its value, so order stops mattering and a typo becomes visible. Here is a parser for four of them:
import sys
DEFAULTS = {"test": None, "seed": 1, "sim": "xcelium", "waves": False}
def parse(argv):
"""--test NAME, --seed N, --sim NAME, --waves. Also accepts --key=value."""
opts = dict(DEFAULTS)
i = 0
while i < len(argv):
arg = argv[i]
if arg == "--waves":
opts["waves"] = True
i += 1
elif arg.startswith("--") and "=" in arg:
key, _, value = arg[2:].partition("=")
opts[key] = value
i += 1
elif arg.startswith("--"):
key = arg[2:]
if i + 1 >= len(argv):
raise SystemExit(f"error: {arg} needs a value")
opts[key] = argv[i + 1]
i += 2
else:
raise SystemExit(f"error: unexpected argument {arg!r}")
return opts
print(parse(sys.argv[1:]))import subprocess
import sys
CASES = [
["--test", "axi_smoke", "--seed", "4"],
["--test=axi_smoke", "--waves"],
["--waves", "--test", "axi_smoke"],
["--test"],
["axi_smoke"],
]
for args in CASES:
done = subprocess.run([sys.executable, "hand.py"] + args, capture_output=True, text=True)
shown = done.stdout.strip() or done.stderr.strip()
print(f"$ hand.py {' '.join(args)}")
print(f" rc={done.returncode} {shown}")$ hand.py --test axi_smoke --seed 4
rc=0 {'test': 'axi_smoke', 'seed': '4', 'sim': 'xcelium', 'waves': False}
$ hand.py --test=axi_smoke --waves
rc=0 {'test': 'axi_smoke', 'seed': 1, 'sim': 'xcelium', 'waves': True}
$ hand.py --waves --test axi_smoke
rc=0 {'test': 'axi_smoke', 'seed': 1, 'sim': 'xcelium', 'waves': True}
$ hand.py --test
rc=1 error: --test needs a value
$ hand.py axi_smoke
rc=1 error: unexpected argument 'axi_smoke'Seventeen lines and it genuinely works. Order no longer matters — the third case puts --waves first and gets the same result as the second. Both --test NAME and --test=NAME are accepted, because both are spellings people use. A flag with no value and a stray positional are both rejected.
The while loop rather than a for is what makes the two-token form possible: consuming a value means advancing the index by two, which a for over argv cannot do. Chapter 10.5's loop, used for the one thing it is better at.
5. Three Defects You Cannot See
import sys
DEFAULTS = {"test": None, "seed": 1, "sim": "xcelium", "waves": False}
def parse(argv):
opts = dict(DEFAULTS)
i = 0
while i < len(argv):
arg = argv[i]
if arg == "--waves":
opts["waves"] = True
i += 1
elif arg.startswith("--") and "=" in arg:
key, _, value = arg[2:].partition("=")
opts[key] = value
i += 1
elif arg.startswith("--"):
key = arg[2:]
if i + 1 >= len(argv):
raise SystemExit(f"error: {arg} needs a value")
opts[key] = argv[i + 1]
i += 2
else:
raise SystemExit(f"error: unexpected argument {arg!r}")
return opts
print(parse(sys.argv[1:]))import subprocess
import sys
CASES = [
(["--test", "axi_smoke", "--seed", "4"], "the seed should be a number"),
(["--sedd", "4"], "a misspelled flag"),
(["--seed", "abc"], "a seed that is not a number"),
]
for args, what in CASES:
done = subprocess.run([sys.executable, "hand.py"] + args, capture_output=True, text=True)
print(f"$ hand.py {' '.join(args):<28} ({what})")
print(f" rc={done.returncode} {done.stdout.strip() or done.stderr.strip()}")$ hand.py --test axi_smoke --seed 4 (the seed should be a number)
rc=0 {'test': 'axi_smoke', 'seed': '4', 'sim': 'xcelium', 'waves': False}
$ hand.py --sedd 4 (a misspelled flag)
rc=0 {'test': None, 'seed': 1, 'sim': 'xcelium', 'waves': False, 'sedd': '4'}
$ hand.py --seed abc (a seed that is not a number)
rc=0 {'test': None, 'seed': 'abc', 'sim': 'xcelium', 'waves': False}Three commands, three exit statuses of zero, three wrong results.
--seed 4 gives the string '4'. The default was the integer 1 and the supplied value is text, so the same key holds two types depending on whether the user passed it. The first seed + 1 downstream raises, and the traceback is nowhere near here.
--sedd 4 is accepted. It creates a key called sedd that nothing reads, and test stays None — so the script runs with no test, or crashes on the None, and the user's command looked right to them. A typo'd flag is not an error, because the parser has no idea which flags exist.
--seed abc is accepted. There is no validation anywhere, which is Chapter 21.7's subject arriving on the command line instead of in a file.
All three share a cause: the parser knows the syntax of a command line and nothing about this script's arguments. It can tell --x from a value; it cannot tell a real flag from a typo, or a number from a word, because nobody told it what to expect.
6. And What It Still Does Not Do
Beyond the three defects, five things a user will reasonably try:
import sys
DEFAULTS = {"test": None, "seed": 1, "sim": "xcelium", "waves": False}
def parse(argv):
opts = dict(DEFAULTS)
i = 0
while i < len(argv):
arg = argv[i]
if arg == "--waves":
opts["waves"] = True
i += 1
elif arg.startswith("--") and "=" in arg:
key, _, value = arg[2:].partition("=")
opts[key] = value
i += 1
elif arg.startswith("--"):
key = arg[2:]
if i + 1 >= len(argv):
raise SystemExit(f"error: {arg} needs a value")
opts[key] = argv[i + 1]
i += 2
else:
raise SystemExit(f"error: unexpected argument {arg!r}")
return opts
print(parse(sys.argv[1:]))import subprocess
import sys
CASES = [
(["-h"], "ask for help"),
(["--help"], "ask for help the other way"),
(["-t", "axi_smoke"], "a short flag"),
(["--waves", "--seed"], "a flag where a value should be"),
([], "no arguments at all"),
]
for args, what in CASES:
done = subprocess.run([sys.executable, "hand.py"] + args, capture_output=True, text=True)
shown = (done.stdout.strip() or done.stderr.strip()).replace("\n", " ")
print(f"$ hand.py {' '.join(args) or '(nothing)':<20} ({what})")
print(f" rc={done.returncode} {shown[:68]}")$ hand.py -h (ask for help)
rc=1 error: unexpected argument '-h'
$ hand.py --help (ask for help the other way)
rc=1 error: --help needs a value
$ hand.py -t axi_smoke (a short flag)
rc=1 error: unexpected argument '-t'
$ hand.py --waves --seed (a flag where a value should be)
rc=1 error: --seed needs a value
$ hand.py (nothing) (no arguments at all)
rc=0 {'test': None, 'seed': 1, 'sim': 'xcelium', 'waves': False}No help. -h is rejected as an unexpected argument and --help is treated as a flag wanting a value. Both are the first thing anybody types at an unfamiliar tool.
No usage text. Even the errors that are correct do not say what the script does accept, so a user who gets one has to open the source.
No short flags. -t is unrecognised, and a tool run fifty times a day will be asked for them.
No required arguments. The last case runs happily with no test at all and exits 0, because nothing says --test is mandatory.
No --. A test genuinely named --weird cannot be passed, because there is no way to say stop parsing flags.
Adding all five is not conceptually hard. It is roughly another sixty lines, it has to be right in every script in the flow, and every one of those lines is a place for the next defect.
7. The Point of Having Written It
import sys
DEFAULTS = {"test": None, "seed": 1, "sim": "xcelium", "waves": False}
def parse(argv):
opts = dict(DEFAULTS)
i = 0
while i < len(argv):
arg = argv[i]
if arg == "--waves":
opts["waves"] = True
i += 1
elif arg.startswith("--") and "=" in arg:
key, _, value = arg[2:].partition("=")
opts[key] = value
i += 1
elif arg.startswith("--"):
key = arg[2:]
if i + 1 >= len(argv):
raise SystemExit(f"error: {arg} needs a value")
opts[key] = argv[i + 1]
i += 2
else:
raise SystemExit(f"error: unexpected argument {arg!r}")
return opts
print(parse(sys.argv[1:]))import subprocess
import sys
WANTED = [
("accepts --flag value", ["--test", "axi_smoke"], True),
("accepts --flag=value", ["--test=axi_smoke"], True),
("order does not matter", ["--waves", "--test", "axi_smoke"], True),
("rejects a stray value", ["axi_smoke"], True),
("rejects a flag with no value", ["--test"], True),
("converts --seed to int", ["--seed", "4"], False),
("rejects a misspelled flag", ["--sedd", "4"], False),
("rejects a non-numeric seed", ["--seed", "abc"], False),
("answers -h", ["-h"], False),
]
print(f"{'behaviour':<30} {'hand-written':<13} argparse")
for label, _, works in WANTED:
print(f"{label:<30} {'yes' if works else 'no':<13} yes")
print()
print("have:", sum(1 for _, _, w in WANTED if w), "of", len(WANTED))behaviour hand-written argparse
accepts --flag value yes yes
accepts --flag=value yes yes
order does not matter yes yes
rejects a stray value yes yes
rejects a flag with no value yes yes
converts --seed to int no yes
rejects a misspelled flag no yes
rejects a non-numeric seed no yes
answers -h no yes
have: 5 of 9Five of nine, in seventeen lines of parser that every script in the flow would have to carry a copy of.
The four missing ones are not a coincidence. They are exactly the four that need the parser to know what this script's arguments are — their names, their types, whether they are required — rather than just how a command line is shaped. That knowledge is what argparse asks you to declare, and everything in the right-hand column follows from the declaration.
That is the canonical purpose of this chapter: the next one is an improvement you understand, because you have met each of the four problems it solves and watched three of them produce exit status 0.
8. Predict, Run, Observe
import sys
print("argv1:", sys.argv[1:])
print("types:", [type(a).__name__ for a in sys.argv[1:]])
print("count:", len(sys.argv))
if len(sys.argv) > 2:
print("swap?:", sys.argv[1], sys.argv[2])import subprocess
import sys
for extra in (["a", "b"], ["--flag", "v"], ["--flag=v"], ["7"], []):
done = subprocess.run([sys.executable, "probe.py"] + extra, capture_output=True, text=True)
print(f"$ probe.py {' '.join(extra) or '(nothing)'}")
for line in done.stdout.strip().splitlines():
print(" " + line)The verified output:
$ probe.py a b
argv1: ['a', 'b']
types: ['str', 'str']
count: 3
swap?: a b
$ probe.py --flag v
argv1: ['--flag', 'v']
types: ['str', 'str']
count: 3
swap?: --flag v
$ probe.py --flag=v
argv1: ['--flag=v']
types: ['str']
count: 2
$ probe.py 7
argv1: ['7']
types: ['str']
count: 2
$ probe.py (nothing)
argv1: []
types: []
count: 1a b — two strings, and nothing in the list says which is which.
--flag v — two elements, both strings. The leading dashes are characters in a string, not a marker Python understands.
--flag=v — one element. The two spellings reach the script completely differently, which is why the hand-written parser needed two branches.
7 — a string, always. Conversion is the script's job.
nothing — an empty slice and a count of 1, which is the shape Chapter 20.5 Section 8 established.
9. Exercises
Exercise 1 — nothing is special. Print sys.argv for a command containing a flag, a value and a filename. Say what distinguishes them in the list.
Exercise 2 — two positionals. Write the canonical two-argument script and run it with the arguments correct, missing and in the wrong order. Record all three.
Exercise 3 — the silent swap. Write a script taking two string positionals and swap them. Confirm it succeeds, and say what it did.
Exercise 4 — how many is too many. For a tool in your own flow, count its positional arguments and say at which one a user would start getting the order wrong.
Exercise 5 — the hand-written parser. Write the seventeen-line parser and confirm it handles --flag value, --flag=value and any order.
Exercise 6 — why while. Replace the while with a for over argv and say which form of flag you can no longer support.
Exercise 7 — the string seed. Pass --seed 4 and print the type. Then add one line of arithmetic downstream and record where it fails.
Exercise 8 — the typo. Pass a misspelled flag. Record the exit status, what the parser stored, and what the real flag ended up as.
Exercise 9 — no validation. Pass --seed abc. Say which chapter's subject this is and where the check belongs.
Exercise 10 — the five missing behaviours. Run -h, --help, a short flag, and no arguments at all. Record what each does.
Exercise 11 — count the cost. Estimate the lines needed to add help, short flags, required arguments and -- to your parser. Say what happens to that estimate across ten scripts.
Exercise 12 — the threshold. Name one script in your flow that should keep hand-parsing and one that should not. Justify each in a sentence.
10. Summary
sys.argvis a list of strings and nothing in it is special — a flag, a value and a filename are indistinguishable until your code says otherwise.- Two positionals is the canonical annotation, and both its failure modes raise, which makes the arrangement look sound.
- It is sound only by luck. Two string positionals can be swapped and the script succeeds, running the simulator as a test.
- One positional needs no flag; two want flags; three need them — with source-and-destination as the universal exception.
- A flag names its value, so order stops mattering and a seventeen-line
whileloop handles--flag value,--flag=valueand any ordering. - A
whileis required, because consuming a value advances the index by two. - Three silent defects, all exiting 0: the value is never converted, a misspelled flag is accepted as a new key, and nothing validates anything.
- A misspelled flag leaves the real one at its default, so the command looks right to the person who typed it.
- All three share one cause — the parser knows the shape of a command line and nothing about this script's arguments.
- Five more things it does not do: no
-h, no usage text, no short flags, no required arguments, no--. - Five behaviours of nine, and the four missing ones are exactly those needing the parser to know the argument names, types and requiredness.
- Declaring that knowledge is what
argparseasks for, and everything it gives back follows from the declaration. - Hand-parsing is still right below the threshold — one argument, one place, no flags.
The next chapter declares those four arguments once and gets all nine behaviours without writing a parser.
Where this fits
Part of the Python curriculum.