From One Log to Any Log

Part II · Module 20 · Chapter 20.5 · From One Log to Any Log

1. The Edit That Should Not Exist

After Chapter 20.4 the script begins:

LOG_PATH = "run.log"

To check runs/axi_smoke_1839274/sim.log you open the file, change that string, save, and run. To check the next one you do it again. Two engineers cannot run it at the same time on two different logs, and there is no way to put it in a nightly job that sweeps a directory.

The fix is one import and one subscript:

import sys
log_path = sys.argv[1]

2. sys.argv Is the Command Line, as a List

When the shell starts a Python program it hands the interpreter the words of the command. sys.argv is that list:

import sys

print("argv     :", sys.argv)
print("count    :", len(sys.argv))
print("script   :", sys.argv[0])
print("arguments:", sys.argv[1:])

import subprocess
import sys

for extra in ([], ["run.log"], ["run.log", "reset.log"]):
    done = subprocess.run([sys.executable, "argv_demo.py"] + extra,
                          capture_output=True, text=True)
    print(f"$ python3 argv_demo.py {' '.join(extra)}".rstrip())
    for line in done.stdout.strip().splitlines():
        print("  " + line)

$ python3 argv_demo.py
  argv     : ['argv_demo.py']
  count    : 1
  script   : argv_demo.py
  arguments: []
$ python3 argv_demo.py run.log
  argv     : ['argv_demo.py', 'run.log']
  count    : 2
  script   : argv_demo.py
  arguments: ['run.log']
$ python3 argv_demo.py run.log reset.log
  argv     : ['argv_demo.py', 'run.log', 'reset.log']
  count    : 3
  script   : argv_demo.py
  arguments: ['run.log', 'reset.log']

The driver above runs the checker as a child interpreter and captures what it printed, which is how every example in this chapter shows a command line and its result. sys.executable is the Python running the driver; it is passed to subprocess.run and never printed.

Three facts from that output, and the first is the one that catches everybody:

sys.argv[0] is the script, not your first argument. With no arguments at all, sys.argv has length 1 and is not empty. Your arguments start at index 1, which is why the canonical annotation reads sys.argv[1].

Everything is a string. ['run.log', 'reset.log'] would still be strings if they were ['1839274', '42'] — Chapter 2.11's conversion is your job, exactly as it was for a regex group in Chapter 19.8.

sys.argv[1:] is every argument, which is the slice you want when the count is not fixed. Section 7 uses it.

3. The Checker, Re-Pointed

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: bad handshake
UVM_ERROR drv.sv(104) @ 180400: timeout

UVM_INFO @ 0: reporter [RNTST] Running test axi_reset_test...
UVM_INFO scb.sv(120) @ 160000: 96 transactions compared

import sys

MARKER = "UVM_ERROR"

log_path = sys.argv[1]

errors = 0
with open(log_path, encoding="utf-8") as handle:
    for line in handle:
        if MARKER in line:
            errors += 1

print(f"{'FAIL' if errors else 'PASS'}  errors={errors}  log={log_path}")

import subprocess
import sys

for log in ["run.log", "reset.log"]:
    done = subprocess.run([sys.executable, "check.py", log], capture_output=True, text=True)
    print(f"$ python3 check.py {log}")
    print("  " + done.stdout.strip())

$ python3 check.py run.log
  FAIL  errors=2  log=run.log
$ python3 check.py reset.log
  PASS  errors=0  log=reset.log

Two logs, two answers, one unmodified script. That is the whole point of the chapter, and it is three lines of change from where Chapter 20.4 left off.

Note that MARKER stayed a constant. Chapter 20.4's test said the path changes every run and the marker changes once a project, and that is exactly how the two have been treated: one became an argument, the other did not.

4. What the Caller Sees When They Get It Wrong

A tool with an interface has a new failure mode: the person running it. Here is what the unguarded version does:

import sys

log_path = sys.argv[1]

errors = 0
with open(log_path, encoding="utf-8") as handle:
    for line in handle:
        if "UVM_ERROR" in line:
            errors += 1

print(f"{'FAIL' if errors else 'PASS'}  errors={errors}")

import subprocess
import sys

done = subprocess.run([sys.executable, "check.py"], capture_output=True, text=True)
print("$ python3 check.py")
print("  rc =", done.returncode)
print("  " + done.stderr.strip().splitlines()[-1])

$ python3 check.py
  rc = 1
  IndexError: list index out of range

IndexError: list index out of range is true and useless. It names a Python concept to someone who wanted to check a log, does not say what was expected, and does not say how to fix it. A colleague who gets this at 2am concludes the tool is broken — which, as an interface, it is.

Compare what it could say.

5. The Guard

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: bad handshake
UVM_ERROR drv.sv(104) @ 180400: timeout

import sys

MARKER = "UVM_ERROR"

if len(sys.argv) != 2:
    print(f"usage: {sys.argv[0]} <log-file>", file=sys.stderr)
    sys.exit(2)

log_path = sys.argv[1]

errors = 0
with open(log_path, encoding="utf-8") as handle:
    for line in handle:
        if MARKER in line:
            errors += 1

print(f"{'FAIL' if errors else 'PASS'}  errors={errors}  log={log_path}")

import subprocess
import sys

for extra in (["run.log"], [], ["missing.log"]):
    done = subprocess.run([sys.executable, "check.py"] + extra,
                          capture_output=True, text=True)
    shown = done.stdout.strip() or done.stderr.strip().splitlines()[-1]
    print(f"$ python3 check.py {' '.join(extra)}".rstrip())
    print(f"  rc = {done.returncode}")
    print("  " + shown)

$ python3 check.py run.log
  rc = 0
  FAIL  errors=2  log=run.log
$ python3 check.py
  rc = 2
  usage: check.py <log-file>
$ python3 check.py missing.log
  rc = 1
  FileNotFoundError: [Errno 2] No such file or directory: 'missing.log'

Three outcomes, three different things went wrong, three different return codes. Reading them in order is the chapter's point:

rc = 0, FAIL on stdout. The script worked perfectly. The simulation failed — and the script's own status is success, because it successfully determined that. Chapter 17.8 made this distinction and it is worth repeating: a checker that exits non-zero because the thing it checked was bad has conflated two different questions.

rc = 2, a usage line on stderr. The caller made the mistake. The message names the script, says what it wanted, and goes to stderr so it does not pollute a pipeline reading stdout — Chapter 17.8's stream discipline. Exit code 2 for a usage error is a convention, formalised in Chapter 22.4.

rc = 1, a FileNotFoundError. The world was not as expected: the argument was fine and the file is not there. Letting that exception escape is a deliberate choice, not an oversight — Chapter 14.4's argument. The traceback names the missing path exactly, and nothing a hand-written message could add would be more useful.

Note len(sys.argv) != 2 and not < 2. Checking for exactly one argument also catches python3 check.py a.log b.log, where a caller who expected both to be checked gets a usage message instead of a silent answer about only the first.

6. More Than One Log

sys.argv[1:] is every argument, so accepting many is almost free:

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: bad handshake
UVM_ERROR drv.sv(104) @ 180400: timeout

UVM_INFO @ 0: reporter [RNTST] Running test axi_reset_test...
UVM_INFO scb.sv(120) @ 160000: 96 transactions compared

UVM_INFO @ 0: reporter [RNTST] Running test axi_burst_test...
UVM_ERROR scb.sv(204) @ 90100: data mismatch at 0x40002000

import sys

MARKER = "UVM_ERROR"


def count(path):
    n = 0
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            if MARKER in line:
                n += 1
    return n


paths = sys.argv[1:]
if not paths:
    print(f"usage: {sys.argv[0]} <log-file> [<log-file> ...]", file=sys.stderr)
    sys.exit(2)

failed = 0
for path in paths:
    errors = count(path)
    if errors:
        failed += 1
    print(f"{path:<12} {'FAIL' if errors else 'PASS'}  errors={errors}")

print(f"{len(paths)} logs, {failed} failing")

import subprocess
import sys

done = subprocess.run([sys.executable, "checkall.py", "run.log", "reset.log", "burst.log"],
                      capture_output=True, text=True)
print("$ python3 checkall.py run.log reset.log burst.log")
for line in done.stdout.strip().splitlines():
    print("  " + line)
print("  rc =", done.returncode)

$ python3 checkall.py run.log reset.log burst.log
  run.log      FAIL  errors=2
  reset.log    PASS  errors=0
  burst.log    FAIL  errors=1
  3 logs, 2 failing
  rc = 0

A per-log row and a summary line. if not paths is Chapter 9.5's truthiness on an empty list, and it replaces the exact-count check because the number of logs is genuinely variable here.

This is also the first version that does something the shell cannot trivially do for you, and it is the shape every later chapter builds on: a loop over inputs, one record per input, one summary at the end.

7. What sys.argv Does Not Give You

Being clear about the boundary, because the next step is a module away:

No flag parsing. --verbose, -o report.csv and --marker UVM_FATAL are just strings in the list; recognising them is code you would have to write.

No types. A seed arrives as '1839274' and int() is yours to call.

No defaults. Every optional argument needs its own len() check.

No usage text. The one-line message in Section 5 was hand-written, and it will drift from the real arguments the first time somebody adds one.

No error messages for bad values. check.py --help reads --help as a log path and raises FileNotFoundError.

All five are solved properly by argparse in Chapter 22.2, and the reason this chapter does not reach for it is that a tool taking exactly one positional argument does not need it. One positional argument and a usage line is a complete, honest interface. The moment there are two optional flags, it stops being one, and that is the signal to move.

8. Predict, Run, Observe

import sys

print("A:", len(sys.argv))
print("B:", sys.argv[0])
print("C:", sys.argv[1:])
print("D:", [type(a).__name__ for a in sys.argv[1:]])
print("E:", sys.argv[1] if len(sys.argv) > 1 else "no argument")

import subprocess
import sys

for extra in ([], ["7", "sim.log"]):
    done = subprocess.run([sys.executable, "show.py"] + extra, capture_output=True, text=True)
    print(f"$ python3 show.py {' '.join(extra)}".rstrip())
    for line in done.stdout.strip().splitlines():
        print("  " + line)

The verified output:

$ python3 show.py
  A: 1
  B: show.py
  C: []
  D: []
  E: no argument
$ python3 show.py 7 sim.log
  A: 3
  B: show.py
  C: ['7', 'sim.log']
  D: ['str', 'str']
  E: 7

A — with no arguments the length is 1, not 0. The script's own name is always there.

B — argv[0] is the script as invoked, which is what a usage message should print.

C — the slice is empty rather than absent, so if not sys.argv[1:] is a clean test.

D — both arguments are str, including the one that looks like a number.

E — guarding with len(sys.argv) > 1 before subscripting is the whole of Section 5 in one expression.

9. Exercises

Exercise 1 — print the list. Write a script that prints sys.argv and run it with zero, one and two arguments. Record all three lists.

Exercise 2 — the off-by-one. Explain in one sentence why the first argument is sys.argv[1], and say what sys.argv[0] is useful for.

Exercise 3 — re-point the checker. Convert your Chapter 20.4 script to take the log path from the command line. Run it against two different logs with no edit between runs.

Exercise 4 — the unguarded failure. Run your checker with no argument. Record the exception and the return code, and write what a colleague would conclude from it.

Exercise 5 — the usage line. Add the guard. Confirm the message names the script, goes to stderr, and exits 2.

Exercise 6 — three outcomes. Produce all three from Section 5: a failing log, a missing argument and a missing file. Tabulate the return code and the stream each one used.

Exercise 7 — exactly, not at least. Change your guard from != 2 to < 2 and run with two log paths. Say what the caller gets and which version you would ship.

Exercise 8 — strings only. Pass a number as an argument and print its type. Say where the conversion belongs.

Exercise 9 — many logs. Accept any number of logs, print a row each and a summary line. Confirm it still works with exactly one.

Exercise 10 — the status question. Add sys.exit(failed) to your multi-log script and say what the shell sees after 256 failing logs. Then say what you would do instead.

Exercise 11 — the boundary. Try to add a --marker flag using only sys.argv. Write down everything you had to handle, and say which of the five gaps in Section 7 you hit first.

10. Summary

• sys.argv is the command line as a list of strings, handed to the program by the shell.
• sys.argv[0] is the script, so your first argument is sys.argv[1] and the list is never empty.
• Everything in it is a string, including arguments that look like numbers.
• sys.argv[1:] is every argument, and it is empty rather than absent when there are none.
• Three lines turn a script about one log into a tool for any log — an import, a subscript, and deleting a constant.
• A constant that changes every run was always an argument. The marker stayed a constant because it does not.
• An unguarded sys.argv[1] gives IndexError: list index out of range, which names a Python concept to somebody who wanted to check a log.
• Three outcomes, three return codes: the script worked (0, verdict on stdout), the caller erred (2, usage on stderr), the world erred (1, the exception escaping).
• A failing simulation is not a failing script. rc != 0 means the program could not do its job.
• Check != 2, not < 2, so a caller passing two logs to a one-log tool is told rather than half-served.
• A loop over sys.argv[1:] with one row per log and a summary line is the shape the rest of this module builds on.
• sys.argv gives no flags, no types, no defaults, no usage text and no error messages — all five are Chapter 22.2's argparse.
• One positional argument and a usage line is a complete interface. Two optional flags is the signal to move on.

The script is now a tool. The next chapter stops adding features to it and decides, before writing another line, what the parser inside it actually owes the code that calls it.