Deciding What Failure Means

Part II · Module 17 · Chapter 17.7 · Deciding What Failure Means

1. The Question This Chapter Answers

Chapter 1.6 listed seven questions and said the fourth — what can go wrong with the input, and what should happen for each case — is the one that gets skipped. That chapter established that you must ask it. This one is about how to answer it, which turns out to be a procedure with three steps rather than an exercise in imagination:

1. Enumerate the ways the input can be wrong.
2. Classify each one: is it a result, or is it an infrastructure problem?
3. Assign a verdict and an exit status to each.

All three happen before any code. The output is a small table, and the code that follows is written against the table rather than inventing policy as it goes.

2. Enumerate: Four Structural Ways In

Every file-consuming script faces the same four shapes of bad input, regardless of the tool that produced the file:

Absent. There is nothing at the path. The job never launched, the queue rejected it, the run directory was cleaned, or the path is simply wrong.

Empty. The file exists and is zero bytes. The tool started, created its log, and died before writing a line — a licence failure, a missing shared library, a bad argument.

Truncated. The file has content and stops. The simulator was killed by the queue at its timeout, or the machine went down. This is the dangerous one, because truncated output is valid text that simply ends.

Foreign. The file is complete and well-formed and is not what you expected — last week's log, a different tool's format, or the same tool after an upgrade renamed a field.

Those four are structural: they follow from the fact that a file is produced by a separate process you do not control. To them you add whatever your specific flow has actually produced — and the way to find those is to ask a colleague what the last three infrastructure incidents looked like.

3. Classify: Result or Infrastructure

Now the step that most enumeration exercises miss, and the one the canonical annotation encodes:

# missing log   -> NO_LOG, exit 2
# empty log     -> EMPTY_LOG, exit 2
# no end marker -> NO_FINISH, exit 1

Two of those exit 2 and one exits 1. The split is not arbitrary.

A result means the design was tested and something is true about it. An infrastructure problem means the design was never tested at all.

A missing log and an empty log mean the simulation did not happen. No statement about the RTL is available — not "it failed", not "it passed", nothing. That is an infrastructure problem, and the people who need to know are whoever owns the farm.

A truncated log is genuinely harder, and reasonable teams differ. The run started, so something was tested, but the verdict is unknown. The annotation above files it as exit 1 — treating an unknown verdict as a failure, on the principle that a design is not proven clean until something says so. Another team might call it exit 2, on the principle that a queue timeout is a farm problem. Both are defensible; what is not defensible is leaving it undecided until 3am.

The reason this classification earns its own step:

4. Assign: the Table Before the Code

The result of the three steps is a table, and the table is the deliverable:

Six rows, written at a desk in five minutes, and every one of them is a decision somebody would otherwise make accidentally at the moment the code was typed.

Note the last row especially. The instinct for an unrecognised format is to do your best with it. The table says the opposite: a parser that cannot recognise its input should say so rather than extracting whatever it can, because a number derived from a format you did not understand is worse than no number.

The code then transcribes the table:

END_MARKER = "Simulation complete"
EXPECTED_HEADER = "Chronologic VCS simulator"


def classify(lines, exists=True):
    """Return (verdict, exit_status) for one run, per the agreed table."""
    if not exists:
        return "NO_LOG", 2
    if not lines:
        return "EMPTY_LOG", 2
    if EXPECTED_HEADER not in lines[0]:
        return "UNKNOWN_FORMAT", 2
    errors = 0
    finished = False
    for line in lines:
        if "UVM_ERROR" in line:
            errors += 1
        if END_MARKER in line:
            finished = True
    if not finished:
        return "NO_FINISH", 1
    if errors:
        return "FAIL", 1
    return "PASS", 0


HEADER = EXPECTED_HEADER
CASES = [
    ("clean",      [HEADER, "UVM_INFO ok", END_MARKER], True),
    ("failed",     [HEADER, "UVM_ERROR drv.sv(88)", END_MARKER], True),
    ("truncated",  [HEADER, "UVM_INFO half way"], True),
    ("empty",      [], True),
    ("missing",    [], False),
    ("foreign",    ["Xcelium(R) Logic Simulation", "xmsim: *E,BADARG"], True),
]

for name, lines, exists in CASES:
    verdict, status = classify(lines, exists)
    print(f"{name:<10} {verdict:<15} exit {status}")

clean      PASS            exit 0
failed     FAIL            exit 1
truncated  NO_FINISH       exit 1
empty      EMPTY_LOG       exit 2
missing    NO_LOG          exit 2
foreign    UNKNOWN_FORMAT  exit 2

Six inputs, six distinct answers, and the function is readable as the table it came from — the branches are in the table's order and nothing in the body decides anything the table did not.

5. A Failure State Needs a Decision, Not a Handler

The instinct after learning exception handling is to reach for try/except. The order matters, and it is the opposite of the instinct:

Catching an error is worthless until somebody has decided what the verdict is for that case.

A handler without a decision produces one of two shapes, both bad. Either it swallows the problem and the script reports a result it did not earn, or it crashes with a traceback that tells a flow wrapper nothing beyond "exit 1".

The table is what turns a caught error into an answer. FileNotFoundError is a fact; NO_LOG, exit 2 is what the team decided that fact means. One of those is in the language and the other is in your flow, and only the second is useful to the person reading the morning summary.

This also decides where the handling goes. Once the table exists, the code has a shape: check the cheap, certain conditions first — does it exist, is it empty — and reserve exception handling for the cases that genuinely can only be discovered by trying. Most of the table's rows are answered without catching anything at all.

6. You Cannot Enumerate Everything

The honest limit, because this chapter invites paralysis.

There is no complete list of ways an input can be wrong. A tool upgrade can produce something nobody imagined; a filesystem can hand you a file that is half one log and half another. Demanding a complete enumeration before writing code means never writing code.

The workable rule has two parts:

Enumerate the four structural cases plus everything you have actually seen. The structural four are free — they apply to every file-consuming script ever written. The flow-specific ones come from history, and asking a colleague about the last three incidents takes ten minutes and is more productive than imagination.

Make the unenumerated case loud. Whatever is left over must produce a distinct, visible outcome rather than being absorbed into a verdict. UNKNOWN_FORMAT in Section 4's table exists for exactly this: it is the row that catches what the other rows did not anticipate, and it exits 2 rather than guessing.

That second part is the one that matters. A script that handles five cases well and files the sixth under PASS is worse than one that handles three cases and shouts about everything else, because the first is confidently wrong and the second is merely incomplete.

7. The Exit Codes Are a Team Convention

One point of precision: 0, 1, 2 carry no meaning to Python. The language only distinguishes zero from non-zero. The three-way split is a convention, and it only works if every tool in the flow uses the same one.

A workable convention, and the one this curriculum uses:

• 0 — the question was answered and the answer is good.
• 1 — the question was answered and the answer is bad. A result.
• 2 — the question could not be answered. An infrastructure problem.

The value of agreeing it is that a wrapper around twenty scripts can then do something useful without knowing what any of them do:

RESULTS = [("smoke", 0), ("cdc", 1), ("burst", 2), ("arb", 0), ("reset", 2)]

passed = failed = broken = 0
for name, status in RESULTS:
    if status == 0:
        passed += 1
    elif status == 1:
        failed += 1
    else:
        broken += 1

print(f"submitted      : {len(RESULTS)}")
print(f"real results   : {passed + failed}  ({passed} pass, {failed} fail)")
print(f"never ran      : {broken}")
if broken:
    print("ESCALATE: infrastructure, not design")

submitted      : 5
real results   : 3  (2 pass, 1 fail)
never ran      : 2
ESCALATE: infrastructure, not design

Two of five jobs produced no result at all, and the wrapper says so rather than reporting a 40% pass rate. Module 22 Chapter 22.6 formalises this as a contract across every tool in a flow; the point here is why two different non-zero codes are needed at all.

8. Exercises

Reasoning, with one short program.

Exercise 1 — Enumerate for a real script

For a script in your flow, list the four structural failure cases plus every flow-specific one you or a colleague have actually seen. Aim for six.

Exercise 2 — Classify them

For each case, decide: result or infrastructure. Where you are unsure, write the argument for both and then pick one.

Exercise 3 — Build the table

Produce the six-row table from Section 4 for your own script, with a "why" column. The why column is the one to check with a colleague.

Exercise 4 — Find the undecided case

Take an existing script and find an input state nobody decided about. Determine what it currently does and whether that was intended.

Exercise 5 — The unrecognised format

Feed a parser a different tool's output and record what it reports. If it produces a verdict rather than refusing, write the row that should have existed.

Exercise 6 — Two non-zero codes

Write the wrapper from Section 7 for a set of results of your own. State what it would have reported if every failure used exit 1.

9. Related Tutorials

Where these failure states come from:

• GLS in Regression — the nightly context in which missing and truncated logs are routine rather than exotic.
• Automating GLS Triage — triage that depends on telling a real failure from an absent one.

What the verdicts feed:

• Coverage Closure and Signoff — a consumer for which "we could not measure" and "we measured zero" must never be confused.
• UVM Report Server — the producer whose format changes are the source of the foreign-input case.

10. Summary

• The procedure is enumerate, classify, assign — and all three happen before any code.
• Four structural failures apply to every file-consuming script: absent, empty, truncated, foreign.
• Flow-specific cases come from history, so ask a colleague about the last three incidents.
• Classify each as a result or an infrastructure problem: was the design tested at all?
• A regression that cannot tell those apart wakes the wrong team and leaves the real problem unreported.
• The deliverable is a table, with a verdict and an exit status per row, written at a desk in five minutes.
• An unrecognised format should refuse, not guess — a number from a format you did not understand is worse than no number.
• A failure state needs a decision, not a handler: catching an error is worthless until the verdict for that case is decided.
• You cannot enumerate everything, so cover the structural four plus what you have seen, and make the remainder loud.
• Exit codes are a team convention, not a language rule: 0 good result, 1 bad result, 2 no result.

The next chapter is the single line that makes any of this visible to something other than a human reading a terminal.