What Separates a Script From a Tool Your Team Trusts

Module 1 · Chapter 1.5 · What Separates a Script From a Tool Your Team Trusts

1. The Engineering Problem

A regression ran overnight. In the morning the dashboard is green, and the block is declared clean.

It was not clean. Eleven tests failed.

Here is what happened. The check script was written eighteen months ago by an engineer who has since moved on. It reads the logs, counts errors, and when it finds failures it does this:

print("REGRESSION FAILED")

Which it did. Eleven times, in the nightly log, where nobody was reading.

The flow that runs the script does not read text. It reads the exit status — the single number every program hands back to whatever launched it when it finishes. Zero means fine; anything else means something went wrong. The script printed its verdict and then finished normally, so it handed back zero.

The shell saw zero. The nightly flow saw zero. The dashboard turned green.

Nothing here is a Python problem. The script's logic was right — it found all eleven failures. What it lacked was a contract: an agreement about how it reports what it found.

2. The Contract

Every automation tool worth depending on answers four questions. Write them down before writing code, and you have a contract:

• What goes in?
• What comes out?
• What counts as failure?
• How does each audience find out which happened?

That last question is the one the §1 script got wrong, and it has two answers, because there are always two audiences.

Read Figure 1 as a promise the tool makes. Inputs are named, so the tool does not depend on what happens to be lying around. It decides one thing, so you can say what correct means. And it reports that decision three times over — once for the engineer reading at 09:00, once for whatever script consumes the results next, and once as a number the flow can test.

3. The Smallest Fix in the Curriculum

Here is the correction to §1:

print("REGRESSION FAILED")
sys.exit(1)

Read the second line as English:

• sys is the name of Python's connection to the system that launched it. (sys has to be brought in with an import at the top of the file — we are not showing that yet, and Module 12 covers it properly.)
• The dot in sys.exit means the exit action belonging to sys. The same way a dotted path in a testbench names something belonging to something else.
• The parentheses ( ) mean do it, with this, and the 1 inside is what gets handed back.
• So the line says: finish now, and hand back 1.

One means something went wrong. Zero would mean fine. That is the whole convention, and everything in the flow above this script already understands it.

The print stays. It is not redundant — it is the human channel, and the engineer at 09:00 needs it. What changed is that the flow now hears the same answer.

And the more usual shape, once there is a result to report on:

sys.exit(0 if result.passed else 1)

0 if result.passed else 1 picks one of two values: 0 when the run passed, 1 when it did not. It reads almost like the English sentence "zero if it passed, otherwise one" — which is the point. Written this way, the exit status cannot disagree with the result, because it is computed from the result rather than set by hand somewhere else.

Module 2 closes with your first script doing exactly this, and Module 12.4 develops it into a full contract with distinct codes for tests failed and could not run at all.

4. The Properties, One at a Time

The §1 script was not badly written. It was a personal script being asked to do a team's job. Here is the honest comparison:

Two things about that table.

First, the right-hand column is the syllabus. Inputs and paths are Module 7. Missing and malformed input is Modules 7 and 16. Structured results are Module 11. Exit status is Module 12. Reproducible commands are Module 13. Tests are Module 17. Handover is Module 21. Nothing in the right-hand column is advanced — it is just a list of small things, and this track is the list in order.

Second, the transition is never a rewrite. Nobody sits down to convert a personal script into a trusted tool. What happens is that a property goes missing at a bad moment — a green dashboard hiding eleven failures — and gets added. Knowing the list in advance simply means you can add them on purpose.

5. Proportional Engineering

Now the necessary correction, because a reader could leave this page believing every script needs all nine rows.

The rule of thumb from Chapter 1.4 applies directly: match the engineering to what the script decides and how many people depend on it. A tool that decides whether a regression passed sits at the top of that scale, which is exactly why it is the tool this curriculum builds most carefully.

6. A Common Wrong Approach

The mistake that produces the §1 bug, stated so you can recognise it in review: treating the printed output as the result.

It shows up in several disguises:

• printing FAIL and exiting normally — the original;
• writing the summary to a file and never checking whether the write succeeded;
• catching every possible error so the script "never crashes", which converts loud failures into silent wrong answers;
• reporting a count of zero when the log could not be found at all, so missing and clean produce the same output.

The last one is worth sitting with. If a log is missing because the simulation never started, and your tool reports zero errors, it has reported a pass for a run that did not happen. That is the same class of bug as §1 — the tool is describing the world it expected rather than the one it found. Chapter 7.4 makes telling those cases apart a skill, and Chapter 10.2 makes it a four-way decision.

7. Exercises

Reasoning only.

Exercise 1 — Write a contract

Take the candidate task you chose in Exercise 2 of Chapter 1.4 and write its contract in four lines: what goes in, what comes out, what counts as failure, and what exit status each outcome should produce. Four lines. This is the habit Chapter 2.2 turns into a routine.

Exercise 2 — Audit an exit status

Find a script in your flow that makes a pass/fail decision. Run it on something that should fail and then check what it handed back — in a shell, echo $? prints the exit status of the command that just finished. If it printed a failure and handed back zero, you have found a live instance of §1.

Exercise 3 — Score a script against the table

Pick a script your team depends on and mark each of the nine rows as personal-script or trusted-tool. Then pick the single row whose absence would cost the most, and say why. That row is usually not the one that looks worst.

Exercise 4 — Distinguish missing from clean

Take a check you rely on and ask what it reports when its input file does not exist at all. If that is indistinguishable from a clean result, describe the situation in which that would mislead somebody.

8. Related Tutorials

What a trustworthy parser has to get right:

• The UVM Report Server — the severities the contract in Figure 1 depends on reading correctly.
• Reporting and Debug Techniques — what a log is expected to contain, which is what makes missing distinguishable from clean.

The same contract, in another track's automation:

• Automating GLS Triage — defined inputs, a structured result and an honest verdict, applied to gate-level runs.

9. Summary

• Printing a failure is not reporting one. The exit status is the channel every program in the flow reads, and a tool that leaves it at zero has told the flow everything is fine.
• Every useful tool has a contract: what goes in, what comes out, what counts as failure, and how each audience finds out. Three outputs, for three readers — a person, the next script, and the flow.
• The exit status must be derived from the verdict, not written separately, because two places to update is one place to forget.
• The personal-script to trusted-tool transition is a list of small properties, not a rewrite: named inputs, honest behaviour on missing and malformed input, a structured result, a real exit status, a reproducible command, tests, and a successful handover. That list is this curriculum's table of contents.
• Match the engineering to the responsibility. Most scripts need none of this. The ones that decide whether a regression passed need all of it — and the dangerous case is a script whose job grew while its engineering did not.
• The review question is "if this were broken, how would we find out?" If the answer is "we would not", nothing else about the code matters yet.

You have now finished Module 1, and you should be able to say where Python sits in your flow, which layer a job belongs to, whether a task is worth automating, and what a trustworthy tool owes the people using it.

What you have not done is write anything. That is next, and it is deliberate: you now have a contract to write against. Module 2 — Your First VLSI Python Script starts by defining the input, output and failure behaviour of a simulation-log checker, and then builds it, line by line, until you can account for every character in it. By the end of Module 2 your first script will name its input, decide one thing, and exit with the truth.