Makefile · Part II · Module 6 · UVM and Regression Automation
Pass/Fail Collection and Regression Reports
Forty runs finished and left forty verdict files. Nobody reads forty files. This chapter builds the one thing the person who launched the regression actually wants — a summary and an exit status — and shows why a run has four possible states rather than two, with all four measured.
Advanced28 min readMakefileGNU MakePart IIRegressionReporting
A regression is only useful if somebody reads the result, and nobody reads forty log files. The question a lead asks in the morning is one sentence long — did it pass? — and the answer has to be one line plus a number, with the detail available underneath for whoever has to fix it.
It also has to be an answer a script can read. A nightly job, a CI pipeline, a merge gate: all of them ask by running a command and looking at its exit status. So the report has two outputs, and the second one matters as much as the first.
1. What 6.3 Left Behind
Chapter 6.3 §7 made a deliberate choice: a run always succeeds, and writes its verdict into a file.
build/axi_burst/1/result.txt
build/axi_burst/2/result.txt
build/axi_decode_err/1/result.txt
build/axi_decode_err/2/result.txt
build/axi_flaky/1/result.txt
build/axi_flaky/2/result.txt
build/axi_random/1/result.txt
build/axi_random/2/result.txt
build/axi_smoke/1/result.txt
build/axi_smoke/2/result.txtEach file holds one word, PASS or FAIL. Collecting them looks like it should be a one-liner, and the naive one-liner has a hole in it that this chapter is mostly about.
2. The Naive Collection, and Its Hole
The obvious approach counts the files it finds:
report:
@p=`grep -l PASS $(BUILD)/*/*/result.txt 2>/dev/null | wc -l`; \
f=`grep -l FAIL $(BUILD)/*/*/result.txt 2>/dev/null | wc -l`; \
echo "$$p passed, $$f failed"; \
[ "$$f" -eq 0 ]For a regression where everything ran, this is correct and it is what many real flows contain. The hole is what happens when a run did not produce a verdict file at all. Then it is in neither count, and the totals quietly describe a smaller regression than the one you asked for.
Two ways that happens, and they are not the same problem:
- The run never started. Make was interrupted, the farm killed the job, the disk filled, somebody pressed Ctrl-C.
- The run started and died partway, leaving a log but no verdict.
Both are invisible to a report that only counts what it finds. And both are far more common on a real regression than on a toy one, because real regressions get interrupted.
3. Four States, Not Two
Once the report walks the expected list, each run resolves to one of four states:
| Files present in the run directory | State | What it means |
|---|---|---|
result.txt says PASS | PASS | Ran and checked out |
result.txt says FAIL | FAIL | Ran and did not |
run.log but no result.txt | INCOMPLETE | Started, died before the verdict |
| Neither file | NOTRUN | Never started |
The two middle-of-the-night states are the interesting ones, and separating them is worth the few extra lines. INCOMPLETE means the simulator ran and there is a log to look at — often the log explains itself, because the run was killed for running out of time or memory. NOTRUN means there is nothing to look at and the question is why the job never launched. Different problem, different place to look, so they should not share a bucket.
Here is the report:
RESULTS := $(foreach t,$(TESTS),$(foreach s,$(SEEDS),$(BUILD)/$(t)/$(s)/result.txt))
REPORT := $(BUILD)/report.txt
.PHONY: regress report report-only
regress: $(RESULTS)
report: regress
@$(MAKE) --no-print-directory -f $(firstword $(MAKEFILE_LIST)) report-only
# Walks the EXPECTED runs, not the files that happen to exist.
report-only:
@mkdir -p $(BUILD)
@: > $(REPORT)
@for t in $(TESTS); do for s in $(SEEDS); do \
d=$(BUILD)/$$t/$$s; \
if [ -f $$d/result.txt ]; then st=`cat $$d/result.txt`; \
elif [ -f $$d/run.log ]; then st=INCOMPLETE; \
else st=NOTRUN; fi; \
printf '%-16s %-5s %s\n' "$$t" "$$s" "$$st" >> $(REPORT); \
done; done
@echo ""
@echo " ================= regression report ================="
@sed 's/^/ /' $(REPORT)
@echo " ====================================================="
@p=`grep -c 'PASS$$' $(REPORT) || true`; \
f=`grep -c 'FAIL$$' $(REPORT) || true`; \
i=`grep -c 'INCOMPLETE$$' $(REPORT) || true`; \
n=`grep -c 'NOTRUN$$' $(REPORT) || true`; \
tot=`wc -l < $(REPORT) | tr -d ' '`; \
printf " %s passed %s failed %s incomplete %s not run (of %s)\n" \
"$$p" "$$f" "$$i" "$$n" "$$tot"; \
if [ "$$f" -gt 0 ] || [ "$$i" -gt 0 ] || [ "$$n" -gt 0 ]; \
then echo " RESULT: FAILED"; exit 1; else echo " RESULT: PASSED"; fiThree details are deliberate. The for loops walk $(TESTS) and $(SEEDS), so a run that produced nothing still gets a row. The verdict file is read with cat rather than pattern-matched, so a corrupt file shows up as nonsense in the report instead of being silently counted as a pass. And INCOMPLETE and NOTRUN both fail the regression — a run whose outcome is unknown is not a pass.
4. The Report, Running
Everything passing, five tests at one seed:
[compile] -> build/simv
[PASS] axi_smoke seed=1
[PASS] axi_burst seed=1
[PASS] axi_random seed=1
[PASS] axi_decode_err seed=1
[PASS] axi_flaky seed=1
================= regression report =================
axi_smoke 1 PASS
axi_burst 1 PASS
axi_random 1 PASS
axi_decode_err 1 PASS
axi_flaky 1 PASS
=====================================================
5 passed 0 failed 0 incomplete 0 not run (of 5)
RESULT: PASSED
make rc=0A seed where one test fails:
================= regression report =================
axi_smoke 5 PASS
axi_burst 5 PASS
axi_random 5 PASS
axi_decode_err 5 PASS
axi_flaky 5 FAIL
=====================================================
4 passed 1 failed 0 incomplete 0 not run (of 5)
RESULT: FAILED
make: *** [report] Error 1
make rc=2The whole sweep completed — no -k, nothing skipped — and the regression still failed, loudly, with a non-zero status a script can act on. That is the arrangement Chapter 6.3 §7 was building towards.
And a sweep wide enough to localise the failure, six seeds across five tests:
axi_decode_err 5 PASS
axi_decode_err 6 PASS
axi_flaky 1 PASS
axi_flaky 2 PASS
axi_flaky 3 PASS
axi_flaky 4 PASS
axi_flaky 5 FAIL
axi_flaky 6 FAIL
=====================================================
28 passed 2 failed 0 incomplete 0 not run (of 30)
RESULT: FAILEDThirty runs, and the report pins the problem to one test at two specific seeds. Both are reproducible with the command Chapter 6.2 §8 put in the failure message.
5. Proving the Other Two States Exist
A report with four branches where two never fire is a report with two branches and some decoration. Both of the unusual states have to be demonstrated, not assumed.
INCOMPLETE. Simulate a run that died after writing its log but before recording a verdict, by deleting the verdict file and leaving the log:
make report SEEDS="1 2" # run everything first
rm -f build/axi_burst/2/result.txtNOTRUN. Ask the report about a seed nothing ever ran — using report-only, which reads results without launching anything:
make report-only SEEDS="1 2 99" ================= regression report =================
axi_smoke 1 PASS
axi_smoke 2 PASS
axi_smoke 99 NOTRUN
axi_burst 1 PASS
axi_burst 2 INCOMPLETE
axi_burst 99 NOTRUN
axi_random 1 PASS
axi_random 2 PASS
axi_random 99 NOTRUN
axi_decode_err 1 PASS
axi_decode_err 2 PASS
axi_decode_err 99 NOTRUN
axi_flaky 1 PASS
axi_flaky 2 PASS
axi_flaky 99 NOTRUN
=====================================================
9 passed 0 failed 1 incomplete 5 not run (of 15)
RESULT: FAILED
make: *** [report-only] Error 1
make rc=2Three of the four states in one table, each in the right row, and the regression correctly fails although not one test actually failed. Nine passes and zero failures, and the answer is still FAILED, because six runs have no known outcome. A report that counted only what it found would have said "9 passed, 0 failed" and exited 0.
6. A Report Is Only As Complete As the Question
Here is a trap I walked into while measuring the section above. After deleting build/axi_burst/2/result.txt, I asked for the report without naming the seeds:
================= regression report =================
axi_smoke 1 PASS
axi_burst 1 PASS
axi_random 1 PASS
axi_decode_err 1 PASS
axi_flaky 1 PASS
=====================================================
5 passed 0 failed 0 incomplete 0 not run (of 5)
RESULT: PASSED
make rc=0RESULT: PASSED, exit 0 — with a broken run sitting on disk at seed 2. Nothing is wrong with the report. SEEDS defaulted to 1, so the question was "how did seed 1 go?", and the honest answer to that question is "fine".
The lesson is about how such a report is used rather than how it is written: it describes exactly the cross product you name and nothing else. Which means the seed list is part of the result. A CI job that runs make report SEEDS="1 2 3 4 5 6 7 8" and a developer who later types make report-only are asking different questions and will correctly get different answers.
The practical defences are cheap:
@echo " tests: $(words $(TESTS)) seeds: $(SEEDS)"
@echo " list : $(TESTLIST)"Print the scope in the report header, and have CI pass its seed list explicitly rather than relying on a default. A report that states the question it answered cannot be misread as answering a bigger one.
7. Recursive Make Does Not Inherit -f
The report target above calls Make again:
report: regress
@$(MAKE) --no-print-directory -f $(firstword $(MAKEFILE_LIST)) report-onlyThat -f looks redundant and it is not. A sub-make does not inherit -f. Measured:
MAKEFILE_LIST = [ probe.mk]
MAKEFLAGS = []
-- now a bare sub-make: which file does it read? --
make[1]: *** No rule to make target `whoami'. Stop.MAKEFLAGS is empty. Make passes many flags down to sub-makes through that variable, but -f is not one of them, so a bare $(MAKE) report-only goes looking for a file called Makefile — and if one exists, it reads that instead.
That is how this chapter's Makefile broke the first time it ran, and the symptom was baffling:
[compile] -> build/simv
[run] axi_smoke seed=1 -> build/axi_smoke/1/run.log
[PASS] axi_smoke seed=1
make[1]: *** No rule to make target `build/axi_burst/1/passed.stamp'.
make[1]: *** No rule to make target `build/axi_random/1/passed.stamp'.
make[1]: *** No rule to make target `build/axi_decode_err/1/passed.stamp'.
make[1]: *** No rule to make target `build/axi_flaky/1/passed.stamp'.One test worked and four did not. The explanation is almost funny: the sub-make read the plain Makefile from 6.1, which has no pattern rule — but it does have a concrete rule for $(BUILD)/$(TEST)/$(SEED)/passed.stamp, and with TEST and SEED at their defaults that expands to exactly build/axi_smoke/1/passed.stamp. The one target that matched the defaults built fine. The other four had no rule at all.
Two fixes, and the second is better:
# Fix 1: name the file explicitly. $(firstword ...) because MAKEFILE_LIST
# has a leading space, so $(MAKEFILE_LIST) alone can expand with it.
THIS := $(firstword $(MAKEFILE_LIST))
report: regress
@$(MAKE) --no-print-directory -f $(THIS) report-only
# Fix 2: do not recurse. `report` already depends on `regress`, and recipe
# lines of a target run after all its prerequisites are built, so the
# collection can simply be the rest of this recipe.Fix 2 is the one to prefer. The recursion in this flow buys nothing — report: regress already guarantees ordering — and every $(MAKE) inside a recipe is a place where flags, variables and the makefile name can fail to travel. The version here keeps the sub-make only because report-only is genuinely useful as a separate entry point.
8. The Exit Status Is the Interface
Everything in this chapter exists to make one line true: make report exits 0 if and only if every run in the named sweep passed.
That single property is what lets the regression be used by things that cannot read:
# A nightly job that only emails on failure
make -j8 report SEEDS="1 2 3 4 5 6 7 8" || mail -s "regression FAILED" team@example.com < build/report.txt
# A merge gate
make -j8 report SEEDS="1 2 3 4" && echo "safe to merge"
# A chain where each step depends on the last
make compile && make -j8 report && make coverageThree rules keep that property true, and all three were earned earlier in Part II:
Only one place decides. The runs record data; the report alone sets the status. Two targets that both decide will eventually disagree.
An unknown outcome is a failure. INCOMPLETE and NOTRUN both fail, because a run that did not finish is not evidence of anything (§3).
Never trust the tool's own status. Chapter 5.4 §5 measured vvp exiting 0 on a run that printed UVM_ERROR, and Chapter 5.1 §4 measured the same asymmetry. The verdict comes from reading the log, every time.
One more property is worth stating because it is easy to lose: the report must not depend on the order runs finished in. It does not, because it walks $(TESTS) and $(SEEDS) in a fixed order rather than listing files. Measured against a parallel run of the same sweep:
serial : bb96e0174e73a90651f2c95aaf1aa3aa
-j8 : bb96e0174e73a90651f2c95aaf1aa3aaIdentical. A report whose text changes with the scheduling is a report nobody can diff against yesterday's.
9. Common Mistakes
Counting only the result files that exist. A regression where most runs were killed reports a small, clean, passing result (§2).
Treating "no result" as "did not fail". An unknown outcome has to fail the regression (§3).
Lumping INCOMPLETE and NOTRUN together. One has a log that usually explains itself; the other needs you to look at why the job never launched (§3).
Leaving a verdict branch untested. Four states with two that never fire is two states and some decoration — delete a result.txt and ask about an unrun seed to prove the others (§5).
Forgetting that the report is scoped to the sweep you name. Measured: RESULT: PASSED with a broken run on disk, because that seed was not in the question (§6).
Letting both the run and the report decide the status. One place decides; everything else records (§8).
A bare $(MAKE) in a recipe. -f is not inherited, so the sub-make silently reads a different Makefile — measured as four "no rule to make target" errors and one test that worked by coincidence (§7).
-f $(MAKEFILE_LIST) instead of $(firstword ...). The variable has a leading space and, after an include, more than one entry (§7).
Recursing when a prerequisite would do. report: regress already orders the work; every sub-make is somewhere flags can get lost (§7).
A report whose text depends on completion order. Then it cannot be diffed against yesterday's run (§8).
10. Check Your Understanding
Question 1. A 200-run regression is launched. The compute farm kills 150 jobs. A report that does grep -l PASS build/*/*/result.txt | wc -l prints 50 passed, 0 failed and exits 0. Exactly what is wrong, and what change fixes it?
Question 2. A run directory contains run.log but no result.txt. Why is it worth reporting that differently from a directory containing neither?
Question 3. make report-only prints RESULT: PASSED and exits 0, while a run that failed sits on disk. Give a reason this can be correct behaviour rather than a bug.
Question 4. A recipe contains @$(MAKE) --no-print-directory report-only and four of five targets fail with "No rule to make target", while the fifth succeeds. What is happening, and why does exactly one of them work?
Question 5. Why must the report walk $(TESTS) and $(SEEDS) rather than list the files it finds, even when nothing was interrupted?
11. Exercise
Build the report from §3 on top of 6.3's regression. Every transcript is real output.
Step 1. Run a passing sweep and check the status:
make report SEEDS=1
echo "rc=$?" 5 passed 0 failed 0 incomplete 0 not run (of 5)
RESULT: PASSED
rc=0Step 2. Run a sweep with a failure. Confirm every test still ran and the status is non-zero:
make clean && make report SEEDS=5 4 passed 1 failed 0 incomplete 0 not run (of 5)
RESULT: FAILED
make: *** [report] Error 1Step 3. Widen the sweep and let it localise the flaky test:
make clean && make -j8 report SEEDS="1 2 3 4 5 6" 28 passed 2 failed 0 incomplete 0 not run (of 30)
RESULT: FAILEDStep 4. Prove the INCOMPLETE and NOTRUN branches fire. Break one run, then ask about a seed nobody ran:
make clean && make report SEEDS="1 2" > /dev/null
rm -f build/axi_burst/2/result.txt
make report-only SEEDS="1 2 99" axi_smoke 99 NOTRUN
axi_burst 2 INCOMPLETE
axi_burst 99 NOTRUN
=====================================================
9 passed 0 failed 1 incomplete 5 not run (of 15)
RESULT: FAILEDStep 5. Confirm the report does not depend on scheduling:
make clean && make report SEEDS="1 2 3 4 5 6 7 8" > /dev/null; md5 -q build/report.txt
make clean && make -j8 report SEEDS="1 2 3 4 5 6 7 8" > /dev/null; md5 -q build/report.txtbb96e0174e73a90651f2c95aaf1aa3aa
bb96e0174e73a90651f2c95aaf1aa3aaThen make the report lie to you, and fix it. Replace the for loops with the naive version from §2 that greps the files it finds, and run the full eight-seed sweep:
36 passed, 4 failed
make: *** [naive] Error 1Correct so far. Now imitate a farm that killed exactly the jobs that were going to fail — remove the FAIL verdicts and their logs — and ask both reports about the same tree:
for f in $(grep -l FAIL build/*/*/result.txt); do rm -f "$f" "$(dirname $f)/run.log"; done
make -f naive.mk naive ; echo "rc=$?"
make report-only SEEDS="1 2 3 4 5 6 7 8" 36 passed, 0 failed
rc=0
36 passed 0 failed 0 incomplete 4 not run (of 40)
RESULT: FAILEDThe same directory, two answers. The naive report sees thirty-six passes, no failures, and exits 0 — a green regression. The four-state report notices that four of the forty runs it asked about are missing entirely and fails. That difference is the whole chapter.
One more thing to try, because it distinguishes the two unusual states. Delete only a result.txt and leave its run.log:
make clean && make report SEEDS="1 2" > /dev/null
rm -f build/axi_burst/2/result.txt
make report-only SEEDS="1 2" | grep -E 'axi_burst|passed' axi_burst 1 PASS
axi_burst 2 INCOMPLETE
9 passed 0 failed 1 incomplete 0 not run (of 10)INCOMPLETE, not NOTRUN — because the log is still there to read. Removing both files is what produces NOTRUN. Predict which you will get before each experiment; that is the fastest way to be sure you have understood the table in §3.
12. Remember
- A regression's output is a summary and an exit status, and the status is what every automated user of it actually reads.
- Walk the runs you expected, not the files you found. Counting found files turns 150 killed jobs into a small, clean, passing report.
- A run has four states, not two: PASS, FAIL, INCOMPLETE (a log but no verdict) and NOTRUN (neither).
- INCOMPLETE and NOTRUN must both fail the regression. An unknown outcome is not a good outcome.
- Keep them separate. One has a log that usually explains itself; the other is a question about why the job never launched.
- Prove every verdict branch can fire. Measured by deleting one
result.txtand asking about an unrun seed — 9 passed, 0 failed, and correctly FAILED. - A report describes exactly the cross product you name. Measured:
RESULT: PASSEDwith a broken run on disk, because that seed was not in the question — so print the scope in the header. - Only one place decides pass or fail. Runs record data; the report sets the status.
- Never trust the simulator's exit status —
vvpexits 0 on a failed test, so the verdict comes from the log. - A sub-make does not inherit
-f.MAKEFLAGSwas measured empty, and the symptom was four missing rules with one target working by coincidence. - Use
$(firstword $(MAKEFILE_LIST)), because the variable carries a leading space and grows with everyinclude. - Prefer a prerequisite over recursion.
report: regressorders the work without a sub-make. - Keep the report order-independent, so it is byte-identical serial and parallel and can be diffed against yesterday's.
Module 6 is complete. The flow selects a test by name, controls and records its seed, expands a test list into a grid of independent runs, executes them in parallel without sharing anything, and reports one answer with an exit status a merge gate can trust. All of it in front of one simulator, invoked as iverilog and vvp.
That is the next thing to break. A real project runs VCS on the farm, Xcelium for a second opinion, and Questa on somebody's desk, with different command names, different seed options and different log formats — and the flow has to pick one without being rewritten. Module 7 integrates the commercial simulators properly, and puts the per-tool differences in one place instead of scattered through every recipe.
Where this fits
Part of the Makefile curriculum.