awk Part III · Module 9 · Simulation and UVM Log Analysis

Per-Test Summaries

Chapters 9.1 to 9.4 reduced one log to a table. A regression has hundreds, and what a verification lead opens first is a single page with one row per test — status, error count, the dominant failure signature. Building it is Chapter 7.4's multi-file work applied to parsed logs, and it contains that chapter's hole in the place it does the most damage: a summary driven by log records cannot report a test whose log is empty, and an empty log means the test never ran at all. That row is both the most urgent in the regression and the one the obvious script silently drops.

Advanced17 min readawkPart IIIUVMRegressionReporting

Part III · Module 9 · Chapter 9.5 · Per-Test Summaries

1. Why This Matters

A regression summary is the most-read artefact a verification flow produces. It is looked at every morning, by people who will not open a log unless the table tells them to.

So the table decides what gets attention, and this chapter is about the row it is most likely to lose.

It is rated critical because of the asymmetry: a test that ran and failed is visible in any summary, and a test that never ran produces nothing to be visible. The second needs the script to go looking.

Figure 1 — the order that matters. Enumerating the expected tests comes first, before any log is opened, because the list of tests cannot be derived from the logs that exist. Everything after that fills in a row that is already present, so a test whose log is empty or missing still has a line in the table to report NO OUTPUT against.

2. The Regression Directory

Five logs, one of them empty:

Azvya Education Pvt. Ltd.VLSI Mentor

logs/alu.log — synthetic. Nine error records from two causes.

UVM_INFO @ 0: reporter [RNTST] Running test alu_test...
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1000: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1100: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1200: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1300: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1400: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1500: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1600: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1700: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_cov.sv(77) @ 2000: uvm_test_top.env.cov [NO_HIT] bin never hit
--- UVM Report Summary ---
UVM_ERROR :    9

Azvya Education Pvt. Ltd.VLSI Mentor

logs/uart.log — synthetic. A clean run.

UVM_INFO @ 0: reporter [RNTST] Running test uart_test...
UVM_INFO ../tb/env/uart_env.sv(50) @ 100: uvm_test_top.env [BUILD] ok
--- UVM Report Summary ---
UVM_ERROR :    0

Azvya Education Pvt. Ltd.VLSI Mentor

logs/ddr.log — synthetic. Died on a fatal, so there is no report summary.

UVM_INFO @ 0: reporter [RNTST] Running test ddr_test...
UVM_FATAL ../tb/env/ddr_env.sv(300) @ 500: uvm_test_top.env [DEADLOCK] no response

Azvya Education Pvt. Ltd.VLSI Mentor

logs/fifo.log — synthetic. One error, same signature as alu.

UVM_INFO @ 0: reporter [RNTST] Running test fifo_test...
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 900: uvm_test_top.env.sb [MISMATCH] exp!=act
--- UVM Report Summary ---
UVM_ERROR :    1

And logs/spi.log, which is empty — the test was killed before it wrote anything:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

: > logs/spi.log
ls logs/
wc -c < logs/spi.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu.log
ddr.log
fifo.log
spi.log
uart.log
       0

Five files, one of them zero bytes.

Note ddr.log has no report summary either — a UVM_FATAL ended the run before UVM printed one. Chapter 9.1 section 6 warned that a parser requiring the summary would refuse exactly the logs that most need reading, and this is that case.

3. The Record-Driven Table

The obvious approach: key everything on FILENAME.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '/^--- UVM Report Summary ---/ { s[FILENAME] = 1 } s[FILENAME] { next } /^UVM_ERROR (\.|\/|@)/ { e[FILENAME]++ } /^UVM_FATAL (\.|\/|@)/ { f[FILENAME]++ } { n[FILENAME]++ } END { for (g in n) { t = g; sub(/.*\//, "", t); sub(/\.log$/, "", t); printf "%-6s %3d rec %3d err %3d fatal\n", t, n[g], e[g] + 0, f[g] + 0 } }' logs/*.log | sort

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu     10 rec   9 err   0 fatal
ddr      2 rec   0 err   1 fatal
fifo     2 rec   1 err   0 fatal
uart     2 rec   0 err   0 fatal

Four rows from a directory of five logs. spi is absent.

It looks complete. Four tests, four rows, counts that add up, no gaps, nothing to suggest a fifth test exists.

The missing row is the test that never ran

Chapter 7.4 section 7 established the mechanism: every rule here is driven by a record, and an empty file supplies none. awk opens spi.log, finds nothing, and moves on — so no key is ever created for it and END has nothing to iterate.

What makes this the worst instance of that bug is what an empty log means.

A test that ran and failed writes a log full of errors. A test whose simulator could not get a licence, whose build was missing, whose queue slot was killed, or which crashed before UVM initialised writes nothing. The empty file is the only evidence that the test was attempted.

So the record-driven table reports every test that managed to run and silently omits every test that did not — and the omitted ones are in the worse state. A lead reading four green-and-red rows has no way to know that a fifth test exists and is in neither category.

It is also the failure that scales badly. On a five-log directory somebody might notice; on a 400-test regression where 12 tests failed to launch, the table shows 388 rows and nobody counts them.

And the obvious defence does not work either. Counting the rows and comparing against the file count catches it only if you know the expected file count — which is section 4's actual fix, arrived at properly.

4. Enumerate First

The fix is Chapter 7.4 section 8's: build the row set from the argument list, in BEGIN, before any file is read.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 } /^--- UVM Report Summary ---/ { s[FILENAME] = 1 } s[FILENAME] { next } /^UVM_ERROR (\.|\/|@)/ { e[FILENAME]++ } /^UVM_FATAL (\.|\/|@)/ { f[FILENAME]++ } { seen[FILENAME]++ } END { for (i = 1; i < ARGC; i++) { g = ARGV[i]; t = g; sub(/.*\//, "", t); sub(/\.log$/, "", t); st = (seen[g] == 0 ? "NO OUTPUT" : (f[g] ? "FATAL" : (e[g] ? "FAIL" : "PASS"))); printf "%-6s %3d rec %3d err %3d fatal  %s\n", t, seen[g], e[g] + 0, f[g] + 0, st } }' logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu     10 rec   9 err   0 fatal  FAIL
ddr      2 rec   0 err   1 fatal  FATAL
fifo     2 rec   1 err   0 fatal  FAIL
spi      0 rec   0 err   0 fatal  NO OUTPUT
uart     2 rec   0 err   0 fatal  PASS

Five rows, and spi is reported as NO OUTPUT.

Two properties, both from Chapter 7.4. The BEGIN loop creates a key for every file whether or not it yields a record, so the row exists independently of the data. And the END loop walks ARGV by position, which gives a defined output order without sort — the rows come out in command-line order, which for a glob is alphabetical and reproducible.

5. Four Statuses, Not Two

The status column has four values and each means something different:

Status	Meaning	What to do
`PASS`	ran, no errors	nothing
`FAIL`	ran, errors reported	debug the design or the test
`FATAL`	stopped mid-run — log is incomplete	debug, and distrust the rest of the log
`NO OUTPUT`	never ran	fix the flow, not the design

The two that are usually collapsed into FAIL are the two that need different people.

A FATAL means UVM called $finish early, so everything the test would have checked after that point was never checked. The log is not a complete record of the run, and a coverage number derived from it is an undercount of what the test would have achieved. Chapter 9.3 section 6 made the verdict-ordering point; this is why.

A NO OUTPUT is not a verification problem at all. It is a licence, a build, a disk or a queue problem, and sending it to whoever triages test failures wastes their morning.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 } /^--- UVM Report Summary ---/ { s[FILENAME] = 1 } s[FILENAME] { next } /^UVM_ERROR (\.|\/|@)/ { e[FILENAME]++ } /^UVM_FATAL (\.|\/|@)/ { f[FILENAME]++ } { seen[FILENAME]++ } END { for (i = 1; i < ARGC; i++) { g = ARGV[i]; if (seen[g] == 0) no++; else if (f[g]) fat++; else if (e[g]) bad++; else ok++ } printf "%d test(s): %d PASS, %d FAIL, %d FATAL, %d NO OUTPUT\n", ARGC - 1, ok + 0, bad + 0, fat + 0, no + 0 }' logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

5 test(s): 1 PASS, 2 FAIL, 1 FATAL, 1 NO OUTPUT

One of each status except FAIL, which has two — agreeing with section 4's table row for row.

Worth adding the arithmetic check Chapter 8.3 section 6 argued for, because four counters built from a chain of else if can silently miss a case:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 } /^--- UVM Report Summary ---/ { s[FILENAME] = 1 } s[FILENAME] { next } /^UVM_ERROR (\.|\/|@)/ { e[FILENAME]++ } /^UVM_FATAL (\.|\/|@)/ { f[FILENAME]++ } { seen[FILENAME]++ } END { ok = 0; bad = 0; fat = 0; no = 0; for (i = 1; i < ARGC; i++) { g = ARGV[i]; if (seen[g] == 0) no++; else if (f[g]) fat++; else if (e[g]) bad++; else ok++ } printf "%d test(s): %d PASS, %d FAIL, %d FATAL, %d NO OUTPUT (sum=%d)\n", ARGC - 1, ok, bad, fat, no, ok + bad + fat + no }' logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

5 test(s): 1 PASS, 2 FAIL, 1 FATAL, 1 NO OUTPUT (sum=5)

sum=5 equals the test count, so every expected test landed in exactly one status. A chain of else if guarantees at most one branch per test and guarantees nothing about whether every test reached one — the sum is what closes that gap.

6. The Pass Rate Has Two Denominators

With the statuses counted, the headline number is a choice:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 } /^--- UVM Report Summary ---/ { s[FILENAME] = 1 } s[FILENAME] { next } /^UVM_ERROR (\.|\/|@)/ { e[FILENAME]++ } /^UVM_FATAL (\.|\/|@)/ { f[FILENAME]++ } { seen[FILENAME]++ } END { ok = 0; no = 0; tot = ARGC - 1; for (i = 1; i < ARGC; i++) { g = ARGV[i]; if (seen[g] == 0) no++; else if (!f[g] && !e[g]) ok++ } ran = tot - no; printf "pass rate over tests that RAN:       %.1f%% (%d of %d)\n", 100 * ok / ran, ok, ran; printf "pass rate over tests ATTEMPTED:      %.1f%% (%d of %d)\n", 100 * ok / tot, ok, tot }' logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

pass rate over tests that RAN:       25.0% (1 of 4)
pass rate over tests ATTEMPTED:      20.0% (1 of 5)

25.0% and 20.0%, both defensible, five points apart on a five-test regression.

Excluding the tests that did not run is how a regression reports progress it did not make

This is Chapter 6.4 section 5's averaging problem in its most consequential setting.

25% answers of the tests that ran, how many passed — a statement about the design. 20% answers of the tests we tried to run, how many passed — a statement about the regression as a whole.

The second is the honest headline, and the first is the one that gets reported, for a reason worth naming: the tests that fail to launch are invisible to the tooling that produces the number. A script built on section 3's record-driven table cannot compute the 20% figure, because it does not know the fifth test exists.

The incentive runs the wrong way too. A flow problem that kills ten tests raises the run-rate denominator's quality — the remaining tests are the ones that worked — so a pass rate over tests-that-ran can improve while the regression gets worse.

The rule, which this track has now arrived at from five directions: print the denominator, and when two denominators are defensible, print both. A table reporting 20.0% (1 of 5) with 1 NO OUTPUT beside it cannot be misread. A table reporting 25% cannot be checked.

7. Adding the Dominant Signature

A status tells a lead which tests to look at; the signature tells them whether two failures are the same one. Chapter 9.4's grouping, per test:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 } /^--- UVM Report Summary ---/ { s[FILENAME] = 1 } s[FILENAME] { next } { seen[FILENAME]++ } /^UVM_(ERROR|FATAL) (\.|\/|@)/ { if ($1 == "UVM_FATAL") f[FILENAME]++; else e[FILENAME]++; id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); c[FILENAME SUBSEP id]++; if (c[FILENAME SUBSEP id] > best[FILENAME]) { best[FILENAME] = c[FILENAME SUBSEP id]; top[FILENAME] = id } } END { for (i = 1; i < ARGC; i++) { g = ARGV[i]; t = g; sub(/.*\//, "", t); sub(/\.log$/, "", t); st = (seen[g] == 0 ? "NO OUTPUT" : (f[g] ? "FATAL" : (e[g] ? "FAIL" : "PASS"))); printf "%-6s %-10s %3d err %3d fatal  %s\n", t, st, e[g] + 0, f[g] + 0, (top[g] == "" ? "-" : top[g] " (" best[g] ")") } }' logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    FAIL         9 err   0 fatal  MISMATCH (8)
ddr    FATAL        0 err   1 fatal  DEADLOCK (1)
fifo   FAIL         1 err   0 fatal  MISMATCH (1)
spi    NO OUTPUT    0 err   0 fatal  -
uart   PASS         0 err   0 fatal  -

Now the table says something a count cannot: alu and fifo share the MISMATCH signature. Two failing tests, one likely cause — which Chapter 9.6 turns into a regression-wide finding.

The - for spi and uart is deliberate rather than blank: a dash says "no signature, and that is expected", where an empty cell says nothing at all.

8. Common Mistakes

Keying the table on FILENAME only. The test with an empty log gets no row — section 3.

Counting the rows as a count of tests. They differ exactly when something went badly wrong.

Collapsing FATAL into FAIL. The run stopped, so the log is incomplete — section 5.

Collapsing NO OUTPUT into FAIL. It is a flow problem, not a verification one.

Reporting a pass rate over tests that ran. It improves when the flow breaks — section 6.

Not printing the denominator. The figure cannot be checked.

Requiring the report summary. A log that died on a fatal has none — section 2.

Not checking that the status counts sum to the test count — section 5.

Deriving the expected test list from the logs. It is the one thing the logs cannot tell you.

9. Exercises

Exercise 1 — Reproduce the missing row

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '{ n[FILENAME]++ } END { c = 0; for (g in n) c++; printf "%d row(s) from %d log file(s)\n", c, ARGC - 1 }' logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

4 row(s) from 5 log file(s)

Four rows, five files — the empty log produced no key.

Exercise 2 — Enumerate first

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 } { seen[FILENAME]++ } END { c = 0; for (g in seen) c++; printf "%d row(s) from %d log file(s)\n", c, ARGC - 1 }' logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

5 row(s) from 5 log file(s)

Five and five, because the keys were created in BEGIN.

Exercise 3 — The four-status table

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 } /^--- UVM Report Summary ---/ { s[FILENAME] = 1 } s[FILENAME] { next } /^UVM_ERROR (\.|\/|@)/ { e[FILENAME]++ } /^UVM_FATAL (\.|\/|@)/ { f[FILENAME]++ } { seen[FILENAME]++ } END { for (i = 1; i < ARGC; i++) { g = ARGV[i]; t = g; sub(/.*\//, "", t); sub(/\.log$/, "", t); st = (seen[g] == 0 ? "NO OUTPUT" : (f[g] ? "FATAL" : (e[g] ? "FAIL" : "PASS"))); printf "%-6s %s\n", t, st } }' logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    FAIL
ddr    FATAL
fifo   FAIL
spi    NO OUTPUT
uart   PASS

Exercise 4 — Both pass rates

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 } /^--- UVM Report Summary ---/ { s[FILENAME] = 1 } s[FILENAME] { next } /^UVM_ERROR (\.|\/|@)/ { e[FILENAME]++ } /^UVM_FATAL (\.|\/|@)/ { f[FILENAME]++ } { seen[FILENAME]++ } END { ok = 0; no = 0; tot = ARGC - 1; for (i = 1; i < ARGC; i++) { g = ARGV[i]; if (seen[g] == 0) no++; else if (!f[g] && !e[g]) ok++ } printf "ran: %.1f%% (%d of %d)   attempted: %.1f%% (%d of %d)\n", 100 * ok / (tot - no), ok, tot - no, 100 * ok / tot, ok, tot }' logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

ran: 25.0% (1 of 4)   attempted: 20.0% (1 of 5)

Five points apart, both with their denominators stated.

Exercise 5 — Engineering application

Produce the regression summary a lead opens: one row per expected test with status, counts and dominant signature, then the totals with both pass rates and an arithmetic check, and a status a flow can branch on.

Azvya Education Pvt. Ltd.VLSI Mentor

regress.awk — synthetic teaching script.

# regress.awk — one row per expected test across a regression directory.
#
# Usage:  awk -f regress.awk logs/*.log
# Exit:   0 every test passed
#         1 at least one test failed or died
#         2 at least one test produced no output (flow problem)

BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 }

/^--- UVM Report Summary ---/ { insum[FILENAME] = 1 }
insum[FILENAME] { next }

{ seen[FILENAME]++ }

/^UVM_(ERROR|FATAL) (\.|\/|@)/ {
    if ($1 == "UVM_FATAL") f[FILENAME]++; else e[FILENAME]++
    id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id)
    c[FILENAME SUBSEP id]++
    if (c[FILENAME SUBSEP id] > best[FILENAME]) { best[FILENAME] = c[FILENAME SUBSEP id]; top[FILENAME] = id }
}

END {
    tot = ARGC - 1
    printf "%-6s %-10s %5s %5s  %s\n", "test", "status", "err", "fatal", "top signature"
    for (i = 1; i < ARGC; i++) {
        g = ARGV[i]; t = g; sub(/.*\//, "", t); sub(/\.log$/, "", t)
        if (seen[g] == 0)      { st = "NO OUTPUT"; no++ }
        else if (f[g])         { st = "FATAL";     fat++ }
        else if (e[g])         { st = "FAIL";      bad++ }
        else                   { st = "PASS";      ok++ }
        printf "%-6s %-10s %5d %5d  %s\n", t, st, e[g] + 0, f[g] + 0, (top[g] == "" ? "-" : top[g] " (" best[g] ")")
    }
    ok += 0; bad += 0; fat += 0; no += 0
    ran = tot - no
    printf "\n%d test(s): %d PASS, %d FAIL, %d FATAL, %d NO OUTPUT (sum=%d)\n", tot, ok, bad, fat, no, ok + bad + fat + no
    if (ran) printf "pass rate: %.1f%% of %d that ran; %.1f%% of %d attempted\n", 100 * ok / ran, ran, 100 * ok / tot, tot
    else     printf "pass rate: no test produced output\n"
    if (no)            exit 2
    if (bad || fat)    exit 1
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f regress.awk logs/*.log
echo "status: $?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

test   status       err fatal  top signature
alu    FAIL           9     0  MISMATCH (8)
ddr    FATAL          0     1  DEADLOCK (1)
fifo   FAIL           1     0  MISMATCH (1)
spi    NO OUTPUT      0     0  -
uart   PASS           0     0  -

5 test(s): 1 PASS, 2 FAIL, 1 FATAL, 1 NO OUTPUT (sum=5)
pass rate: 25.0% of 4 that ran; 20.0% of 5 attempted
status: 2

Everything a lead needs on one page, and three properties worth naming.

The status is 2, not 1 — because a test that did not run means the regression could not be fully evaluated, and Chapter 8.3 section 10's convention reserves 2 for exactly that. A flow owner needs to see this before a verification engineer starts debugging.

The sum=5 confirms the four status counts account for every expected test, so no row was double-counted or dropped.

And both pass rates are printed, so the 25% figure cannot be quoted without the 20% beside it.

The engineering caveat: MISMATCH (8) is the dominant signature by record count, which Chapter 9.3 section 7 established is a measure of how loud a failure is rather than how large. And 9 err for alu is nine records from two distinct checks — Chapter 9.4's collapse is not applied here because the per-test view wants a single number, which is a deliberate simplification the table should not be read past.

Exercise 6 — Challenge: three states, not two

Section 8's last mistake is that the expected tests cannot be derived from the logs. The cure is a test list from outside, and it turns out to reveal a third state.

Azvya Education Pvt. Ltd.VLSI Mentor

tests.txt — synthetic. The tests the regression was told to run.

alu
ddr
fifo
spi
uart
newtest

Six expected tests against five log files. Joining them with Chapter 7.5's two-file idiom:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'FNR == NR { want[$1]; next } { f = FILENAME; sub(/.*\//, "", f); sub(/\.log$/, "", f); have[f] = 1 } END { for (t in want) if (!(t in have)) printf "no records for: %s\n", t | "sort" }' tests.txt logs/*.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

no records for: newtest
no records for: spi

Two names, and they are in different situations. newtest has no log file at all — the glob never saw it, so it is not even in ARGV. spi has a log file that is empty.

The join cannot tell them apart, and the reason is the one this chapter opened with: have[f] = 1 runs in a record-driven rule, so an empty log registers exactly like a missing one. The second phase of the join has section 3's bug.

Combining both techniques — ARGV for which files exist, the test list for which should — separates all three states:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 2; i < ARGC; i++) { f = ARGV[i]; sub(/.*\//, "", f); sub(/\.log$/, "", f); exists[f] = 1 } } FNR == NR { want[$1]; next } { f = FILENAME; sub(/.*\//, "", f); sub(/\.log$/, "", f); rec[f]++ } END { for (t in want) { if (!(t in exists)) printf "%-8s NO LOG FILE   (test did not launch)\n", t; else if (!(t in rec)) printf "%-8s EMPTY LOG     (launched, died before writing)\n", t } }' tests.txt logs/*.log | sort

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

newtest  NO LOG FILE   (test did not launch)
spi      EMPTY LOG     (launched, died before writing)

Three states, three different causes, and they go to different people:

State	Evidence	Likely cause
has records	the log has content	the test ran; read its status
empty log	file exists, zero records	launched, then killed — queue, licence, crash at startup
no log file	not in `ARGV`	never launched — the test list and the flow disagree

Note the ARGV loop starts at 2, not 1, because ARGV[1] is tests.txt — the key file, not a log. An off-by-one there would classify tests as a test whose log is missing.

The general principle, and the last of this module's measurement lessons: a report can only omit what its row set does not contain. ARGV gives you the files that exist; a test list gives you the tests that should exist; and the two together give you a row for everything, which is the only arrangement in which nothing can disappear. Chapter 9.6 builds the regression verdict on exactly this pair.

10. Summary

A regression summary decides what gets attention, so the row it loses matters more than the rows it shows.
A table keyed on FILENAME omits the test whose log is empty, because every rule is record-driven.
An empty log means the test never ran — a licence, build or queue problem, not a verification one.
Enumerate the expected tests in BEGIN from ARGV, so every row exists before any log is read.
Walking ARGV by position also gives a defined output order without sort.
Four statuses, not two: PASS, FAIL, FATAL (stopped mid-run, log incomplete), NO OUTPUT (never ran).
Check that the status counts sum to the test count.
A pass rate has two defensible denominators — 25% of those that ran, 20% of those attempted. Print both.
A rate over tests-that-ran improves when the flow breaks, which is why it is the one usually quoted.
A missing log is invisible even to ARGV, and a record-driven join cannot tell a missing log from an empty one. ARGV plus a test list separates all three states.

One page now describes a whole regression. Chapter 9.6 — Regression Failure Analysis turns that page into a verdict: the pass rate with its denominator chosen deliberately, the failure signature shared across tests, which tests are newly failing against a previous run, and why a test that produced no log is not a pass.

Where this fits

Part of the awk curriculum.