awk Part III · Module 9 · Simulation and UVM Log Analysis

Regression Failure Analysis

Chapter 9.5 produced the page a verification lead opens. This chapter turns that page into the three statements a morning triage actually needs: how much of the regression passed and over what denominator, which failures are the same failure, and which of today's failures are new. The second is where the work halves — two failing tests sharing one signature are usually one bug — and the third is where the obvious comparison goes wrong, because a test that did not run today looks exactly like a test that started failing, and the two go to different people.

Advanced17 min readawkPart IIIUVMRegressionTriage

Part III · Module 9 · Chapter 9.6 · Regression Failure Analysis

1. Why This Matters

A per-test table answers "which tests failed". A triage needs three further answers, and none of them is visible row by row:

How much of the regression passed, over a denominator somebody chose deliberately.
Which failures are the same failure. Two rows with the same signature are one investigation.
Which failures are new. A failure that was there yesterday is already somebody's job.

This is the module's synthesis chapter, and it is rated critical because these three numbers are what get quoted in a stand-up. The pass rate is the headline, the shared signature decides how many people are assigned, and the new-versus-known split decides whether anybody is woken up.

Figure 1 — a regression verdict needs two runs, not one. Today's statuses alone cannot distinguish a failure that is new from one that has been there for a week, and the distinction decides who is told. The three outcome classes on the right go to different people: a new regression to whoever changed the design, a known failure to whoever already owns it, and a test that did not run to whoever owns the flow.

2. The Fixture Set

This chapter reuses Chapter 9.5's regression directory. Recreating it here so the commands below are runnable on their own:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

mkdir -p logs
{ echo 'UVM_INFO @ 0: reporter [RNTST] Running test alu_test...'
  for t in 1000 1100 1200 1300 1400 1500 1600 1700; do
    echo "UVM_ERROR ../tb/env/alu_sb.sv(145) @ $t: uvm_test_top.env.sb [MISMATCH] exp!=act"
  done
  echo 'UVM_ERROR ../tb/env/alu_cov.sv(77) @ 2000: uvm_test_top.env.cov [NO_HIT] bin never hit'
  echo '--- UVM Report Summary ---'
  echo 'UVM_ERROR :    9'
} > logs/alu.log
{ echo 'UVM_INFO @ 0: reporter [RNTST] Running test uart_test...'
  echo 'UVM_INFO ../tb/env/uart_env.sv(50) @ 100: uvm_test_top.env [BUILD] ok'
  echo '--- UVM Report Summary ---'
  echo 'UVM_ERROR :    0'
} > logs/uart.log
{ echo 'UVM_INFO @ 0: reporter [RNTST] Running test ddr_test...'
  echo 'UVM_FATAL ../tb/env/ddr_env.sv(300) @ 500: uvm_test_top.env [DEADLOCK] no response'
} > logs/ddr.log
{ echo 'UVM_INFO @ 0: reporter [RNTST] Running test fifo_test...'
  echo 'UVM_ERROR ../tb/env/alu_sb.sv(145) @ 900: uvm_test_top.env.sb [MISMATCH] exp!=act'
  echo '--- UVM Report Summary ---'
  echo 'UVM_ERROR :    1'
} > logs/fifo.log
: > logs/spi.log
ls logs/

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu.log
ddr.log
fifo.log
spi.log
uart.log

Five logs: alu failing with nine records from two causes, uart clean, ddr dying on a fatal with no report summary, fifo failing once with the same signature as alu, and spi empty because it never ran.

All synthetic, as Chapter 9.1 section 1 stated for the whole module.

3. Today's Statuses, as Data

Chapter 9.5's table, emitted as two columns so the rest of this chapter can join on it:

Azvya Education Pvt. Ltd.VLSI Mentor

today.awk — synthetic teaching script. Emit one status per expected test.

BEGIN { for (i = 1; i < ARGC; i++) seen[ARGV[i]] = 0 }
/^--- UVM Report Summary ---/ { insum[FILENAME] = 1 }
insum[FILENAME] { next }
{ seen[FILENAME]++ }
/^UVM_(ERROR|FATAL) (\.|\/|@)/ { if ($1 == "UVM_FATAL") f[FILENAME]++; else e[FILENAME]++ }
END {
  for (i = 1; i < ARGC; i++) {
    g = ARGV[i]; t = g; sub(/.*\//, "", t); sub(/\.log$/, "", t)
    st = (seen[g] == 0 ? "NO_OUTPUT" : (f[g] ? "FATAL" : (e[g] ? "FAIL" : "PASS")))
    printf "%s %s\n", t, st
  }
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f today.awk logs/*.log | sort

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu FAIL
ddr FATAL
fifo FAIL
spi NO_OUTPUT
uart PASS

Five tests, four statuses present. The ARGV enumeration from Chapter 9.5 section 4 is what puts spi in the list at all.

Emitting this as a file rather than a formatted table is the point: a verdict needs to join it against yesterday's, and Chapter 7.6's reasoning says a program-readable form is a delimited one.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f today.awk logs/*.log > today.txt
wc -l < today.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

4. The Pass Rate, With Its Denominator Chosen

Chapter 9.5 section 6 established that two denominators are defensible. As a verdict, both get printed:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '{ tot++; if ($2 == "PASS") ok++; if ($2 == "NO_OUTPUT") no++ } END { ran = tot - no; printf "pass rate: %.1f%% of %d that ran; %.1f%% of %d attempted (%d did not run)\n", 100 * ok / ran, ran, 100 * ok / tot, tot, no }' today.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

pass rate: 25.0% of 4 that ran; 20.0% of 5 attempted (1 did not run)

Both figures, both denominators, and the count that explains the gap.

The engineering position, restated because this is the chapter where the number leaves the script: the attempted figure is the honest headline. A regression's job is to run the tests it was given, and a test that could not be launched is a failure of that job even though it is not a failure of the design. Reporting only the run-rate figure means a flow that drops ten tests makes the regression look better.

5. The Shared Signature

Here is where the work halves. Grouping Chapter 9.4's signatures across the whole directory rather than within one log:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '/^--- UVM Report Summary ---/ { insum[FILENAME] = 1 } insum[FILENAME] { next } /^UVM_(ERROR|FATAL) (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); t = FILENAME; sub(/.*\//, "", t); sub(/\.log$/, "", t); if (!((id, t) in seen)) { seen[id, t]; ntests[id]++ } nrec[id]++ } END { for (k in ntests) printf "%-10s %d test(s), %d record(s)%s\n", k, ntests[k], nrec[k], (ntests[k] > 1 ? "   <- SHARED across tests" : "") }' logs/*.log | sort -k2 -rn

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

MISMATCH   2 test(s), 9 record(s)   <- SHARED across tests
NO_HIT     1 test(s), 1 record(s)
DEADLOCK   1 test(s), 1 record(s)

MISMATCH appears in two tests. alu and fifo are both failing at the same check, which makes them one investigation rather than two.

A signature shared across tests is the cheapest halving of work available

This is the single most useful number in a regression triage, and a per-test table cannot show it.

Chapter 9.5's table has one row per test, and a lead reading it sees alu FAIL and fifo FAIL as two problems. Nothing in that view says they fail at the same line of the same scoreboard — which, if true, means one fix clears both rows.

The mechanism is Chapter 6.2 section 5's distinct-counting, applied across files: count distinct (signature, test) pairs per signature, using in rather than a comparison so the membership test does not create phantom keys (Chapter 6.1 §5). The ntests column is the number of tests affected; nrec is the number of records, which Chapter 9.3 section 7 established measures loudness rather than size.

Two readings, and the second is why the column is worth having:

One signature, several tests — probably one design bug exercised by several tests. Assign one person.
Several signatures, one test — probably one test hitting several distinct problems, or one bug tripping several checks. Still one person, but more to find.

The direction to be careful about is the first: it is a hypothesis, not a fact. Two tests can fail the same check for genuinely different reasons — the same scoreboard comparison can be broken by two different upstream defects. So the right use is to look at those two logs together first, not to close one of them.

And a shared signature across most tests usually means something environmental — a bad model, a misconfigured clock, a stale compile — which is the other reason this column earns its place: it is how a regression-wide problem announces itself.

6. New Versus Known

A failure that was there yesterday is already assigned. Comparing against the previous run needs its statuses kept:

Azvya Education Pvt. Ltd.VLSI Mentor

yesterday.txt — synthetic. The previous night's statuses.

alu FAIL
ddr PASS
fifo PASS
spi PASS
uart PASS

The obvious comparison — anything not passing now that was passing then is newly failing:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'FNR == NR { y[$1] = $2; next } { t = $1; st = $2; prev = (t in y ? y[t] : "UNKNOWN"); bad_now = (st != "PASS"); bad_then = (prev != "PASS" && prev != "UNKNOWN"); if (bad_now && !bad_then) cls = "NEWLY FAILING"; else if (!bad_now && bad_then) cls = "FIXED"; else if (bad_now && bad_then) cls = "still failing"; else cls = "still passing"; printf "%-6s %-10s was %-10s %s\n", t, st, prev, cls }' yesterday.txt today.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    FAIL       was FAIL       still failing
ddr    FATAL      was PASS       NEWLY FAILING
fifo   FAIL       was PASS       NEWLY FAILING
spi    NO_OUTPUT  was PASS       NEWLY FAILING
uart   PASS       was PASS       still passing

Three newly failing. And the third of them is wrong.

7. A Test That Did Not Run Is Not a Regression

spi is reported as NEWLY FAILING. It did not fail — it did not run.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'FNR == NR { y[$1] = $2; next } { t = $1; st = $2; prev = (t in y ? y[t] : "UNKNOWN"); if (st == "NO_OUTPUT") cls = "NOT RUN (flow)"; else if (st == "PASS" && prev != "PASS") cls = "FIXED"; else if (st == "PASS") cls = "still passing"; else if (prev == "PASS") cls = "NEW REGRESSION"; else cls = "known failure"; printf "%-6s %-10s was %-10s %s\n", t, st, prev, cls }' yesterday.txt today.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    FAIL       was FAIL       known failure
ddr    FATAL      was PASS       NEW REGRESSION
fifo   FAIL       was PASS       NEW REGRESSION
spi    NO_OUTPUT  was PASS       NOT RUN (flow)
uart   PASS       was PASS       still passing

Two new regressions, not three, and spi is in a class of its own.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'FNR == NR { y[$1] = $2; next } { t = $1; st = $2; prev = (t in y ? y[t] : "UNKNOWN"); if (st == "NO_OUTPUT") nr++; else if (st == "PASS" && prev != "PASS") fx++; else if (st == "PASS") sp++; else if (prev == "PASS") rg++; else kn++ } END { printf "%d NEW REGRESSION, %d known failure, %d FIXED, %d still passing, %d NOT RUN (sum=%d)\n", rg + 0, kn + 0, fx + 0, sp + 0, nr + 0, rg + kn + fx + sp + nr }' yesterday.txt today.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

2 NEW REGRESSION, 1 known failure, 0 FIXED, 1 still passing, 1 NOT RUN (sum=5)

Five classes summing to five tests.

Conflating the two sends a queue problem to whoever debugs RTL

The difference between three new regressions and two is not a rounding question. It is who gets told.

A new regression means something that worked yesterday is broken today, and the first suspect is whatever changed — an RTL commit, a testbench edit, a new compile option. That goes to the person who made the change.

A test that did not run means the licence server was down, the queue killed the job, the build output was missing, or the disk filled. That goes to whoever owns the flow, and nothing about the design is implicated.

Reporting the second as the first costs a verification engineer a morning: they bisect commits looking for a change that broke spi, and there is no such change.

The reason the naive comparison makes this mistake is instructive. It tests st != "PASS", which is true for FAIL, FATAL and NO_OUTPUT — three statuses collapsed into "bad" exactly as Chapter 9.5 section 5 warned. The four statuses exist precisely so that a comparison like this can keep them apart, and a run-to-run diff is where the distinction pays.

One more case the refined version handles and the naive one does not: a test that was NO_OUTPUT yesterday and passes today is reported FIXED, which is right in that nothing is wrong now, and worth knowing because it was never actually broken.

8. What a Previous Run Cannot Tell You

Two limits worth stating, because a run-to-run comparison invites more confidence than it deserves.

A test missing from yesterday's file is not comparable. The command above records it as UNKNOWN and classifies on today's status alone:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf 'alu FAIL\nddr PASS\nfifo PASS\nuart PASS\n' > partial.txt
awk 'FNR == NR { y[$1] = $2; next } { t = $1; st = $2; prev = (t in y ? y[t] : "UNKNOWN"); printf "%-6s %-10s was %s\n", t, st, prev }' partial.txt today.txt | sort

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    FAIL       was FAIL
ddr    FATAL      was PASS
fifo   FAIL       was PASS
spi    NO_OUTPUT  was UNKNOWN
uart   PASS       was PASS

spi is UNKNOWN because yesterday's file has no row for it — a newly added test, or a test the previous run also failed to launch. Either way "newly failing" is not a claim the data supports, and the refined classifier puts it in NOT RUN on today's status rather than guessing.

A status is not a cause. Two runs both reporting FAIL for alu does not mean the same failure. The signature is what establishes that, and comparing signatures rather than statuses is the stronger check:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '/^--- UVM Report Summary ---/ { insum[FILENAME] = 1 } insum[FILENAME] { next } /^UVM_(ERROR|FATAL) (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); t = FILENAME; sub(/.*\//, "", t); sub(/\.log$/, "", t); sig[t SUBSEP id] = 1 } END { for (k in sig) { split(k, p, SUBSEP); printf "%s %s\n", p[1], p[2] } }' logs/*.log | sort

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu MISMATCH
alu NO_HIT
ddr DEADLOCK
fifo MISMATCH

Keeping that between runs lets you see a test whose status is unchanged and whose failure mode has changed — which is a real event, usually meaning one bug was fixed and another exposed, and which a status-only comparison reports as still failing.

9. The Verdict

All of it in one script:

Azvya Education Pvt. Ltd.VLSI Mentor

verdict.awk — synthetic teaching script. Compare two runs and report a verdict.

# verdict.awk — regression verdict from today's and yesterday's statuses.
#
# Usage:  awk -f verdict.awk yesterday.txt today.txt
# Exit:   0 no new regressions and nothing unrun
#         1 at least one new regression
#         2 at least one test did not run (flow problem)

FNR == NR { y[$1] = $2; next }

{
    t = $1; st = $2
    prev = (t in y ? y[t] : "UNKNOWN")
    tot++
    if (st == "NO_OUTPUT")                  { cls = "NOT RUN";        notrun++ }
    else if (st == "PASS" && prev != "PASS"){ cls = "FIXED";          fixed++; ran++; ok++ }
    else if (st == "PASS")                  { cls = "still passing";  pass++;  ran++; ok++ }
    else if (prev == "PASS")                { cls = "NEW REGRESSION"; regr++;  ran++ }
    else                                    { cls = "known failure";  known++; ran++ }
    printf "%-6s %-10s was %-10s %s\n", t, st, prev, cls
}

END {
    printf "\n%d test(s): %d new regression(s), %d known, %d fixed, %d passing, %d not run (sum=%d)\n", tot, regr + 0, known + 0, fixed + 0, pass + 0, notrun + 0, regr + known + fixed + pass + notrun
    if (ran) printf "pass rate: %.1f%% of %d that ran; %.1f%% of %d attempted\n", 100 * ok / ran, ran, 100 * ok / tot, tot
    if (notrun) exit 2
    if (regr)   exit 1
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f verdict.awk yesterday.txt today.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    FAIL       was FAIL       known failure
ddr    FATAL      was PASS       NEW REGRESSION
fifo   FAIL       was PASS       NEW REGRESSION
spi    NO_OUTPUT  was PASS       NOT RUN
uart   PASS       was PASS       still passing

5 test(s): 2 new regression(s), 1 known, 0 fixed, 1 passing, 1 not run (sum=5)
pass rate: 25.0% of 4 that ran; 20.0% of 5 attempted

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f verdict.awk yesterday.txt today.txt >/dev/null
echo "status: $?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

status: 2

Three statements and a status. The status is 2 rather than 1, because a test that did not run means the regression could not be fully evaluated — Chapter 8.3 section 10's convention, and the ordering says a flow problem outranks a design regression for the purpose of "can this result be trusted".

The sum=5 is the arithmetic check. The two pass rates are both printed. And the five classes are the four statuses of Chapter 9.5 crossed with yesterday's result, collapsed to the five distinctions that change who is told.

10. Common Mistakes

Reporting only the run-rate pass figure. It improves when the flow breaks — section 4.

Reading a per-test table row by row. The shared signature is invisible that way — section 4.

Treating a shared signature as proof of one cause. It is a hypothesis; read both logs.

Classifying NO_OUTPUT as newly failing. It is a flow problem — section 6.

Collapsing FAIL, FATAL and NO_OUTPUT into "bad". The four statuses exist so a diff can keep them apart.

Assuming a test absent from yesterday's file was passing. It is UNKNOWN — section 7.

Comparing statuses rather than signatures. A test can fail differently and still read still failing — section 7.

Not checking that the classes sum to the test count.

Ranking by record count. It measures loudness — Chapter 9.3 §7.

11. Exercises

Exercise 1 — Emit today's statuses

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f today.awk logs/*.log | sort

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu FAIL
ddr FATAL
fifo FAIL
spi NO_OUTPUT
uart PASS

Exercise 2 — Both pass rates

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f today.awk logs/*.log > today.txt
awk '{ tot++; if ($2 == "PASS") ok++; if ($2 == "NO_OUTPUT") no++ } END { printf "%.1f%% of %d ran; %.1f%% of %d attempted\n", 100 * ok / (tot - no), tot - no, 100 * ok / tot, tot }' today.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

25.0% of 4 ran; 20.0% of 5 attempted

Exercise 3 — Find the shared signature

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '/^--- UVM Report Summary ---/ { insum[FILENAME] = 1 } insum[FILENAME] { next } /^UVM_(ERROR|FATAL) (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); t = FILENAME; sub(/.*\//, "", t); sub(/\.log$/, "", t); if (!((id, t) in seen)) { seen[id, t]; ntests[id]++ } } END { for (k in ntests) printf "%-10s %d test(s)%s\n", k, ntests[k], (ntests[k] > 1 ? "   <- SHARED" : "") }' logs/*.log | sort -k2,2rn -k1,1

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

MISMATCH   2 test(s)   <- SHARED
DEADLOCK   1 test(s)
NO_HIT     1 test(s)

alu and fifo share a signature — one investigation, not two.

Exercise 4 — Debug a misclassified flow problem

A triage reports three new regressions and one of them never ran.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'FNR == NR { y[$1] = $2; next } { t = $1; st = $2; prev = (t in y ? y[t] : "UNKNOWN"); bad_now = (st != "PASS"); bad_then = (prev != "PASS" && prev != "UNKNOWN"); if (bad_now && !bad_then) cls = "NEWLY FAILING"; else if (bad_now) cls = "still failing"; else cls = "passing"; printf "%-6s %-10s %s\n", t, st, cls }' yesterday.txt today.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    FAIL       still failing
ddr    FATAL      NEWLY FAILING
fifo   FAIL       NEWLY FAILING
spi    NO_OUTPUT  NEWLY FAILING
uart   PASS       passing

st != "PASS" is true for NO_OUTPUT as well as for FAIL and FATAL, so a test that did not run was classified as a design regression.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'FNR == NR { y[$1] = $2; next } { t = $1; st = $2; prev = (t in y ? y[t] : "UNKNOWN"); if (st == "NO_OUTPUT") cls = "NOT RUN (flow)"; else if (st == "PASS") cls = "passing"; else if (prev == "PASS") cls = "NEW REGRESSION"; else cls = "known failure"; printf "%-6s %-10s %s\n", t, st, cls }' yesterday.txt today.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    FAIL       known failure
ddr    FATAL      NEW REGRESSION
fifo   FAIL       NEW REGRESSION
spi    NO_OUTPUT  NOT RUN (flow)
uart   PASS       passing

Two regressions and one flow problem — three different owners instead of one.

Exercise 5 — Engineering application

Run the full verdict and read each part of it.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f verdict.awk yesterday.txt today.txt | tail -3
awk -f verdict.awk yesterday.txt today.txt >/dev/null; echo "status: $?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet


5 test(s): 2 new regression(s), 1 known, 0 fixed, 1 passing, 1 not run (sum=5)
pass rate: 25.0% of 4 that ran; 20.0% of 5 attempted
status: 2

Four things a stand-up needs: the counts with their arithmetic check, both pass rates, and a status that distinguishes "could not evaluate" from "evaluated and failing".

The caveats this module has accumulated all apply at once, and a verdict is where they must be stated rather than assumed:

2 new regression(s) counts tests, not bugs. Both could be one cause, which section 5's shared-signature column is how you find out.
1 known means it was failing yesterday too, not that it is understood.
25.0% and 20.0% are both true; the second is the honest headline.
1 not run is not a verification finding. It belongs to whoever owns the flow.
A signature count is an upper bound on defects, since one bug can trip several checks — Chapter 9.4 §11.

Exercise 6 — Challenge: a status-only comparison misses a changed failure mode

Section 8 claimed that comparing statuses hides a changed failure mode. Demonstrate it.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf 'alu FAIL\n' > y2.txt
printf 'alu FAIL\n' > t2.txt
awk 'FNR == NR { y[$1] = $2; next } { printf "%-6s %s was %s -> %s\n", $1, $2, y[$1], ($2 == y[$1] ? "unchanged" : "CHANGED") }' y2.txt t2.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    FAIL was FAIL -> unchanged

By status, nothing happened. Now the signatures behind those two identical statuses:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf 'alu MISMATCH\n' > ysig.txt
printf 'alu DEADLOCK\n' > tsig.txt
awk 'FNR == NR { y[$1] = $2; next } { printf "%-6s %s was %s -> %s\n", $1, $2, y[$1], ($2 == y[$1] ? "unchanged" : "FAILURE MODE CHANGED") }' ysig.txt tsig.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu    DEADLOCK was MISMATCH -> FAILURE MODE CHANGED

Same status both nights, different failure. alu was failing a data comparison and is now deadlocking — which usually means the original bug was fixed and a second one is now reachable, and which a status-only comparison reports as still failing.

That is the last of this module's measurement lessons and it is the same one as all the others: a comparison can only detect a change in what it compares. A status has four values, so it can register four kinds of change; a signature has as many values as the testbench has checks. Chapter 9.3's cross-check against the log's own summary, Chapter 9.5's ARGV enumeration and this are three instances of one idea — choose the thing you measure so that the failure you care about is visible in it.

12. Summary

A verdict needs three statements: how much passed and over what denominator, which failures are the same, and which are new.
Print both pass rates. The attempted figure is the honest headline; the run-rate figure improves when the flow breaks.
A signature shared across tests is the cheapest halving of work available, and a per-test table cannot show it.
Treat a shared signature as a hypothesis — read both logs together rather than closing one.
A shared signature across most tests is usually environmental.
A run-to-run comparison needs the previous statuses kept as data, not as a formatted table.
NO_OUTPUT is not a new regression. st != "PASS" collapses three statuses and sends a flow problem to whoever debugs RTL.
A test absent from the previous run is UNKNOWN, not previously passing.
Compare signatures, not just statuses — a failure mode can change while the status does not.
Check that the classes sum to the test count, and exit 2 when anything did not run.

That completes Module 9. A directory of simulation logs is now a verdict, and every number in it has a stated denominator and a stated grouping.

Chapter 10.1 opens Module 10 — Timing and STA Report Analysis, which is the hardest parsing job in the flow. A report_timing block is not one record per line: a startpoint, an endpoint, a delay table and a slack line are separate lines that belong to one logical path, so the techniques here — a pattern per record type — stop being enough, and carrying state across records becomes the whole problem.

Where this fits

Part of the awk curriculum.