awk Part III · Module 11 · Synthesis and Coverage Report Analysis
Aggregating Coverage Metrics Correctly
Every chapter in this module has deferred the same question: given coverage numbers per unit, per signal, per machine and per covergroup, what single figure may honestly be quoted? The answer is not a formula, because the two defensible aggregations can move in opposite directions on identical data — this chapter shows a pair of nightly runs in which all three design units improved, the aggregate coverage fell by thirteen points, and the mean of the per-unit percentages rose by four. Neither number is wrong. Both are useless without the three facts that have to be stated beside them.
Advanced19 min readawkPart IIICoverageAggregationReporting
Part III · Module 11 · Chapter 11.8 · Aggregating Coverage Metrics Correctly
1. Why This Matters
Coverage closure is reported as a number. One number, in a status slide or a gate script, standing for the state of verification on a block.
The five chapters before this one each produced a reason that number is underdetermined: percentages cannot be averaged (11.4), bits and lines are different populations (11.5), state and transition coverage cannot be combined (11.6), and a covergroup's score is a weighted mean that a bin ratio does not reproduce (11.7). This chapter closes the module by establishing what may actually be said — and it is marked Critical because the failure in section 4 is one where the number moves the wrong way while every underlying fact moves the right way.
2. Each Metric, Aggregated From Its Own Counts
The correct per-metric figures, each from the counts that produced it:
awk '/^ Line Coverage/ { lc += substr($4,2)+0; lt += $6+0 } /^ Toggle Coverage/ { tc += substr($4,2)+0; tt += $6+0 } /^ Branch Coverage/ { bc += substr($4,2)+0; bt += $6+0 }
END { printf " %-22s %4d of %4d = %6.2f%%\n", "line (statements)", lc, lt, 100*lc/lt
printf " %-22s %4d of %4d = %6.2f%%\n", "toggle (bits)", tc, tt, 100*tc/tt
printf " %-22s %4d of %4d = %6.2f%%\n", "branch (branches)", bc, bt, 100*bc/bt }' cov.rpt
awk '/^TOTAL State/ { rs = substr($4,2)+0; rst = $(NF-1)+0 } /^TOTAL Transition/ { rt = substr($4,2)+0; rtt = $(NF-1)+0 }
END { printf " %-22s %4d of %4d = %6.2f%%\n", "fsm states", rs, rst, 100*rs/rst
printf " %-22s %4d of %4d = %6.2f%%\n", "fsm transitions", rt, rtt, 100*rt/rtt }' fsm.rpt
awk -F, 'NR>1 { w += $6; ws += $6*(100*$4/$5); bh += $4; bt += $5 } END { printf " %-22s %4s %6.2f%% (weighted; bins %d of %d = %.2f%%)\n", "functional", "-", ws/w, bh, bt, 100*bh/bt }' funccov.csv line (statements) 192 of 235 = 81.70%
toggle (bits) 295 of 420 = 70.24%
branch (branches) 39 of 53 = 73.58%
fsm states 20 of 26 = 76.92%
fsm transitions 31 of 59 = 52.54%
functional - 69.73% (weighted; bins 72 of 181 = 39.78%)Six figures over six populations. Each has a denominator naming what it counts, and the functional row has two figures because Chapter 11.7 showed its score and its bin ratio are different quantities.
That table is the honest summary. The rest of the chapter is about what happens when it is compressed.
3. Two Ways to Compress, Both Wrong
Pooling the counts:
awk '/^ (Line|Toggle|Branch) Coverage/ { c += substr($4,2)+0; t += $6+0 } END { printf " cov.rpt three metrics pooled: %d of %d = %.2f%% <- three populations\n", c, t, 100*c/t }' cov.rpt cov.rpt three metrics pooled: 526 of 708 = 74.29% <- three populations526 of 708 has the form of a rigorous ratio. The 708 items are 235 statements, 420 bits and 53 branches — so bits carry 59% of the weight because the design happens to have more of them than statements. Chapter 11.4 Exercise 6 and Chapter 11.6 section 5 both measured this; it is the third instance in the module.
Averaging the figures:
awk 'BEGIN { split("81.70 70.24 73.58 76.92 52.54", v, " "); for (i=1;i<=5;i++) s += v[i]; printf " mean of the five metric figures = %.2f%% <- a number with no denominator\n", s/5 }' mean of the five metric figures = 71.00% <- a number with no denominator71.00% of nothing. It weights FSM transitions (59 items) equally with toggle bits (420), and there is no population of which 71.00% is covered.
Both compressions produce a plausible single number from correct inputs, and neither can be defended when asked "71.00% of what?"
4. Every Unit Improved and the Number Fell
Two nightly runs of the same block:
unit,covered,total
u_alu,75,100
u_ctrl,20,50
u_io,18,20unit,covered,total
u_alu,77,100
u_ctrl,180,400
u_io,19,20Per unit:
awk -F, 'NR==FNR { if (FNR>1) { c[$1]=$2; t[$1]=$3 } ; next } FNR>1 { pa = 100*c[$1]/t[$1]; pb = 100*$2/$3; printf " %-8s %6.2f%% -> %6.2f%% %-4s (coverable items %d -> %d)\n", $1, pa, pb, (pb > pa ? "UP" : "DOWN"), t[$1], $3 }' run1.csv run2.csv u_alu 75.00% -> 77.00% UP (coverable items 100 -> 100)
u_ctrl 40.00% -> 45.00% UP (coverable items 50 -> 400)
u_io 90.00% -> 95.00% UP (coverable items 20 -> 20)All three units improved. Now the two summaries:
for f in run1.csv run2.csv; do printf " %-10s " "$f"; awk -F, 'FNR>1 { c += $2; t += $3; p += 100*$2/$3; n++ } END { printf "aggregate %3d of %3d = %6.2f%% mean of percentages = %6.2f%%\n", c, t, 100*c/t, p/n }' $f; done run1.csv aggregate 113 of 170 = 66.47% mean of percentages = 68.33%
run2.csv aggregate 276 of 520 = 53.08% mean of percentages = 72.33%awk -F, 'FNR==1 { f++ } FNR>1 { c[f] += $2; t[f] += $3; p[f] += 100*$2/$3; n[f]++ } END { a1 = 100*c[1]/t[1]; a2 = 100*c[2]/t[2]; m1 = p[1]/n[1]; m2 = p[2]/n[2]; printf " aggregate moved %+.2f points; mean of percentages moved %+.2f points -- OPPOSITE DIRECTIONS\n", a2-a1, m2-m1 }' run1.csv run2.csv aggregate moved -13.39 points; mean of percentages moved +4.00 points -- OPPOSITE DIRECTIONSThe aggregate fell 13.39 points. The mean rose 4.00. Every unit improved.
5. The Procedure
Step 5 is the one most often skipped and the only one with detection value. Every report in this module prints its own totals — but reading them exposes something worth a section of its own, because the two reports put the denominator in different fields:
awk '/^TOTAL/ { printf " cov.rpt NF=%d $(NF-1)=[%s] $NF=[%s]\n", NF, $(NF-1), $NF }' cov.rpt
awk '/^TOTAL/ { printf " fsm.rpt NF=%d $(NF-1)=[%s] $NF=[%s]\n", NF, $(NF-1), $NF }' fsm.rpt cov.rpt NF=7 $(NF-1)=[of] $NF=[235)]
cov.rpt NF=7 $(NF-1)=[of] $NF=[420)]
cov.rpt NF=7 $(NF-1)=[of] $NF=[53)]
fsm.rpt NF=7 $(NF-1)=[26] $NF=[states)]
fsm.rpt NF=7 $(NF-1)=[59] $NF=[transitions)]Both have NF=7 and the denominator is at $NF in one and $(NF-1) in the other, because fsm.rpt writes the item name inside the parentheses and cov.rpt does not. Chapter 11.6 section 3 established $(NF-1) for that report and it is wrong here — so a rule carried between two reports from the same flow silently reads the wrong field.
Locating the number by pattern instead works on both:
for f in cov.rpt fsm.rpt; do
awk -v F=$f '/^TOTAL/ && match($0, /of +[0-9]+/) { d = substr($0, RSTART+3, RLENGTH-3); gsub(/ /, "", d); printf " %-10s %-11s denominator %s\n", F, $2, d }' $f
done cov.rpt Line denominator 235
cov.rpt Toggle denominator 420
cov.rpt Branch denominator 53
fsm.rpt State denominator 26
fsm.rpt Transition denominator 59match($0, /of +[0-9]+/) finds the number after the word of wherever it sits, and RSTART/RLENGTH carve it out — the technique from Chapter 9.4, where a field-position extraction broke on a record of a different shape. With it, the reconciliation is portable:
awk '/^ Line Coverage/ { c += substr($4,2)+0; t += $6+0 }
/^TOTAL Line/ && match($0, /\(([0-9]+) of ([0-9]+)\)/) { split(substr($0, RSTART+1, RLENGTH-2), p, / of /); rc = p[1]+0; rt = p[2]+0 }
END { printf " computed %d of %d, report says %d of %d -> %s\n", c, t, rc, rt, (c == rc && t == rt ? "RECONCILES" : "MISMATCH") }' cov.rpt computed 192 of 235, report says 192 of 235 -> RECONCILESInteger counts, so == is correct here — the exemption Chapter 11.1 section 8 noted, where floating-point areas needed a tolerance.
6. What a Single Number May Say
Given all of the above, a defensible headline is a metric name, an aggregation and a denominator:
awk '/^ Line Coverage/ { c += substr($4,2)+0; t += $6+0 } END { printf " line coverage, aggregated over statements: %d of %d = %.2f%%\n", c, t, 100*c/t }' cov.rpt line coverage, aggregated over statements: 192 of 235 = 81.70%That sentence can be compared against next week's, argued with, and recomputed by someone else. 81.70% alone cannot.
Where a single cross-metric figure is genuinely required — a dashboard cell, a one-line status — the honest construction is to name the governing metric rather than to blend:
awk '/^ (Line|Toggle|Branch) Coverage/ { m = $1; c[m] += substr($4,2)+0; t[m] += $6+0 }
END { split("Line Toggle Branch", o, " "); for (i=1;i<=3;i++) { k=o[i]; p = 100*c[k]/t[k]; if (i==1 || p < lo) { lo = p; lk = k; lc = c[k]; lt = t[k] } }
printf " governing metric: %s at %.2f%% (%d of %d) -- the lowest of the three\n", lk, lo, lc, lt }' cov.rpt governing metric: Toggle at 70.24% (295 of 420) -- the lowest of the three"Toggle coverage, 295 of 420 bits, 70.24% — the weakest of three metrics" is a single number that survives being questioned. It names what it measures, it is reproducible, and the choice of which metric governs is stated rather than hidden inside an average.
7. Common Mistakes
Averaging per-unit percentages. 76.13% against a true 81.70% in Chapter 11.4 — and here the mean moved opposite to the aggregate.
Pooling metrics into one denominator. 526 of 708 weights bits at 59% by accident of count — section 3.
Averaging metric figures. 71.00% of nothing — section 3.
Comparing a percentage across runs without the denominators. Every unit improved and the figure fell 13.39 points — section 4.
Reading a falling aggregate as a regression. The covered count more than doubled.
Reproducing a covergroup score as a bin ratio. The tool defines a weighted mean — Chapter 11.7.
Skipping the reconciliation against the report's own totals. It is the only step that catches a parse error — section 5.
Using a tolerance on integer counts, or == on computed areas. The exemption runs the other way — Chapter 11.1 section 8.
Quoting a number without its metric, aggregation and denominator. It cannot be compared to anything, including itself a week later.
8. Exercises
Exercise 1 — Aggregate one metric properly
awk '/^ Line Coverage/ { c += substr($4,2)+0; t += $6+0 } END { printf "line coverage: %d of %d = %.2f%%\n", c, t, 100*c/t }' cov.rptline coverage: 192 of 235 = 81.70%Exercise 2 — Every metric side by side
awk '/^ (Line|Toggle|Branch) Coverage/ { m = $1; c[m] += substr($4,2)+0; t[m] += $6+0 } END { split("Line Toggle Branch", o, " "); for (i=1;i<=3;i++) { k=o[i]; printf "%-8s %4d of %4d = %6.2f%%\n", k, c[k], t[k], 100*c[k]/t[k] } }' cov.rptLine 192 of 235 = 81.70%
Toggle 295 of 420 = 70.24%
Branch 39 of 53 = 73.58%Exercise 3 — Debug a coverage regression that is not one
A nightly dashboard reports line coverage falling from 66.47% to 53.08% and an engineer is asked to explain the thirteen-point regression.
for f in run1.csv run2.csv; do printf "%-10s " "$f"; awk -F, 'FNR>1 { c += $2; t += $3 } END { printf "%3d of %3d = %6.2f%%\n", c, t, 100*c/t }' $f; donerun1.csv 113 of 170 = 66.47%
run2.csv 276 of 520 = 53.08%The covered count went from 113 to 276 and the coverable count from 170 to 520. Per unit:
awk -F, 'NR==FNR { if (FNR>1) { c[$1]=$2; t[$1]=$3 } ; next } FNR>1 { printf "%-8s %6.2f%% -> %6.2f%% items %d -> %d\n", $1, 100*c[$1]/t[$1], 100*$2/$3, t[$1], $3 }' run1.csv run2.csvu_alu 75.00% -> 77.00% items 100 -> 100
u_ctrl 40.00% -> 45.00% items 50 -> 400
u_io 90.00% -> 95.00% items 20 -> 20Every unit improved. u_ctrl grew from 50 coverable statements to 400 — new RTL in the design's least-covered block — so the mix shifted and the aggregate fell. There is no regression to explain.
Exercise 4 — Debug a pooled figure
A report quotes 74.29% as "overall coverage".
awk '/^ (Line|Toggle|Branch) Coverage/ { m = $1; c[m] += substr($4,2)+0; t[m] += $6+0; C += substr($4,2)+0; T += $6+0 } END { split("Line Toggle Branch", o, " "); for (i=1;i<=3;i++) { k=o[i]; printf " %-8s %4d of %4d (%4.1f%% of the pooled denominator)\n", k, c[k], t[k], 100*t[k]/T } printf " pooled: %4d of %4d = %.2f%%\n", C, T, 100*C/T }' cov.rpt Line 192 of 235 (33.2% of the pooled denominator)
Toggle 295 of 420 (59.3% of the pooled denominator)
Branch 39 of 53 ( 7.5% of the pooled denominator)
pooled: 526 of 708 = 74.29%Toggle bits are 59.3% of the pooled denominator, so the "overall" figure is mostly a toggle figure, weighted by nothing more principled than how many bits the design has.
Exercise 5 — Engineering application
The coverage summary a sign-off pack needs: one row per metric with its own denominator, the report's totals reconciled, the night-over-night comparison stated in counts, and no blended number anywhere.
# covsum.awk — per-metric coverage summary with denominators.
#
# Usage: awk -f covsum.awk cov.rpt
# awk -v gate=80 -f covsum.awk cov.rpt
# Exit: 0 every metric at or above the gate
# 1 a metric is below the gate
# 2 a computed total does not match the report's own
BEGIN { if (gate == "") gate = 80; split("Line Toggle Branch", order, " ") }
/^ (Line|Toggle|Branch) Coverage/ {
m = $1
if ($3 == "n/a" || $6 + 0 == 0) { nskip[m]++; next }
c[m] += substr($4, 2) + 0
t[m] += $6 + 0
nu[m]++
}
# locate both counts by pattern, because the field holding the denominator
# differs between this report and fsm.rpt -- see section 5
/^TOTAL/ && match($0, /\([0-9]+ of [0-9]+\)/) {
split(substr($0, RSTART + 1, RLENGTH - 2), pp, / of /)
rc[$2] = pp[1] + 0; rt[$2] = pp[2] + 0
}
END {
printf "%-8s %6s %7s %7s %9s %s\n", "metric", "units", "covered", "total", "coverage", "note"
for (i = 1; i <= 3; i++) {
m = order[i]
if (t[m] == 0) { printf "%-8s %6s %7s %7s %9s no coverable items\n", m, "-", "-", "-", "-"; continue }
p = 100 * c[m] / t[m]
note = (p < gate ? "BELOW " gate "%" : "[ok]")
if (nskip[m]) note = note "; " nskip[m] " unit(s) with nothing to cover"
if (p < gate) nbelow++
printf "%-8s %6d %7d %7d %8.2f%% %s\n", m, nu[m], c[m], t[m], p, note
# the report prints its own totals -- the only check that catches a parse error
if (rt[m] != "" && (c[m] != rc[m] || t[m] != rt[m])) {
printf "ERROR: %s computed %d of %d, report says %d of %d\n", m, c[m], t[m], rc[m], rt[m] > "/dev/stderr"
bad++
}
}
printf "#\n# each row is aggregated over its OWN population; the rows are not combined\n"
printf "# statements, bits and branches are %d, %d and %d different items\n", t["Line"], t["Toggle"], t["Branch"]
if (bad) exit 2
if (nbelow) exit 1
}awk -f covsum.awk cov.rpt
echo "status: $?"metric units covered total coverage note
Line 6 192 235 81.70% [ok]
Toggle 6 295 420 70.24% BELOW 80%
Branch 5 39 53 73.58% BELOW 80%; 1 unit(s) with nothing to cover
#
# each row is aggregated over its OWN population; the rows are not combined
# statements, bits and branches are 235, 420 and 53 different items
status: 1Three rows, three denominators, no blended figure, and the n/a branch unit from Chapter 11.4 reported as having nothing to cover rather than counted as a zero.
The reconciliation against the report's own TOTAL rows runs silently when it passes. Giving it a report whose totals do not match is what proves it works:
sed 's/^TOTAL Line Coverage 81.70% (192 of 235)/TOTAL Line Coverage 81.70% (192 of 999)/' cov.rpt > badtot.rpt
awk -f covsum.awk badtot.rpt 2>&1 >/dev/null
echo "status: $?"ERROR: Line computed 192 of 235, report says 192 of 999
status: 2Exercise 6 — Challenge: show that no single number can be both comparable and complete
Produce the one figure a dashboard would show for each night, and show that whichever aggregation is chosen, the figure misrepresents one of the two true statements from section 4.
for f in run1.csv run2.csv; do printf " %-10s " "$f"; awk -F, 'FNR>1 { c += $2; t += $3; p += 100*$2/$3; n++; if (n == 1 || 100*$2/$3 < lo) lo = 100*$2/$3 } END { printf "aggregate %6.2f%% mean %6.2f%% worst unit %6.2f%% covered %3d coverable %3d\n", 100*c/t, p/n, lo, c, t }' $f; done run1.csv aggregate 66.47% mean 68.33% worst unit 40.00% covered 113 coverable 170
run2.csv aggregate 53.08% mean 72.33% worst unit 45.00% covered 276 coverable 520Four candidate headline figures, and each tells a different story about the same night:
- Aggregate, 66.47% → 53.08%: a regression. Contradicts "every unit improved".
- Mean, 68.33% → 72.33%: progress. Contradicts "a smaller fraction of the design is covered".
- Worst unit, 40.00% → 45.00%: progress, and it is the only one of the three that cannot be gamed by the mix — but it discards everything about the other units.
- Covered count, 113 → 276: progress, with no notion of how much is left.
No one of them is both comparable across the two nights and complete. The aggregate and the mean are each arithmetically correct and individually misleading, which is the strongest form of this module's recurring result — in Chapter 10.3 two counting methods disagreed because one was wrong; here both are right.
So the deliverable is not a number but a sentence, and the minimum honest form is the one the procedure in section 5 produces:
for f in run1.csv run2.csv; do awk -F, -v f=$f 'FNR>1 { c += $2; t += $3 } END { printf " %s: line coverage, aggregated over statements, %d of %d = %.2f%%\n", f, c, t, 100*c/t }' $f; done run1.csv: line coverage, aggregated over statements, 113 of 170 = 66.47%
run2.csv: line coverage, aggregated over statements, 276 of 520 = 53.08%With the counts present, a reader can see that the second night covers 163 more statements on a design that grew by 350 — and can reach their own conclusion, which is what reporting is for.
9. Summary
- Six metrics, six populations. Statements, bits, branches, states, arcs and covergroup items are not interchangeable.
- Pooling them gives
526 of 708, weighting bits at 59.3% by accident of count. - Averaging their figures gives 71.00% of nothing.
- Every unit improved, the aggregate fell 13.39 points, the mean rose 4.00 — opposite directions on identical data, and all three statements are true.
- The cause is the denominators. The weakest unit grew from 50 coverable items to 400.
- The aggregate answers the sign-off question; the mean answers "how are the units doing"; neither answers "is verification progressing".
- A figure compared across runs is meaningless without its denominators.
113 of 170to276 of 520is not a regression. - Reconcile against the report's own totals. It is the only step that detects a parse error, and integer counts compare with
==. - Name the governing metric rather than blending, when one figure is unavoidable.
- State the metric, the aggregation and the denominator. The deliverable is a sentence, not a number.
That closes Module 11. Synthesis reports hand you measurements whose labels collide; coverage reports hand you ratios whose denominators are the whole story. In both cases the report's own internal arithmetic is the only reference available, and every chapter here ended at the same place as Module 10 did — a check made on the input, before the number is believed.
Where this fits
Part of the awk curriculum.