awk Part IV · Module 13 · Real-World AWK Engineering Projects

Project: Regression Metrics Exporter

The last of the five scripts combines the other four into one dated row, and the design question is sharper than it looks: what must a row contain to still be comparable in six months, after the reports have changed format and nobody remembers how the figures were computed? The answer is counts rather than percentages, denominators beside every numerator, a schema number, and — the part most rows omit — a weighted mean stored as its numerator and denominator so the weighting itself stays auditable.

Advanced19 min readawkPart IVProjectMetricsSchema

Part IV · Module 13 · Chapter 13.5 · Project: Regression Metrics Exporter

1. What the Script Has to Do

One row per night, appended to a file a dashboard reads. Four requirements, and the fourth is the one that decides whether the row is still useful next quarter:

Combine all four extractors' results into one record.
Carry an identity — the run date and the design revision — or the row cannot be joined to anything.
Store counts, not percentages, so a reader can compute any ratio and none is frozen in.
Carry a schema number, because the row outlives the script that wrote it.

2. Read the Delimited Outputs, Not the Reports

Each of the previous four chapters produced two things: a formatted report for a person and a delimited stream for a program. The exporter reads the second kind, and the reason is measured rather than stylistic.

Across Chapters 13.1 to 13.4, four index-based re-parses of a formatted line took the wrong field: a test-state grep that also matched a signature row, a $(NF-1) that landed on a test name, a $3 that returned module(s);, and an $11 carrying a stray comma. Every one was in an exercise written by someone who had just read the script that produced the line.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

cat > reg.tsv <<'EOF'
test	state	errors
alu_smoke	PASS	0
alu_random	FAIL	2
alu_corner	FAIL	2
fifo_basic	PASS	0
fifo_stress	FAIL	2
dma_desc	FAIL	1
dma_stress	FATAL	0
cdc_sync	NOLOG	0
io_loopback	EMPTY	0
EOF
cat > sig.tsv <<'EOF'
signature	records	tests
MISMATCH ./env/sb.sv(148)	5	3
OVERFLOW ./env/fifo_sb.sv(77)	1	1
MISMATCH ./env/dma_sb.sv(203)	1	1
NOCLK ./env/drv.sv(12)	1	1
EOF
cat > viol.tsv <<'EOF'
startpoint	endpoint	clock	check	slack
u_regfile/dout_reg[3]	u_alu/sum_reg[7]	clk_core	setup	-0.018
u_dma/len_reg[4]	u_dma/cnt_reg[4]	clk_core	setup	-0.112
u_alu/acc_reg[0]	u_alu/acc_reg[1]	clk_core	hold	-0.041
u_io/pad_reg[0]	u_io/out_reg[0]	clk_io	hold	-0.023
u_mem/rd_reg[0]	u_mem/dout_reg[0]	clk_ddr	setup	-0.205
EOF
cat > cov.csv <<'EOF'
block,weight,line_cov,line_tot,toggle_cov,toggle_tot,branch_cov,branch_tot,functional
u_alu,3,520,560,410,620,88,96,72.50
u_regfile,1,210,210,248,256,18,18,98.00
u_ctrl,2,300,420,190,380,52,88,61.00
u_io,1,96,96,64,64,0,0,
u_dma,2,180,400,120,340,30,70,44.50
u_debug,0,0,120,0,160,0,24,0.00
EOF
awk 'FNR == 1 { f = FILENAME; sub(/.*\//, "", f); printf "  %-10s %d field(s) in the header\n", f, NF }' FS='[\t,]' reg.tsv sig.tsv viol.tsv cov.csv

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  reg.tsv    3 field(s) in the header
  sig.tsv    3 field(s) in the header
  viol.tsv   5 field(s) in the header
  cov.csv    9 field(s) in the header

Each file has a fixed field count and a header naming the fields — the two properties a formatted table cannot have, because its annotations are optional and its columns are padded.

3. The Procedure

Figure 1 — The exporter. It identifies each input by filename rather than by argument position, refuses if any is absent, and emits one row of counts with an identity and a schema number. Nothing it writes is a ratio.

4. The Complete Script

Azvya Education Pvt. Ltd.VLSI Mentor

export.awk — synthetic teaching script.

# export.awk — regression metrics exporter.
#
# Usage:  awk -v run=2026-10-12 -v rev=a1b2c3d -f export.awk \
#             reg.tsv sig.tsv viol.tsv cov.csv
# Output: one key=value row on stdout
# Exit:   0 clean   1 failures present   2 an input was missing or unreadable
#
# Reads the DELIMITED outputs of the earlier scripts, never their formatted
# reports -- four index-based re-parses of a formatted line took the wrong
# field across Chapters 13.1 to 13.4.                    (12.4, 13.3, 13.4)
#
# Design notes:
#   - Every value is a COUNT or a (count, total) pair. No percentages, so any
#     ratio can be recomputed and none is baked in.                 (11.8)
#   - The row carries schema=N. A row without one cannot be compared across a
#     format change, which is the whole point of storing it.
#   - The row carries run= and rev=, because a metrics row with no identity
#     cannot be joined to anything.
#   - Inputs are identified by FILENAME, not by argument position, so the
#     caller may pass them in any order.
#   - A missing input is exit 2, never a zero: an absent file and a clean
#     result are different facts.                              (12.1, 12.6)

BEGIN { SCHEMA = 1; if (run == "") run = "unknown"; if (rev == "") rev = "unknown" }

FNR == 1 { f = FILENAME; sub(/.*\//, "", f); which = f; seen[f] = 1; next }

which == "reg.tsv"  { split($0, a, "\t"); ntest++; state[a[2]]++; errs += a[3] + 0 }
which == "sig.tsv"  { split($0, a, "\t"); nsig++; if (a[3] + 0 > 1) nshared++ }
which == "viol.tsv" { split($0, a, "\t"); nviol++
                      v = a[5] + 0
                      if (nviol == 1 || v < worst) { worst = v; worstkey = a[4] "/" a[3] } }
which == "cov.csv"  { split($0, a, ",")
                      lc += a[3]; lt += a[4]; tc += a[5]; tt += a[6]; bc += a[7]; bt += a[8]
                      if (a[9] != "" && a[2] + 0 > 0) { fnum += a[2] * a[9]; fden += a[2] }
                      if (a[2] + 0 == 0) nzw++ }

END {
    split("reg.tsv sig.tsv viol.tsv cov.csv", need, " ")
    for (i = 1; i <= 4; i++) if (!(need[i] in seen)) { miss = miss " " need[i] }
    if (miss != "") { printf "ERROR: missing input(s):%s\n", miss > "/dev/stderr"; exit 2 }

    ran = state["PASS"] + state["FAIL"]
    unusable = ntest - ran

    printf "schema=%d run=%s rev=%s", SCHEMA, run, rev
    printf " tests=%d ran=%d passed=%d failed=%d errors=%d unusable=%d", ntest, ran, state["PASS"] + 0, state["FAIL"] + 0, errs + 0, unusable
    printf " nolog=%d empty=%d fatal=%d", state["NOLOG"] + 0, state["EMPTY"] + 0, state["FATAL"] + 0
    printf " signatures=%d shared_signatures=%d", nsig + 0, nshared + 0
    printf " violations=%d worst_slack=%.3f worst_group=%s", nviol + 0, worst + 0, (worstkey == "" ? "none" : worstkey)
    printf " line_cov=%d line_tot=%d toggle_cov=%d toggle_tot=%d branch_cov=%d branch_tot=%d", lc, lt, tc, tt, bc, bt
    printf " func_weighted_num=%.2f func_weight=%d zero_weight_blocks=%d", fnum + 0, fden + 0, nzw + 0
    print ""

    if (unusable) exit 2
    if (state["FAIL"] || nviol) exit 1
}

Three choices worth naming.

which = FILENAME rather than an argument index. A caller that passes the files in a different order, or that passes an absolute path, still works — and the seen[] array then gives the missing-input check for free.

FNR == 1 { … ; next } skips each file's header. Chapter 11.7 measured what happens without it: the header row parses as data and the arithmetic runs on field names.

worst_slack is stored with its group. -0.205 alone would be the figure Chapter 13.2 spent a section refusing to produce; worst_group=setup/clk_ddr beside it makes it a statement rather than a number.

5. The Row

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -v run=2026-10-12 -v rev=a1b2c3d -f export.awk reg.tsv sig.tsv viol.tsv cov.csv
echo "status: $?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

schema=1 run=2026-10-12 rev=a1b2c3d tests=9 ran=6 passed=2 failed=4 errors=7 unusable=3 nolog=1 empty=1 fatal=1 signatures=4 shared_signatures=1 violations=5 worst_slack=-0.205 worst_group=setup/clk_ddr line_cov=1306 line_tot=1806 toggle_cov=1032 toggle_tot=1820 branch_cov=188 branch_tot=296 func_weighted_num=526.50 func_weight=8 zero_weight_blocks=1
status: 2

Status 2, because three tests produced nothing usable — the same grading as Chapter 12.6, and the reason the row is still emitted is that a dashboard needs to record the night even when the night was broken.

6. Every Ratio Is Still Recomputable

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -v run=2026-10-12 -v rev=a1b2c3d -f export.awk reg.tsv sig.tsv viol.tsv cov.csv 2>/dev/null > row.txt
awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); v[kv[1]] = kv[2] } }
 END { printf "  pass rate (ran)       %d of %d = %6.2f%%\n", v["passed"], v["ran"], 100*v["passed"]/v["ran"]
       printf "  pass rate (attempted) %d of %d = %6.2f%%\n", v["passed"], v["tests"], 100*v["passed"]/v["tests"]
       printf "  line coverage         %d of %d = %6.2f%%\n", v["line_cov"], v["line_tot"], 100*v["line_cov"]/v["line_tot"]
       printf "  toggle coverage       %d of %d = %6.2f%%\n", v["toggle_cov"], v["toggle_tot"], 100*v["toggle_cov"]/v["toggle_tot"]
       printf "  branch coverage       %d of %d = %6.2f%%\n", v["branch_cov"], v["branch_tot"], 100*v["branch_cov"]/v["branch_tot"]
       printf "  functional weighted   %.2f / %d = %6.2f%%\n", v["func_weighted_num"], v["func_weight"], v["func_weighted_num"]/v["func_weight"]
       printf "  errors per failing    %d / %d = %.2f\n", v["errors"], v["failed"], v["errors"]/v["failed"] }' row.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  pass rate (ran)       2 of 6 =  33.33%
  pass rate (attempted) 2 of 9 =  22.22%
  line coverage         1306 of 1806 =  72.31%
  toggle coverage       1032 of 1820 =  56.70%
  branch coverage       188 of 296 =  63.51%
  functional weighted   526.50 / 8 =  65.81%
  errors per failing    7 / 4 = 1.75

A weighted mean stored as its numerator and denominator

func_weighted_num=526.50 func_weight=8 is the field pair most metrics rows get wrong, and the recomputation above is why it is worth the two columns.

Had the row stored func_coverage=65.81, three things would be impossible:

Re-weighting. If a later reviewer decides u_debug should carry weight 1 instead of 0, the row cannot be adjusted — the old weighting is baked into the quotient.
Auditing. 65.81% does not say what it is 65.81% of. Chapter 11.7 measured a covergroup whose score excluded a 66.67%-covered coverpoint entirely, and only the weight column revealed it. Here func_weight=8 against six blocks says immediately that not every block contributed.
Aggregating across nights. Two nights' quotients cannot be combined; two nights' numerator-and-weight pairs can.

526.50 / 8 = 65.81 recovers the figure exactly, so nothing is lost by storing the pair — and zero_weight_blocks=1 beside it is the flag that says one block is missing from that denominator.

The same principle is why line_cov and line_tot are separate fields rather than line_pct. It is Chapter 11.8's rule stated as a storage format: a stored figure must carry its denominator, because the aggregation that produced it is a decision and decisions get revisited.

7. The Refusal, and Accumulation

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -v run=2026-10-12 -f export.awk reg.tsv sig.tsv cov.csv 2>&1 >/dev/null
echo "status: $?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

ERROR: missing input(s): viol.tsv
status: 2

Named rather than inferred, and exit 2 rather than a row with violations=0 — which would have been indistinguishable from a night with no timing violations. Chapter 12.1 section 6 measured that an absent input and a zero result collapse into the same value unless something refuses.

Two nights in one file:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -v run=2026-10-11 -v rev=9f8e7d6 -f export.awk reg.tsv sig.tsv viol.tsv cov.csv 2>/dev/null > hist.txt
cat row.txt >> hist.txt
awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); v[kv[1]] = kv[2] }
  printf "  %-11s tests=%-3s failed=%-3s unusable=%-3s line=%d/%d worst=%s\n", v["run"], v["tests"], v["failed"], v["unusable"], v["line_cov"], v["line_tot"], v["worst_slack"] }' hist.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  2026-10-11  tests=9   failed=4   unusable=3   line=1306/1806 worst=-0.205
  2026-10-12  tests=9   failed=4   unusable=3   line=1306/1806 worst=-0.205

Identical, because both rows were generated from the same inputs — which is the right behaviour to check first. A format whose two rows differ when the inputs did not has a timestamp or an ordering leaking into it.

Note line=1306/1806 rather than 72.31%. Chapter 11.8's measurement was that a coverage percentage compared across runs is meaningless without its denominators: every unit improved while the aggregate fell thirteen points, purely because the denominator grew. A row of counts makes that visible; a row of percentages hides it.

8. Every Decision and Its Source

decision	alternative that fails	measured in
read the delimited outputs	four index re-parses took the wrong field	12.4, 13.1, 13.3, 13.4
identify inputs by `FILENAME`	argument order becomes part of the interface	—
`FNR == 1 { next }` per file	the header parses as data	11.7
refuse a missing input with exit 2	`violations=0` is indistinguishable from absent	12.1, 12.6
counts, never percentages	a stored percentage cannot be re-aggregated	11.8
`cov` and `tot` as separate fields	a figure without its denominator is not comparable	11.8
weighted mean as numerator + weight	the weighting is baked in and unauditable	11.7
`zero_weight_blocks` as its own field	a block absent from a denominator is invisible	11.7
`worst_group` beside `worst_slack`	a bare worst slack mixes setup with hold	10.5, 13.2
`schema=1`	the row cannot survive a format change	—
`run=` and `rev=`	the row cannot be joined to anything	—
emit the row even on exit 2	the broken night goes unrecorded	12.6

Twelve decisions. Eleven of them answer a failure measured earlier in this track; the two without a chapter reference — the schema number and the identity — are the ones that come from the row being stored rather than read.

9. Common Mistakes

Parsing the formatted reports. Four re-parses took the wrong field across Module 13 — section 2.

Identifying inputs by position. The caller's argument order becomes part of the interface.

Forgetting FNR == 1 { next }. The header row parses as data.

A missing input as a zero. violations=0 and "no timing report" are different facts.

Storing percentages. They cannot be re-aggregated or re-weighted.

Storing a weighted mean as a quotient. The weighting is then unauditable — section 6.

Omitting the zero-weight count. A block excluded from a denominator leaves no trace.

A bare worst_slack. It mixes setup with hold; carry the group.

No schema number. The row outlives the script and cannot be compared across a change.

No run identity. The row cannot be joined to a commit, a date or anything else.

Suppressing the row on a broken night. The night then has no record at all.

10. Exercises

Exercise 1 — The row, field by field

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -v run=2026-10-12 -v rev=a1b2c3d -f export.awk reg.tsv sig.tsv viol.tsv cov.csv 2>/dev/null | awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); printf "  %-20s %s\n", kv[1], kv[2] } }'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  schema               1
  run                  2026-10-12
  rev                  a1b2c3d
  tests                9
  ran                  6
  passed               2
  failed               4
  errors               7
  unusable             3
  nolog                1
  empty                1
  fatal                1
  signatures           4
  shared_signatures    1
  violations           5
  worst_slack          -0.205
  worst_group          setup/clk_ddr
  line_cov             1306
  line_tot             1806
  toggle_cov           1032
  toggle_tot           1820
  branch_cov           188
  branch_tot           296
  func_weighted_num    526.50
  func_weight          8
  zero_weight_blocks   1

Twenty-six fields. Three are identity, one is the worst-slack group, and every other one is an integer or a numerator-denominator half.

Exercise 2 — The triage summary a human would read

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -v run=2026-10-12 -f export.awk reg.tsv sig.tsv viol.tsv cov.csv 2>/dev/null | awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); v[kv[1]] = kv[2] } }
 END { printf "  %s: %d of %d tests passed (%d unusable); %d signature(s), %d shared\n", v["run"], v["passed"], v["tests"], v["unusable"], v["signatures"], v["shared_signatures"]
       printf "  timing: %d violation(s), worst %s in %s\n", v["violations"], v["worst_slack"], v["worst_group"]
       printf "  coverage: line %.1f%%, toggle %.1f%%, functional %.1f%% over weight %d\n", 100*v["line_cov"]/v["line_tot"], 100*v["toggle_cov"]/v["toggle_tot"], v["func_weighted_num"]/v["func_weight"], v["func_weight"] }'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  2026-10-12: 2 of 9 tests passed (3 unusable); 4 signature(s), 1 shared
  timing: 5 violation(s), worst -0.205 in setup/clk_ddr
  coverage: line 72.3%, toggle 56.7%, functional 65.8% over weight 8

Three lines a person can read, every figure computed from the row's counts at the point of display — so the stored row stays neutral and the presentation layer decides what to show. functional 65.8% over weight 8 carries the denominator with it, which is what makes it comparable to next week's.

The numerator and weight are available separately when the weighting itself is the question:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -v run=2026-10-12 -f export.awk reg.tsv sig.tsv viol.tsv cov.csv 2>/dev/null | awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); v[kv[1]] = kv[2] } }
 END { printf "  functional numerator %s, weight %s, quotient %.2f%%\n", v["func_weighted_num"], v["func_weight"], v["func_weighted_num"]/v["func_weight"] }'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  functional numerator 526.50, weight 8, quotient 65.81%

526.50 / 8 = 65.81 — and weight 8 against six blocks is the signal that one block did not contribute, which section 6's callout is about.

Exercise 3 — Debug a row with a zero that means absent

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -v run=2026-10-12 -f export.awk reg.tsv sig.tsv cov.csv 2>&1 >/dev/null
echo "status: $?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

ERROR: missing input(s): viol.tsv
status: 2

Without the seen[] check the row would have carried violations=0 worst_slack=0.000 worst_group=none, which is exactly what a night with clean timing looks like.

Exercise 4 — Debug a header parsed as data

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'FNR == 1 { f = FILENAME; sub(/.*\//, "", f); which = f; seen[f] = 1 } which == "reg.tsv" { split($0, a, "\t"); n++; if (!(a[2] in st)) { st[a[2]] = 1; o[++k] = a[2] } } END { printf "  WITHOUT the next: %d row(s), %d distinct state(s), extra key present: %s\n", n, k, (("state" in st) ? "yes -- the header" : "no") }' reg.tsv
awk 'FNR == 1 { f = FILENAME; sub(/.*\//, "", f); which = f; seen[f] = 1; next } which == "reg.tsv" { split($0, a, "\t"); n++; if (!(a[2] in st)) { st[a[2]] = 1; o[++k] = a[2] } } END { printf "  WITH    the next: %d row(s), %d distinct state(s), extra key present: %s\n", n, k, (("state" in st) ? "yes -- the header" : "no") }' reg.tsv

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  WITHOUT the next: 10 row(s), 6 distinct state(s), extra key present: yes -- the header
  WITH    the next: 9 row(s), 5 distinct state(s), extra key present: no

Ten rows and six states without the next — the header's second field, the literal word state, became a test state, and tests=10 would have gone into the row. The check tests for the key by name rather than listing the keys, because for (s in st) has no defined order and an order-dependent listing differs between runs.

Exercise 5 — Engineering application

Azvya Education Pvt. Ltd.VLSI Mentor

nightly_export.sh — synthetic teaching script.

#!/bin/sh
# nightly_export.sh — run the exporter and append the row to the history.
#
# The row is appended whatever the status, because a broken night still needs
# a record (Chapter 12.6). The status decides what gets SAID, not what gets
# stored.

run=${1:-$(date +%Y-%m-%d)}
rev=${2:-unknown}

awk -v run="$run" -v rev="$rev" -f export.awk reg.tsv sig.tsv viol.tsv cov.csv > row.new 2> row.err
status=$?

if [ -s row.new ]; then
  cat row.new >> metrics.log
  echo "row appended to metrics.log ($(wc -l < metrics.log | tr -d ' ') row(s) total)"
fi

case $status in
  0) echo "CLEAN" ;;
  1) echo "FAILURES: see the row" ;;
  2) echo "RUN NOT CONCLUSIVE:"; cat row.err ;;
  *) echo "EXPORTER FAILED ($status)" ;;
esac
exit $status

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

rm -f metrics.log
sh nightly_export.sh 2026-10-11 9f8e7d6 > /dev/null
sh nightly_export.sh 2026-10-12 a1b2c3d
echo "status: $?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

row appended to metrics.log (2 row(s) total)
RUN NOT CONCLUSIVE:
status: 2

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); v[kv[1]] = kv[2] } printf "  %s %s  tests=%s failed=%s unusable=%s\n", v["run"], v["rev"], v["tests"], v["failed"], v["unusable"] }' metrics.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  2026-10-11 9f8e7d6  tests=9 failed=4 unusable=3
  2026-10-12 a1b2c3d  tests=9 failed=4 unusable=3

Two rows, each with its own identity, appended regardless of status. row.err is empty in the second run's output because the missing-input error is the only thing written there and all four inputs were present — the RUN NOT CONCLUSIVE came from exit 2 for the three unusable tests, which is reported in the row rather than on stderr.

Exercise 6 — Challenge: show that the row survives a schema change and a percentage row does not

Suppose a later version adds a field. Show that a reader written against schema=1 still works, and that a row of percentages could not have supported the same change.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '{ print $0 " assertions=142 assertions_passed=140" }' row.txt > row_v2.txt
awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); v[kv[1]] = kv[2] }
  printf "  schema %s: line %d of %d = %.2f%%", v["schema"], v["line_cov"], v["line_tot"], 100*v["line_cov"]/v["line_tot"]
  printf "   assertions: %s\n", (("assertions" in v) ? v["assertions_passed"] "/" v["assertions"] : "not in this schema") }' row.txt row_v2.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  schema 1: line 1306 of 1806 = 72.31%   assertions: not in this schema
  schema 1: line 1306 of 1806 = 72.31%   assertions: 140/142

The same reader handles both rows, because key=value pairs are self-describing and an unknown key is simply absent from v[]. The schema field is what tells the reader whether to expect it — a row still at schema=1 with assertion fields is a producer that forgot to bump it, which is itself worth detecting.

Now the counterfactual. Suppose the row had stored percentages:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); v[kv[1]] = kv[2] } }
 END { printf "  stored as counts:      line %d/%d -> can re-weight, re-aggregate, and combine with another night\n", v["line_cov"], v["line_tot"]
       printf "  stored as a percentage: line %.2f%% -> cannot recover either number\n", 100*v["line_cov"]/v["line_tot"]
       printf "  two nights of counts:   (%d+%d) of (%d+%d) = %.2f%%\n", v["line_cov"], v["line_cov"], v["line_tot"], v["line_tot"], 100*(2*v["line_cov"])/(2*v["line_tot"])
       printf "  two nights of percentages: mean of %.2f%% and %.2f%% = %.2f%% -- which is only right because the denominators happen to match\n", 100*v["line_cov"]/v["line_tot"], 100*v["line_cov"]/v["line_tot"], 100*v["line_cov"]/v["line_tot"] }' row.txt

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  stored as counts:      line 1306/1806 -> can re-weight, re-aggregate, and combine with another night
  stored as a percentage: line 72.31% -> cannot recover either number
  two nights of counts:   (1306+1306) of (1806+1806) = 72.31%
  two nights of percentages: mean of 72.31% and 72.31% = 72.31% -- which is only right because the denominators happen to match

The last line is the point, and it is Chapter 11.8's result restated as a storage argument: averaging two nights' percentages gives the right answer only when the denominators are equal, and Chapter 11.8 measured a case where a growing denominator moved the aggregate thirteen points against every individual unit. A row of counts handles that; a row of percentages cannot, and nothing in the stored row would reveal which situation you were in.

So the schema number protects the row's shape and the counts protect its meaning. Both are cheap, both are easy to omit, and both are only missed months later when the row is the only record of a night nobody remembers.

11. Summary

One row per night, combining all four extractors, appended to a file a dashboard reads.
Read the delimited outputs, never the formatted reports — four index-based re-parses took the wrong field across Module 13.
Identify inputs by FILENAME, so argument order is not part of the interface, and get the missing-input check for free.
FNR == 1 { next } per file, or the header becomes a row and invents a state called state.
A missing input is exit 2, because violations=0 looks exactly like a clean night.
Counts, never percentages. Twenty-six fields, of which twenty-two are integers or numerator-denominator halves.
A weighted mean is stored as its numerator and its weight — 526.50 and 8 recover 65.81% and keep the weighting auditable.
zero_weight_blocks is its own field, because a block excluded from a denominator is otherwise invisible.
worst_group travels with worst_slack, so the figure does not mix setup with hold.
schema=, run= and rev= are what make the row comparable in six months.
Emit the row even when the status is 2. A broken night still needs a record.
When a derived figure disagrees with a verified one, print the inputs to the formula — Exercise 2's 56.7% was an argument-count slip, not a data problem.

That closes Module 13. Five scripts, and not one of them contains a technique Modules 9 to 12 had not already measured — what Part IV adds is the integration, the refusals, and the delimited interfaces that let the five compose into one row.

Chapter 14.1 — Debugging Incorrect or Empty Output opens the last module, and starts where every one of these scripts started: awk printed nothing, and there is an order in which to ask why.

Where this fits

Part of the awk curriculum.