Grouping Records

Part II · Module 6 · Chapter 6.3 · Grouping Records

1. Why This Matters

A report with one row per instance is data. A report with one row per module is information, and getting from the first to the second is grouping.

This is the shape of nearly every number an engineer quotes from a tool. Area per module. Violations per clock domain. Coverage per block. Failures per test. In every case the raw report has many rows per group and somebody wants one.

awk does it in one pass, because Chapter 6.1's arrays let you keep an accumulator per key without knowing the keys. The step from Chapter 6.2 is small: instead of adding 1, add a value.

The reason this chapter is longer than that step is section 6.

2. Grouping Is Counting With a Value

Chapter 6.2's idiom added 1 per record:

u_alu comb 1820
u_regfile seq 2410
u_alu comb 640
u_io pad 300
u_regfile seq 1100

awk '{ n[$1]++ } END { for (k in n) print k, n[k] | "sort" }' area.rpt

u_alu 2
u_io 1
u_regfile 2

Replace the 1 with a field and it becomes a per-key total:

awk '{ t[$1] += $3 } END { for (k in t) print k, t[k] | "sort" }' area.rpt

u_alu 2460
u_io 300
u_regfile 3510

u_alu has two rows summing to 2460, u_regfile two summing to 3510, u_io one at 300. The += works without initialisation for the same reason ++ did — an unset element is 0 in arithmetic (Chapter 5.1, section 3).

That is the entire mechanism. Everything else in this chapter is about reporting it honestly.

3. Total and Count Together

One array is rarely enough. Two keyed the same way give a total and its denominator:

awk '{ t[$1] += $3; c[$1]++ } END { for (k in t) printf "%-12s total %5d  n %d\n", k, t[k], c[k] | "sort" }' area.rpt

u_alu        total  2460  n 2
u_io         total   300  n 1
u_regfile    total  3510  n 2

The n column is Chapter 5.3's denominator habit applied per group, and it does real work here. u_io has a total of 300 from a single row; u_regfile has 3510 from two. Without the count those look like comparable measurements, and they are not — one is a measurement and the other is an aggregate.

Any number of arrays can share the key. There is nothing linking them except that you use the same subscript, which is also the one thing to be careful about — section 8.

4. Per-Key Mean

With a total and a count, the mean follows:

awk '{ t[$1] += $3; c[$1]++ } END { for (k in t) printf "%-12s total %5d  n %d  mean %7.1f\n", k, t[k], c[k], t[k]/c[k] | "sort" }' area.rpt

u_alu        total  2460  n 2  mean  1230.0
u_io         total   300  n 1  mean   300.0
u_regfile    total  3510  n 2  mean  1755.0

No division-by-zero guard is needed here, and it is worth being precise about why: c[k] can never be 0 for a key that exists, because the key only exists if a record created it. The guard Chapter 5.3 insisted on protects against an empty file, not an empty group — and a grouped report has no empty groups by construction.

What it does need is the same honesty about what the mean means. u_io's "mean of 300.0" is the mean of one value, which is just the value wearing a statistic's clothes. Printing n is what lets a reader see that.

5. Per-Key Maximum

A worst case per group needs a comparison rather than an accumulation:

awk '{ if ($3 + 0 > mx[$1] + 0) { mx[$1] = $3 + 0; which[$1] = $2 } } END { for (k in mx) printf "%-12s max %5d in %s\n", k, mx[k], which[k] | "sort" }' area.rpt

u_alu        max  1820 in comb
u_io         max   300 in pad
u_regfile    max  2410 in seq

Correct — 1820 is the larger of u_alu's two rows, 2410 of u_regfile's.

And this command contains a bug that this data cannot reveal.

6. The Per-Key Seeding Bug

Chapter 5.1 section 9 showed that a naive worst-case tracker fails silently when every value sits on one side of zero. Grouped, it fails worse.

A slack report with two clock domains — one failing timing, one meeting it:

clk_core path_a -0.124
clk_core path_b -0.081
clk_core path_c -1.200
clk_io path_d 0.250
clk_io path_e 0.410

Ask for the worst slack per domain, with section 5's idiom:

awk '{ if ($3 + 0 < worst[$1] + 0) { worst[$1] = $3 + 0; wp[$1] = $2 } } END { for (k in worst) printf "%-10s worst [%s] at [%s]\n", k, worst[k], wp[k] | "sort" }' slack.rpt

clk_core   worst [-1.2] at [path_c]
clk_io     worst [] at []

clk_core is right. clk_io is blank — the row is there, the values are not.

The mirror case is exactly as bad. Best slack per domain:

awk '{ if ($3 + 0 > best[$1] + 0) { best[$1] = $3 + 0; bp[$1] = $2 } } END { for (k in best) printf "%-10s best [%s] at [%s]\n", k, best[k], bp[k] | "sort" }' slack.rpt

clk_core   best [] at []
clk_io     best [0.41] at [path_e]

Now clk_core is the blank one.

Two rules from earlier chapters produce this, and a third decides how it looks.

The unset-0 seed is Chapter 5.1's bug: worst[$1] starts unset, which is 0 in a comparison, and every slack in clk_io is above 0, so $3 + 0 < 0 is never true and nothing is ever assigned.

The row appears anyway because of Chapter 6.1 section 5: worst[$1] + 0 is a read, and a read creates the element. So the key exists and for (k in worst) dutifully emits a row for it.

awk '{ if ($3 + 0 < worst[$1] + 0) worst[$1] = $3 + 0 } END { for (k in worst) printf "key %-10s value [%s]%s\n", k, worst[k], (worst[k] == "" ? "  <- created by the comparison, never assigned" : "") | "sort" }' slack.rpt

key clk_core   value [-1.2]
key clk_io     value []  <- created by the comparison, never assigned

7. Grouping by Two Keys

Chapter 6.1 section 11 established that a[x,y] joins subscripts with SUBSEP. That is how you group by a pair:

awk '{ t[$1,$2] += $3; c[$1,$2]++ } END { for (k in t) { split(k, p, SUBSEP); printf "%-12s %-6s %5d over %d row(s)\n", p[1], p[2], t[k], c[k] } }' area.rpt | sort

u_alu        comb    2460 over 2 row(s)
u_io         pad      300 over 1 row(s)
u_regfile    seq     3510 over 2 row(s)

Grouped by instance and kind together. Each distinct pair gets its own row.

Here the pair adds nothing, because in this data each instance has exactly one kind. It matters when a key appears with several values of the second field — a module containing both combinational and sequential cells, or a clock domain with both setup and hold paths. Then t[$1] alone collapses a distinction you wanted and t[$1,$2] keeps it.

The honest limitation from Chapter 6.1 stands: there is no way to ask for "every key whose first part is u_alu" without iterating everything and splitting. If that is the dominant question, keep a second array keyed on the first part alone, accumulated in the same pass.

8. The Key Is a String You Chose

The arrays in a grouping command are linked only by the subscript you write, which puts the burden on the key being right.

Chapter 6.1 section 3 established that subscripts are strings, so values that are numerically equal but textually different land in different groups:

u_alu 100
U_ALU 200
u_alu  300

awk '{ t[$1] += $2; c[$1]++ } END { for (k in t) printf "[%s] total %d over %d\n", k, t[k], c[k] | "sort" }' mixed.rpt

[U_ALU] total 200 over 1
[u_alu] total 400 over 2

Two groups where an engineer would see one module. The third record's extra space was absorbed by the default field splitting (Chapter 3.1), so u_alu correctly got two rows — but U_ALU is a separate group because case matters to a string subscript.

Normalising the key with tolower() merges them:

awk '{ k = tolower($1); t[k] += $2; c[k]++ } END { for (k in t) printf "[%s] total %d over %d\n", k, t[k], c[k] | "sort" }' mixed.rpt

[u_alu] total 600 over 3

One group, 600 over 3 rows, which is what an engineer reading the report would expect.

The per-record view confirms the normalisation rather than leaving it to inference:

awk '{ print NR, "[" $1 "]", "-> [" tolower($1) "]", $2 }' mixed.rpt

1 [u_alu] -> [u_alu] 100
2 [U_ALU] -> [u_alu] 200
3 [u_alu] -> [u_alu] 300

All three records map to the same key. tolower() is Module 8's function; the point here is where it goes — compute the key once into a variable, then use that variable as the subscript for every array in the group, so the arrays cannot drift apart.

Whether to normalise is a judgement about the data, not a rule. Case-folding instance names is usually right because tools are inconsistent about case and the engineer means one module. Case-folding a signal name can be wrong, because Verilog is case-sensitive and RESET and reset really are two nets. The question to answer before writing tolower() is whether the two spellings denote the same thing in the domain — and if you are unsure, group both ways and compare the row counts.

9. Groups of One

A group built from a single record is arithmetically fine and editorially dangerous:

awk '{ t[$1] += $3; c[$1]++ } END { for (k in t) printf "%-12s %5d  (%d row%s)%s\n", k, t[k], c[k], (c[k] == 1 ? "" : "s"), (c[k] == 1 ? "  <- single row, not an aggregate" : "") | "sort" }' area.rpt

u_alu         2460  (2 rows)
u_io           300  (1 row)  <- single row, not an aggregate
u_regfile     3510  (2 rows)

Flagging it costs one ternary and stops u_io's 300 being read as a comparable statistic.

This is the per-group version of Chapter 5.3's rule and the reason the n column has appeared in every table in this chapter. A mean over one value, a maximum over one value and a total over one value are all just that value.

10. Common Mistakes

A per-key extremum with no per-key seed. Produces blank rows for exactly the groups on the far side of zero — section 6.

Testing array[key] == "" to decide whether to seed. Creates the key, and is also true for a real value of zero. Use !(key in array).

Grouping on an unnormalised key. Case and spelling differences split one module into several groups — section 8.

Omitting the per-key count. A group of one and a group of fifty look identical.

Guarding per-key division against zero. Unnecessary: a key that exists has at least one record. Guard the empty file instead.

Grouping by one key when the data has two dimensions. t[$1] collapses a distinction t[$1,$2] would keep.

Relying on for (k in a) order. Still unspecified — every table here pipes to sort.

11. Exercises

Exercise 1 — Predict the output

awk 'BEGIN { t["a"] += 5; t["a"] += 3; t["b"] += 2; for (k in t) print k, t[k] | "sort" }'

a 8
b 2

Both += on a accumulated from an unset start of 0.

Exercise 2 — Total and count per group

Report the cell total and row count per instance.

u_alu comb 1820
u_regfile seq 2410
u_alu comb 640
u_io pad 300
u_regfile seq 1100

awk '{ t[$1] += $3; c[$1]++ } END { for (k in t) printf "%-12s %5d cells over %d row(s)\n", k, t[k], c[k] | "sort" }' area.rpt

u_alu         2460 cells over 2 row(s)
u_io           300 cells over 1 row(s)
u_regfile     3510 cells over 2 row(s)

Exercise 3 — Debug a blank grouped row

A timing summary reports the worst slack per clock domain and one domain comes back empty.

clk_core path_a -0.124
clk_core path_b -0.081
clk_core path_c -1.200
clk_io path_d 0.250
clk_io path_e 0.410

awk '{ if ($3 + 0 < worst[$1] + 0) { worst[$1] = $3 + 0; wp[$1] = $2 } } END { for (k in worst) printf "%-10s worst [%s] at [%s]\n", k, worst[k], wp[k] | "sort" }' slack.rpt

clk_core   worst [-1.2] at [path_c]
clk_io     worst [] at []

worst["clk_io"] starts unset, which compares as 0, and both of clk_io's slacks are above 0 — so the assignment never ran. The row exists only because worst[$1] + 0 read the element into being.

awk '!($1 in worst) { worst[$1] = $3 + 0; wp[$1] = $2; next } $3 + 0 < worst[$1] { worst[$1] = $3 + 0; wp[$1] = $2 } END { for (k in worst) printf "%-10s worst [%s] at [%s]\n", k, worst[k], wp[k] | "sort" }' slack.rpt

clk_core   worst [-1.2] at [path_c]
clk_io     worst [0.25] at [path_d]

Note which domain was blank: the one meeting timing. Asking for the maximum instead blanks the failing domain. Either way the empty row is the one you were looking at.

Exercise 4 — Two-key grouping

Group the area report by instance and logic kind together.

awk '{ t[$1,$2] += $3; c[$1,$2]++ } END { for (k in t) { split(k, p, SUBSEP); printf "%-12s %-6s %5d over %d\n", p[1], p[2], t[k], c[k] } }' area.rpt | sort

u_alu        comb    2460 over 2
u_io         pad      300 over 1
u_regfile    seq     3510 over 2

Exercise 5 — Debug a module that appears twice

A per-module area summary lists u_alu twice with different totals, and the engineer insists there is only one such module.

u_alu 100
U_ALU 200
u_alu  300

awk '{ t[$1] += $2; c[$1]++ } END { for (k in t) printf "[%s] total %d over %d\n", k, t[k], c[k] | "sort" }' mixed.rpt

[U_ALU] total 200 over 1
[u_alu] total 400 over 2

Two groups, because a subscript is a string and U_ALU is not the string u_alu. Note that the third record's extra space caused no trouble at all — default field splitting collapses runs of whitespace (Chapter 3.1), so only the case differed.

awk '{ k = tolower($1); t[k] += $2; c[k]++ } END { for (k in t) printf "[%s] total %d over %d\n", k, t[k], c[k] | "sort" }' mixed.rpt

[u_alu] total 600 over 3

One group, 600 over 3. The two partial totals summed to 600 all along — the bug never lost data, it split it, which is why neither row looked wrong on its own.

Exercise 6 — Engineering application

Produce a per-domain timing summary: path count, worst slack with its path, and the count of violating paths — flagging any domain represented by a single path.

awk '!($1 in worst) { worst[$1] = $3 + 0; wp[$1] = $2 } $3 + 0 < worst[$1] { worst[$1] = $3 + 0; wp[$1] = $2 } { n[$1]++; if ($3 + 0 < 0) v[$1]++ } END { for (d in n) printf "%-10s %d path(s), %d violating, worst %7.3f at %s%s\n", d, n[d], v[d] + 0, worst[d], wp[d], (n[d] == 1 ? "  <- single path" : "") | "sort" }' slack.rpt

clk_core   3 path(s), 3 violating, worst  -1.200 at path_c
clk_io     2 path(s), 0 violating, worst   0.250 at path_d

Every domain answered, with its denominator, its violation count and the worst path named. clk_core is entirely violating; clk_io is clean and still reports a worst-case slack, which is the margin it has.

Note the seeding rule has no next here, because a later rule must also run for the first record of each domain — so the second rule's < comparison re-tests the record just seeded, finds it is not less than itself, and harmlessly does nothing.

The engineering caveat: a slack of 0.250 is the smallest margin among the paths this report listed. A path the tool did not analyse — unconstrained, cut, or excluded by a false path — does not appear and is not covered by this number, which is why a timing signoff reads the unconstrained-path report alongside this one.

Exercise 7 — Challenge: prove every group got a real answer

Section 6's bug produces blank cells. Write a grouped worst-case report that cannot emit one, by checking its own output.

awk '!($1 in worst) { worst[$1] = $3 + 0; wp[$1] = $2; seeded[$1] = 1 } $3 + 0 < worst[$1] { worst[$1] = $3 + 0; wp[$1] = $2 } { n[$1]++ } END { bad = 0; for (d in n) if (!(d in seeded)) { printf "BUG: domain %s has %d record(s) and no seeded value\n", d, n[d]; bad++ } if (bad) exit 3; for (d in n) printf "%-10s worst %7.3f at %-8s over %d path(s)\n", d, worst[d], wp[d], n[d] | "sort" }' slack.rpt
echo "status: $?"

clk_core   worst  -1.200 at path_c   over 3 path(s)
clk_io     worst   0.250 at path_d   over 2 path(s)
status: 0

The seeded[] array records which keys were explicitly initialised. In END, any key that has records but was never seeded is reported as a bug in the script, not as missing data — and the non-zero exit makes it impossible to ignore.

Running the check against the naive version proves the check can fail, which is the only thing that makes it worth having:

awk '{ if ($3 + 0 < worst[$1] + 0) { worst[$1] = $3 + 0; seeded[$1] = 1 } n[$1]++ } END { bad = 0; for (d in n) if (!(d in seeded)) { printf "BUG: domain %s has %d record(s) and no seeded value\n", d, n[d]; bad++ } if (bad) exit 3; print "all domains seeded" }' slack.rpt
echo "status: $?"

BUG: domain clk_io has 2 record(s) and no seeded value
status: 3

Caught, named, and non-zero. A guard that has never been shown failing is not a guard — it is a comment.

12. Summary

• Grouping is counting with a value: t[key] += field instead of n[key]++.
• Several arrays can share one key, giving total, count, maximum and more in a single pass.
• Always carry the per-key count. A group of one and a group of fifty are not comparable measurements.
• Per-key division needs no zero guard — a key that exists has at least one record. Guard the empty file.
• A per-key extremum needs a per-key seed. !(key in array) is true exactly once per key.
• Without it, the table has blank rows — present keys, empty values — because the unset-0 seed never matched and the comparison's read created the key anyway.
• The blank rows are the groups you were investigating: all-positive domains for a minimum, all-negative for a maximum.
• Never test array[key] == "" to decide whether to seed. It creates the key and is true for a real zero.
• Normalise the key deliberately — case and spelling split one module into several groups.
• a[x,y] groups by a pair, recovered with split() on SUBSEP.

You can now turn a raw report into a grouped table and defend every number in it.

Chapter 6.4 — Summary Calculations takes the grouped arrays and produces the figures a human reads — total, average, minimum, maximum and percentage — with the empty-input handling that stops a summary of nothing from looking like a summary of something.