Frequency Counting

Part II · Module 6 · Chapter 6.2 · Frequency Counting

1. Why This Matters

Counting occurrences is the most common thing anybody does to a log. How many of each error? How many cells of each type? How many tests per status?

Chapter 6.1 gave you everything needed to do it, and section 2 is the one line. If this chapter were about syntax it would end there.

It is not, because a count is an answer to a question you did not state, and the questions are easy to confuse. A number labelled "errors: 47" could be 47 records, 47 distinct failures, or 47 occurrences of one bug. Those lead to different decisions, and awk prints all three identically.

The five sections after the idiom are the five ways I have seen this go wrong, each with the output that shows it.

2. The Idiom

TIMEOUT uart
ASSERT alu
TIMEOUT fifo
ASSERT alu
TIMEOUT spi
XCHECK ddr

awk '{ n[$1]++ } END { for (sig in n) print n[sig], sig }' errors.txt

1 XCHECK
3 TIMEOUT
2 ASSERT

That is the whole thing. n[$1]++ builds one counter per distinct value of field 1, and the END loop reports them.

No list of expected signatures, no initialisation, no second pass over the file. The program did not know TIMEOUT existed until it read it.

3. Why It Needs No Initialisation

The reason is two rules you already have. Chapter 6.1: an array entry is created by being used. Chapter 5.1: an unset value is 0 in arithmetic.

So on a signature's first record, n["TIMEOUT"] does not exist, reads as 0, and ++ makes it 1. On the fourth record with that signature it is already 3 and becomes 4. The same expression handles both cases.

Compare with what you would write without arrays:

awk '$1 == "TIMEOUT" { t++ } $1 == "ASSERT" { a++ } $1 == "XCHECK" { x++ } END { print t+0, a+0, x+0 }' errors.txt

3 2 1

Same numbers, and it only works because I read the file first and knew the three signatures. A fourth signature appearing next week is counted by neither branch and vanishes silently — a new failure mode becoming invisible precisely because it is new.

That is the real argument for the array form. It is not brevity; it is that a counter keyed on the data cannot miss a category it has not seen before.

4. Counting by a Different Key

Nothing about the idiom is tied to $1. Counting by the test name instead:

awk '{ n[$2]++ } END { for (t in n) print n[t], t | "sort -rn" }' errors.txt

2 alu
1 uart
1 spi
1 fifo
1 ddr

Five distinct tests, with alu appearing twice.

The key can be any expression — a field, part of a field, a combination, or something computed. Counting by the first character:

awk '{ n[substr($1, 1, 1)]++ } END { for (c in n) print c, n[c] | "sort" }' errors.txt

A 2
T 3
X 1

substr() is Module 8's function. The point is that the subscript is just a string expression, so "count by X" needs only an expression producing X.

5. Records Are Not Things

The first way a count misleads. Here is a simulation log where one signature repeats:

UVM_WARNING clk_gate_unused env
UVM_WARNING clk_gate_unused env
UVM_WARNING clk_gate_unused env
UVM_WARNING clk_gate_unused env
UVM_WARNING clk_gate_unused env
UVM_ERROR scoreboard_mismatch alu
UVM_FATAL reset_deadlock ddr

Counting records per severity:

awk '{ n[$1]++ } END { for (s in n) print n[s], s | "sort -rn" }' sim.log

5 UVM_WARNING
1 UVM_FATAL
1 UVM_ERROR

Seven records. Now the same log counted two ways at once — records, and distinct tests affected:

awk '{ recs[$1]++; if (!(($1,$3) in seen)) { seen[$1,$3]; tests[$1]++ } } END { for (s in recs) printf "%-12s %d record(s) across %d test(s)\n", s, recs[s], tests[s] | "sort" }' sim.log

UVM_ERROR    1 record(s) across 1 test(s)
UVM_FATAL    1 record(s) across 1 test(s)
UVM_WARNING  5 record(s) across 1 test(s)

The warning fired five times and all five were the same test. 5 is a count of lines; 1 is a count of affected tests, and only the second is a measure of how widespread the problem is.

The distinct count works by keying on the pair — severity and test together (Chapter 6.1, section 11) — and testing the pair with in:

The engineering form of this point: a record count measures how loud a problem is, not how big it is. One deadlock that prints every cycle produces thousands of records; one wrong result prints once.

6. The Most Frequent Is Rarely the Most Important

The second way a count misleads, and the one that matters most in practice.

Counting the same log by message id, carrying the severity along:

awk '{ n[$2]++; sev[$2] = $1 } END { for (k in n) printf "%-22s %d  %s\n", k, n[k], sev[k] | "sort -k2 -rn" }' sim.log

clk_gate_unused        5  UVM_WARNING
scoreboard_mismatch    1  UVM_ERROR
reset_deadlock         1  UVM_FATAL

The table is sorted most-frequent-first, which puts a harmless warning at the top and a fatal deadlock at the bottom.

clk_gate_unused is an environment warning that fires five times per run and has fired five times per run for a year. reset_deadlock is a UVM_FATAL — the run died. Sorted by frequency, the thing that killed the simulation is the last row.

7. Percentages and the Denominator

A count without its total is hard to act on. The share of each signature:

awk '{ n[$1]++ } END { for (k in n) printf "%-8s %3d  %5.1f%%\n", k, n[k], 100*n[k]/NR | "sort -k2 -rn" }' errors.txt

TIMEOUT    3   50.0%
ASSERT     2   33.3%
XCHECK     1   16.7%

NR is the denominator — the total records read — which is safe here because nothing was filtered. Chapter 5.6 established that NR counts lines read, not records kept, so the moment a next appears in the program this breaks:

awk '/^#/ { next } { n[$1]++ } END { for (k in n) printf "%-8s %3d  %5.1f%% of NR=%d\n", k, n[k], 100*n[k]/NR, NR | "sort -k2 -rn" }' commented.txt

# generated by the regression harness
TIMEOUT uart
ASSERT alu
TIMEOUT fifo
# end of section
ASSERT alu
TIMEOUT spi
XCHECK ddr

TIMEOUT    3   37.5% of NR=8
ASSERT     2   25.0% of NR=8
XCHECK     1   12.5% of NR=8

The percentages sum to 75%, because two of the eight lines were comments. Counting the kept records gives the honest figure:

awk '/^#/ { next } { n[$1]++; kept++ } END { for (k in n) printf "%-8s %3d  %5.1f%% of %d\n", k, n[k], 100*n[k]/kept, kept | "sort -k2 -rn" }' commented.txt

TIMEOUT    3   50.0% of 6
ASSERT     2   33.3% of 6
XCHECK     1   16.7% of 6

Now they sum to 100% and the denominator is printed, so a reader can check it. Chapter 5.3's rule, arriving in its most common disguise: percentages that do not sum to 100 are the symptom of a denominator you did not choose.

8. Finding the Maximum, and Hidden Ties

The most frequent key without an external sort:

awk '{ n[$1]++ } END { for (k in n) if (n[k] > best) { best = n[k]; bk = k }; print "most frequent:", bk, best }' errors.txt

most frequent: TIMEOUT 3

Correct, and it has the flaw Chapter 5.1 section 9 warned about in a new form. With two keys tied:

A x
B y

awk '{ n[$1]++ } END { for (k in n) if (n[k] > best) { best = n[k]; bk = k }; print "most frequent:", bk, best }' tie.txt

most frequent: A 1

A is reported as the most frequent, and B is equally frequent. The > comparison keeps whichever tied key the loop reached first — and Chapter 6.1 section 10 established that iteration order is unspecified, so which of the tied keys you get is not even stable across awks.

Counting the ties makes it honest:

awk '{ n[$1]++ } END { for (k in n) { if (n[k] > best) { best = n[k]; bk = k; ties = 1 } else if (n[k] == best) ties++ }; printf "most frequent: %s at %d (%d key(s) tied at that count)\n", bk, best, ties }' tie.txt

most frequent: A at 1 (2 key(s) tied at that count)

And on data with a clear winner it says so:

awk '{ n[$1]++ } END { for (k in n) { if (n[k] > best) { best = n[k]; bk = k; ties = 1 } else if (n[k] == best) ties++ }; printf "most frequent: %s at %d (%d key(s) tied at that count)\n", bk, best, ties }' errors.txt

most frequent: TIMEOUT at 3 (1 key(s) tied at that count)

One key tied, meaning no tie. Reporting "the top signature" from a table where four signatures share the top count is the kind of statement that survives into a slide and then into a decision.

9. A Missing Field Counts as a Category

The fourth way a count misleads, and it is silent.

a 1
b
c 3

awk '{ n[$2]++ } END { for (k in n) printf "key [%s] -> %d\n", k, n[k] | "sort" }' ragged.txt

key [1] -> 1
key [3] -> 1
key [] -> 1

Three categories from three records, and one of them has an empty name. Record b has no $2, so $2 is the empty string, so n[""]++ created a counter under the empty key.

Nothing failed. The distinct-category count is now 3 when the data has 2 real categories, and a report listing "key" values has a blank row.

The guard is an NF check, with the skip counted:

awk 'NF < 2 { skipped++; next } { n[$2]++ } END { for (k in n) printf "key [%s] -> %d\n", k, n[k] | "sort"; close("sort"); printf "(%d record(s) skipped for a missing field)\n", skipped + 0 }' ragged.txt

key [1] -> 1
key [3] -> 1
(1 record(s) skipped for a missing field)

Two real categories and an explicit note about the record that could not be counted — Chapter 5.3's "count what you excluded" habit.

Chapter 8.4 is devoted to missing and empty fields. What matters here is that an empty key is a valid key, so a counting idiom will happily build a category out of absent data.

10. Ordering the Output — and a Trap

Chapter 6.1 established that for (k in a) has no defined order, and the commands above have been piping to sort to impose one. That pipe has a consequence worth knowing before you hit it.

Look again at the guarded command in section 9. It contains close("sort"), and here is what happens without it:

awk 'NF < 2 { skipped++; next } { n[$2]++ } END { for (k in n) printf "key [%s] -> %d\n", k, n[k] | "sort"; printf "(%d record(s) skipped)\n", skipped + 0 }' ragged.txt

(1 record(s) skipped)
key [1] -> 1
key [3] -> 1

The trailer came out first. The rows that were supposed to precede it appear after.

The reason is that | "sort" starts a separate process, and nothing it produces can appear until it has received all its input and been allowed to finish — which, if you never close the pipe, is when awk itself exits. Meanwhile the printf that goes straight to standard output is written immediately. So the direct write wins.

close("sort") ends the pipe's input, waits for sort to flush, and only then continues:

awk 'NF < 2 { skipped++; next } { n[$2]++ } END { for (k in n) printf "key [%s] -> %d\n", k, n[k] | "sort"; close("sort"); printf "(%d record(s) skipped)\n", skipped + 0 }' ragged.txt

key [1] -> 1
key [3] -> 1
(1 record(s) skipped)

11. Common Mistakes

Reporting a record count as a count of problems. Five records can be one bug — section 5.

Using a comparison rather than in for the distinct-pair test. It can create the element it is testing and make the distinct count equal the record count — section 5.

Sorting a triage table by frequency. The noisiest row is usually the least important — section 6.

Dividing by NR after a next. NR counts lines read, so the percentages will not sum to 100 — section 7.

Reporting "the most frequent" from a naive max loop. It hides ties, and which tied key wins is unstable — section 8.

Counting a field that may be missing. The empty string is a valid key and becomes its own category — section 9.

Mixing | "sort" with direct print and no close(). The direct output appears first — section 10.

Omitting the denominator. A count without its total cannot be checked.

12. Exercises

Exercise 1 — Predict the output

awk 'BEGIN { n["a"]++; n["a"]++; n["b"]++; for (k in n) print k, n[k] | "sort" }'

a 2
b 1

Two increments on a, one on b. The first n["a"]++ operated on an element that did not exist, read as 0, and became 1.

Exercise 2 — Count by a chosen key

Count the error records per test name, most frequent first.

TIMEOUT uart
ASSERT alu
TIMEOUT fifo
ASSERT alu
TIMEOUT spi
XCHECK ddr

awk '{ n[$2]++ } END { for (t in n) print n[t], t | "sort -rn" }' errors.txt

2 alu
1 uart
1 spi
1 fifo
1 ddr

alu twice, four tests once each. Note the four single-record tests are tied, so their relative order comes from sort, not from awk.

Exercise 3 — Records against distinct things

The log below has seven records. Report, per severity, both the record count and the number of distinct tests affected.

UVM_WARNING clk_gate_unused env
UVM_WARNING clk_gate_unused env
UVM_WARNING clk_gate_unused env
UVM_WARNING clk_gate_unused env
UVM_WARNING clk_gate_unused env
UVM_ERROR scoreboard_mismatch alu
UVM_FATAL reset_deadlock ddr

awk '{ recs[$1]++; if (!(($1,$3) in seen)) { seen[$1,$3]; tests[$1]++ } } END { for (s in recs) printf "%-12s %d record(s), %d test(s)\n", s, recs[s], tests[s] | "sort" }' sim.log

UVM_ERROR    1 record(s), 1 test(s)
UVM_FATAL    1 record(s), 1 test(s)
UVM_WARNING  5 record(s), 1 test(s)

Five warning records, one affected test. The 5 says the problem is repetitive; the 1 says it is localised.

Exercise 4 — Debug a distinct count that equals the record count

A script reports that a warning affected 5 tests. The log contains one test.

awk '{ recs[$1]++; if (seen[$1,$3] == "") tests[$1]++; junk = seen[$1,$3] } END { for (s in recs) printf "%-12s %d record(s), %d test(s)\n", s, recs[s], tests[s] | "sort" }' sim.log

UVM_ERROR    1 record(s), 1 test(s)
UVM_FATAL    1 record(s), 1 test(s)
UVM_WARNING  5 record(s), 5 test(s)

The seen[$1,$3] == "" test created the element, empty, and nothing ever assigned to it — so the test was true on all five records and the "distinct" counter counted each one.

Notice how the bug presents: the distinct count silently became a second copy of the record count. Two columns that agree exactly are worth being suspicious of, because that is what a broken distinct count looks like.

awk '{ recs[$1]++; if (!(($1,$3) in seen)) { seen[$1,$3]; tests[$1]++ } } END { for (s in recs) printf "%-12s %d record(s), %d test(s)\n", s, recs[s], tests[s] | "sort" }' sim.log

UVM_ERROR    1 record(s), 1 test(s)
UVM_FATAL    1 record(s), 1 test(s)
UVM_WARNING  5 record(s), 1 test(s)

Exercise 5 — Debug percentages that do not add up

A report's shares sum to 75%. Find out why.

# generated by the regression harness
TIMEOUT uart
ASSERT alu
TIMEOUT fifo
# end of section
ASSERT alu
TIMEOUT spi
XCHECK ddr

awk '/^#/ { next } { n[$1]++ } END { for (k in n) printf "%-8s %5.1f%%\n", k, 100*n[k]/NR | "sort -k2 -rn" }' commented.txt

TIMEOUT   37.5%
ASSERT    25.0%
XCHECK    12.5%

NR is 8 — every line awk read, including the two comments the next discarded. The counted records number 6.

awk '/^#/ { next } { n[$1]++; kept++ } END { for (k in n) printf "%-8s %5.1f%% of %d\n", k, 100*n[k]/kept, kept | "sort -k2 -rn" }' commented.txt

TIMEOUT   50.0% of 6
ASSERT    33.3% of 6
XCHECK    16.7% of 6

Summing to 100%, with the denominator stated so the next reader need not re-derive it.

Exercise 6 — Engineering application

Produce a triage table ordered by severity rather than frequency: severity, message id, record count, and distinct tests affected.

awk 'function rank(s) { return s == "UVM_FATAL" ? 1 : s == "UVM_ERROR" ? 2 : 3 } { n[$2]++; sev[$2] = $1; if (!(($2,$3) in seen)) { seen[$2,$3]; t[$2]++ } } END { for (k in n) printf "%d|%-11s %-22s %d rec  %d test(s)\n", rank(sev[k]), sev[k], k, n[k], t[k] | "sort -t\"|\" -k1,1n" }' sim.log

1|UVM_FATAL   reset_deadlock         1 rec  1 test(s)
2|UVM_ERROR   scoreboard_mismatch    1 rec  1 test(s)
3|UVM_WARNING clk_gate_unused        5 rec  1 test(s)

The fatal is first. The leading rank column and | separator exist only to give sort a stable key and would be stripped in a real report — Chapter 7.7 covers doing this without the scaffolding.

The engineering caveat: these are counts of message records, not of defects. One root cause emits many records, and a test that died before printing anything contributes nothing at all — which Chapter 5.6 established is the case most likely to matter and least likely to appear.

Exercise 7 — Challenge: a count that reports its own blind spots

Write a counting summary that states how many records it counted, how many it skipped and why, and whether the top key is tied — so a reader can see what the number does not cover.

awk 'NF < 2 { bad++; next } { n[$1]++; kept++ } END { if (!kept) { print "ERROR: no usable records"; exit 2 } for (k in n) { if (n[k] > best) { best = n[k]; bk = k; ties = 1 } else if (n[k] == best) ties++ } printf "%d records counted, %d skipped (missing field)\n", kept, bad + 0; printf "top key: %s at %d", bk, best; printf (ties > 1 ? " -- TIED with %d other(s)\n" : "\n"), ties - 1 }' errors.txt

6 records counted, 0 skipped (missing field)
top key: TIMEOUT at 3

On the ragged input the same command reports what it could not use:

awk 'NF < 2 { bad++; next } { n[$1]++; kept++ } END { if (!kept) { print "ERROR: no usable records"; exit 2 } for (k in n) { if (n[k] > best) { best = n[k]; bk = k; ties = 1 } else if (n[k] == best) ties++ } printf "%d records counted, %d skipped (missing field)\n", kept, bad + 0; printf "top key: %s at %d", bk, best; printf (ties > 1 ? " -- TIED with %d other(s)\n" : "\n"), ties - 1 }' ragged.txt

2 records counted, 1 skipped (missing field)
top key: a at 1 -- TIED with 1 other(s)

Two counted, one skipped, and the top key honestly flagged as tied — three facts the bare count of 1 would have hidden.

And on empty input it refuses to answer:

: > empty.txt
awk 'NF < 2 { bad++; next } { n[$1]++; kept++ } END { if (!kept) { print "ERROR: no usable records"; exit 2 } for (k in n) { if (n[k] > best) { best = n[k]; bk = k; ties = 1 } else if (n[k] == best) ties++ } printf "%d records counted, %d skipped\n", kept, bad + 0 }' empty.txt
echo "status: $?"

ERROR: no usable records
status: 2

Chapter 5.6's three-way exit status, applied to a count.

13. Summary

• n[key]++ is the whole idiom. No initialisation, no list of expected values, one pass.
• A counter keyed on the data cannot miss a category it has not seen — which is the real reason to prefer it over named counters.
• The key is any string expression, so "count by X" needs only an expression producing X.
• A record count measures how loud a problem is, not how big. Count distinct affected things with a pair key and in.
• Use in for the distinct test. A comparison can create the element and collapse the distinct count into the record count.
• The most frequent key is usually the least important. Noisy problems are benign; severe ones are quiet. Order by severity, then frequency.
• Print the denominator, and after a next count the kept records rather than dividing by NR.
• A naive max loop hides ties, and which tied key wins is unstable because iteration order is unspecified. Count the ties.
• An empty string is a valid key, so a missing field becomes its own category. Guard with NF and report the skips.
• One output path per report, or an explicit close(). Mixing a pipe with direct output reorders it.

You can now count anything in one pass and say what the count covers.

Counting is the special case where the accumulated value is always 1. Chapter 6.3 — Grouping Records generalises it: per-key sums, per-key averages and per-key worst cases, so one pass over an area or timing report produces a table grouped by module or clock — along with the per-key seeding bug that makes a grouped worst case silently empty for exactly the groups that needed it.