Summary Calculations

Part II · Module 6 · Chapter 6.4 · Summary Calculations

1. Why This Matters

A summary is where a script stops reporting data and starts making a claim.

4870 is data. mean 1623 cells per instance is a claim, and it can be wrong in ways the raw number cannot — by dividing by the wrong denominator, by weighting things that should not be weighted equally, or by summarising an empty file into a confident zero.

The mechanics are short, because Chapter 5.1 built the accumulators and Chapter 6.3 grouped them. Sections 2 to 4 finish that job.

Section 5 is the reason this chapter is rated as it is. Averaging percentages is the default instinct, it looks obviously reasonable, and on real coverage data it produces a number that is wrong by tens of points in the optimistic direction.

2. The Five Numbers in One Pass

Total, count, mean, minimum and maximum, from a single read:

alu,95,100
fifo,72,100
uart,88,100
spi,45,100

awk -F, '{ s += $2; n++; if (n == 1 || $2 + 0 < mn) mn = $2 + 0; if (n == 1 || $2 + 0 > mx) mx = $2 + 0 } END { if (!n) { print "ERROR: no records"; exit 2 } printf "n=%d  total=%d  mean=%.2f  min=%d  max=%d\n", n, s, s/n, mn, mx }' cov.csv

n=4  total=300  mean=75.00  min=45  max=95

Everything is accumulated during the rule phase and nothing is computed until END, which is Chapter 2.4's three-phase model doing exactly what it is for.

3. The Empty-Input Guard

Chapter 5.3 established that division by zero in awk is fatal. A summary over an empty file divides by zero, so the guard is not politeness — it is the difference between a message and a crash.

Without it:

awk -F, '{ s += $2; n++ } END { printf "mean=%.2f\n", s/n }' empty.csv

awk: division by zero
 source line number 1

Fatal, exit status 2, and no output at all. The command produced no summary and no explanation that a human would read.

With the guard:

awk -F, '{ s += $2; n++ } END { if (!n) { print "ERROR: no records"; exit 2 } printf "mean=%.2f\n", s/n }' empty.csv
echo "rc=$?"

ERROR: no records
rc=2

Same exit status, and now a sentence saying what went wrong. Chapter 5.6's point about status codes applies: both versions exit 2, but only one tells the person reading the log why.

4. Minimum and Maximum Need Seeding

The n == 1 || in section 2's command is Chapter 5.1 section 9's lesson, in its scalar form. Without it:

path_a,-0.124
path_b,-0.081
path_c,-1.200

awk -F, '{ if ($2 + 0 > mx) mx = $2 + 0 } END { printf "max=[%s]\n", mx }' neg.csv

max=[]

Blank, because mx started at an implicit 0 and no value exceeded it.

awk -F, '{ n++; if (n == 1 || $2 + 0 > mx) mx = $2 + 0 } END { if (!n) { print "no records"; exit 2 } printf "max=%s over %d values\n", mx, n }' neg.csv

max=-0.081 over 3 values

n == 1 makes the first record seed the tracker from real data. Chapter 6.3 section 6 showed the grouped version of the same bug, where the test is !(key in array) instead — the same idea, once per key rather than once per file.

The rule in one line: never let an extremum start from a value that is not in the data.

5. Averaging Percentages Is Usually Wrong

Here is the chapter's main point, and it is worth the whole module.

A coverage report with three blocks of very different sizes:

blockA,180,200
blockB,5,500
blockC,95,100

Their individual percentages:

awk -F, '{ printf "%-8s %3d of %3d = %5.1f%%\n", $1, $2, $3, 100*$2/$3 }' bins.csv

blockA   180 of 200 =  90.0%
blockB     5 of 500 =   1.0%
blockC    95 of 100 =  95.0%

Now summarise, the obvious way — average the three percentages:

awk -F, '{ p = 100*$2/$3; s += p; n++ } END { printf "mean of percentages: %.2f%% over %d blocks\n", s/n, n }' bins.csv

mean of percentages: 62.00% over 3 blocks

And the other way — add up the bins, then divide once:

awk -F, '{ hit += $2; tot += $3 } END { printf "weighted coverage: %.2f%% (%d of %d bins)\n", 100*hit/tot, hit, tot }' bins.csv

weighted coverage: 35.00% (280 of 800 bins)

62% and 35%. Same three rows, same arithmetic rules, a twenty-seven point gap.

The two numbers can be printed side by side, which is the honest way to show a reviewer that the distinction is real:

awk -F, '{ hit += $2; tot += $3; p += 100*$2/$3; n++ } END { printf "weighted: %.2f%% (%d of %d bins)\nunweighted mean of %d block percentages: %.2f%%\n", 100*hit/tot, hit, tot, n, p/n }' bins.csv

weighted: 35.00% (280 of 800 bins)
unweighted mean of 3 block percentages: 62.00%

Labelling the second one as an unweighted mean of block percentages is what makes it harmless. A number is dangerous when its name does not say what it measured.

6. The Median, and What awk Does Not Have

A mean is distorted by an outlier; a median is not. awk will compute one, and the work involved says something about the language.

There is no sort function in POSIX awk. To get a median you store the values and sort them yourself:

awk -F, '{ v[n++] = $2 + 0 } END { if (!n) { print "no records"; exit 2 } for (i = 0; i < n; i++) for (j = i + 1; j < n; j++) if (v[j] < v[i]) { t = v[i]; v[i] = v[j]; v[j] = t } printf "median of %d values: %s\n", n, (n % 2 ? v[(n-1)/2] : (v[n/2-1] + v[n/2]) / 2) }' cov.csv

median of 4 values: 80

Correct — the sorted values are 45, 72, 88, 95, and the median of an even count is the mean of the middle two, (72 + 88) / 2 = 80. Compare the mean of 75.00 from section 2: the low outlier at 45 pulls the mean below the median.

That nested loop is a selection sort, which compares every pair — fine for a few hundred values and quadratic beyond that. On a million-row timing report it is the wrong tool, and the right answer is to let sort do it:

awk -F, '{ print $2 }' cov.csv | sort -n | awk '{ v[n++] = $1 } END { printf "median of %d values: %s\n", n, (n % 2 ? v[(n-1)/2] : (v[n/2-1] + v[n/2]) / 2) }'

median of 4 values: 80

Same answer, and the sorting is done by a program written for it.

This is the first point in the track where awk's limits are the honest answer rather than a workaround. awk has no sort, no standard-deviation function, no percentiles and no date handling. When a summary needs those, the choice is to write them by hand, to shell out, or to use Python — which Chapter 12.5 treats as a legitimate design decision rather than a defeat.

7. Stating What the Number Covers

Every summary in this chapter prints its denominator, and this is the section that says why in general terms.

awk -F, '{ s += $2; n++ } END { if (!n) { print "ERROR: no records"; exit 2 } printf "mean %.2f over %d blocks (total %d)\n", s/n, n, s }' cov.csv

mean 75.00 over 4 blocks (total 300)

Three numbers, and any two of them let a reader reconstruct the third. That is the property that makes a figure auditable.

The engineering caveats this track has been accumulating all apply here at once, and they are worth collecting:

• mean 75.00 is the mean of the rows in this file, which is the mean over the design only if the file lists every block exactly once (Chapter 1.2).
• It is a mean of covered-bin counts, not of coverage percentages, and section 5 showed those differ.
• A block that produced no row contributes nothing and is invisible — Chapter 5.6's point about records that never arrived.
• 4 blocks is a count of rows, not a count of blocks in the design.

None of those is fixed by better arithmetic. They are fixed by naming the quantity precisely, which costs a few words in a format string:

awk -F, '{ s += $2; n++ } END { if (!n) { print "ERROR: no records"; exit 2 } printf "mean covered bins per listed block: %.2f (%d bins over %d rows)\n", s/n, s, n }' cov.csv

mean covered bins per listed block: 75.00 (300 bins over 4 rows)

Longer, and impossible to misquote.

8. Common Mistakes

No empty-input guard. Division by zero is fatal and produces no summary at all — section 3.

Guarding by printing 0 and exiting 0. A missing file then reads as terrible data — section 3.

An extremum seeded from an implicit 0. Blank output whenever every value is on one side of zero — section 4.

Averaging percentages. Systematically optimistic, and most wrong exactly when a large block is poorly covered — section 5.

Printing a percentage without its two counts. The reader cannot check it and cannot detect section 5's error.

Calling an unweighted mean of percentages a coverage figure. Label it, or do not print it.

Using a hand-rolled sort on a large file. It is quadratic; pipe to sort.

Assuming awk has statistics functions. No sort, no percentiles, no standard deviation.

Describing a row count as a count of design objects. They differ whenever the report is partial.

9. Exercises

Exercise 1 — Predict the output

awk 'BEGIN { n = 0; printf "%s\n", (n ? "has records" : "empty") }'

empty

n is 0, which is false (Chapter 5.1). This is the guard condition in its smallest form.

Exercise 2 — The five numbers

Report count, total, mean, minimum and maximum of the covered-bin column.

alu,95,100
fifo,72,100
uart,88,100
spi,45,100

awk -F, '{ s += $2; n++; if (n == 1 || $2 + 0 < mn) mn = $2 + 0; if (n == 1 || $2 + 0 > mx) mx = $2 + 0 } END { if (!n) { print "ERROR: no records"; exit 2 } printf "n=%d total=%d mean=%.2f min=%d max=%d\n", n, s, s/n, mn, mx }' cov.csv

n=4 total=300 mean=75.00 min=45 max=95

Exercise 3 — Debug a fatal summary

A nightly job's summary step produces no output and a non-zero status. The coverage file was not written that night.

awk -F, '{ s += $2; n++ } END { printf "mean=%.2f\n", s/n }' empty.csv 2>&1
echo "rc=$?"

awk: division by zero
 source line number 1
rc=2

n is 0 because no record was read, and s/n is a fatal division by zero — Chapter 5.3 section 7. The run aborts before printing anything.

awk -F, '{ s += $2; n++ } END { if (!n) { print "ERROR: coverage file is empty or missing"; exit 2 } printf "mean=%.2f\n", s/n }' empty.csv
echo "rc=$?"

ERROR: coverage file is empty or missing
rc=2

Same status, and now the log says which of several possible failures it was.

Exercise 4 — Debug a blank maximum

A slack summary reports an empty maximum for a report full of numbers.

path_a,-0.124
path_b,-0.081
path_c,-1.200

awk -F, '{ if ($2 + 0 > mx) mx = $2 + 0 } END { printf "max=[%s]\n", mx }' neg.csv

max=[]

Every slack is negative and mx begins as an implicit 0, so the comparison is never true.

awk -F, '{ n++; if (n == 1 || $2 + 0 > mx) mx = $2 + 0 } END { if (!n) { print "no records"; exit 2 } printf "max=%s over %d values\n", mx, n }' neg.csv

max=-0.081 over 3 values

Exercise 5 — The two coverage figures

Compute both summaries of the bin data in one pass, and decide which to report.

blockA,180,200
blockB,5,500
blockC,95,100

awk -F, '{ hit += $2; tot += $3; p += 100*$2/$3; n++ } END { printf "weighted:   %.2f%% (%d of %d bins)\n", 100*hit/tot, hit, tot; printf "unweighted: %.2f%% (mean of %d block percentages)\n", p/n, n }' bins.csv

weighted:   35.00% (280 of 800 bins)
unweighted: 62.00% (mean of 3 block percentages)

Both figures from a single read, each labelled with what it measured.

Report the weighted figure. It is the fraction of the design's bins that are covered, which is the question. The unweighted mean is a statement about three percentages and gives blockB — 500 of the 800 bins, at 1% — the same vote as blockC's 100 bins at 95%.

Print the unweighted number only alongside its label. 62.00% under a heading that says "coverage" is a misreport; unweighted mean of 3 block percentages: 62.00% is a fact about the block list and cannot mislead anyone who reads it.

Checking the accumulators directly is worth doing once, so the two formulas are visibly built from the same four values:

awk -F, '{ hit += $2; tot += $3; p += 100*$2/$3; n++ } END { printf "hit=%d tot=%d p=%.4f n=%d  ->  weighted %.2f  unweighted %.2f\n", hit, tot, p, n, 100*hit/tot, p/n }' bins.csv

hit=280 tot=800 p=186.0000 n=3  ->  weighted 35.00  unweighted 62.00

280/800 against 186/3. Neither is an arithmetic error — they are two different quotients, and choosing between them is the whole decision.

Exercise 6 — Engineering application

Produce a coverage summary a reviewer can audit: the weighted figure with both counts, the worst block named, and a non-zero status if any block falls below 50%.

awk -F, '{ hit += $2; tot += $3; n++; pc = 100*$2/$3; if (n == 1 || pc < wpc) { wpc = pc; wb = $1 }; if (pc < 50) low++ } END { if (!tot) { print "ERROR: no bins"; exit 2 } printf "coverage %.2f%% (%d of %d bins over %d blocks)\n", 100*hit/tot, hit, tot, n; printf "worst block: %s at %.1f%%\n", wb, wpc; if (low) { printf "%d block(s) below 50%%\n", low; exit 1 } }' bins.csv
echo "status: $?"

coverage 35.00% (280 of 800 bins over 3 blocks)
worst block: blockB at 1.0%
1 block(s) below 50%
status: 1

Everything a reviewer needs: the weighted figure, both counts, the block count, the worst offender named, and a status a pipeline can branch on. blockB at 1% is the finding — and it is also the block that averaging would have buried.

On data where every block passes, the same command exits 0:

awk -F, '{ hit += $2; tot += $3; n++; pc = 100*$2/$3; if (n == 1 || pc < wpc) { wpc = pc; wb = $1 }; if (pc < 50) low++ } END { if (!tot) { print "ERROR: no bins"; exit 2 } printf "coverage %.2f%% (%d of %d bins over %d blocks)\n", 100*hit/tot, hit, tot, n; printf "worst block: %s at %.1f%%\n", wb, wpc; if (low) { printf "%d block(s) below 50%%\n", low; exit 1 } }' cov.csv
echo "status: $?"

coverage 75.00% (300 of 400 bins over 4 blocks)
worst block: spi at 45.0%
1 block(s) below 50%
status: 1

spi at 45% trips the threshold, so this file exits 1 too — which is the correct behaviour and a useful reminder that a 75% headline can still contain a block below the bar. The headline figure and the per-block minimum are different facts and a gate needs both.

Exercise 7 — Challenge: prove the weighting matters

Write a command that reports both figures and the gap between them, so a reviewer can see at a glance whether the weighting choice affects the conclusion.

awk -F, '{ hit += $2; tot += $3; p += 100*$2/$3; n++ } END { if (!tot) { print "ERROR: no bins"; exit 2 } w = 100*hit/tot; u = p/n; d = w - u; if (d < 0) d = -d; printf "weighted %.2f%%  unweighted %.2f%%  gap %.2f points\n", w, u, d; if (d > 5) print "WARNING: the weighting choice changes the answer materially" }' bins.csv

weighted 35.00%  unweighted 62.00%  gap 27.00 points
WARNING: the weighting choice changes the answer materially

On data where the blocks are the same size, the two agree and the warning stays silent:

awk -F, '{ hit += $2; tot += $3; p += 100*$2/$3; n++ } END { if (!tot) { print "ERROR: no bins"; exit 2 } w = 100*hit/tot; u = p/n; d = w - u; if (d < 0) d = -d; printf "weighted %.2f%%  unweighted %.2f%%  gap %.2f points\n", w, u, d; if (d > 5) print "WARNING: the weighting choice changes the answer materially" }' cov.csv

weighted 75.00%  unweighted 75.00%  gap 0.00 points

Identical, because every block in cov.csv has 100 bins — and that is the condition under which averaging percentages is safe. Equal denominators make the weighted and unweighted figures the same number.

Which is exactly why the mistake persists: it is harmless on uniform test data and badly wrong on real data, where block sizes vary by orders of magnitude. The two commands above are the same command, and the only thing that changed was whether the denominators were equal.

10. Summary

• Accumulate in the rule phase, compute in END. Chapter 2.4's phases, used as designed.
• Guard the empty input before dividing. Division by zero is fatal and leaves no summary.
• Name the condition and exit non-zero — an empty input usually means something upstream failed, which matters more than any number.
• Never print 0 and exit 0 for an empty file. It reads as real, terrible data.
• Seed a minimum or maximum from the first record, never from an implicit 0.
• Averaging percentages is usually wrong. Aggregate the counts and divide once — 35% against 62% on the same three rows.
• The error is systematically optimistic, and worst exactly when a large block is poorly covered.
• Equal denominators make the two agree, which is why the mistake survives testing on uniform data.
• Print both counts with every percentage. A figure a reader cannot check is a figure they must trust.
• awk has no sort, no percentiles, no standard deviation. Hand-roll it, pipe to sort, or use Python.
• Name the quantity precisely. mean covered bins per listed block cannot be misquoted; mean 75.00 can.

You can now produce the numbers a report carries, and defend how each was computed.

Every command in this module has been a single line, and the one in Exercise 6 is pushing what a single line should be asked to hold. Chapter 6.5 — User-Defined Functions is where a script stops being a one-liner: how to define a function, and awk's genuinely unusual rules — arrays pass by reference while scalars pass by value, there is no local scope at all, and the only locals you get are extra parameters you declare and deliberately never pass.