Output Ordering

Part II · Module 7 · Chapter 7.7 · Output Ordering

1. Why This Matters

A nightly report is compared against last night's. That is the single most common thing anybody does with one, and it requires that the rows appear in the same order both times.

for (key in array) does not guarantee that. Chapter 6.1 section 10 said so and showed one example. This chapter shows why one example was not enough, and what to do instead.

The practical stake: a report whose row order varies cannot be diffed, so a change in the data becomes indistinguishable from a change in the ordering. On a hundred-row table, every line shows as modified and the one real difference is invisible.

2. The Order Depends on the Key Names

Here is the demonstration. The same program — build eight keys with a numeric suffix, then iterate — run with three different prefixes:

for pre in k key sig_; do printf "%-5s -> " "$pre"; awk -v p="$pre" 'BEGIN { for (i = 1; i <= 8; i++) a[p i] = i; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'; done

k     -> k1 k2 k3 k4 k5 k6 k7 k8
key   -> key3 key4 key5 key6 key7 key8 key1 key2
sig_  -> sig_4 sig_5 sig_6 sig_7 sig_8 sig_1 sig_2 sig_3

Three runs, three orders. Nothing about the program changed; only the text of the keys.

• k1–k8 came out in numeric order, which looks like a guarantee.
• key1–key8 started at key3, ran to key8, then wrapped.
• sig_1–sig_8 started at sig_4.

3. Sort Outside awk

The simplest reliable answer, and the right default: have awk emit rows in any order and let the shell sort them.

TIMEOUT uart
ASSERT alu
TIMEOUT fifo
ASSERT alu
TIMEOUT spi
XCHECK ddr

awk '{ n[$1]++ } END { for (k in n) print n[k], k }' errors.txt | sort -rn

3 TIMEOUT
2 ASSERT
1 XCHECK

One pipe, and sort is a program written for sorting — it handles large inputs, spills to disk, and offers numeric, reverse, field-keyed and stable sorting without any of it being your problem.

This should be the first thing you reach for. The only reasons not to are if awk's output is going somewhere other than a pipe, or if the program emits several differently-ordered sections that cannot share one sort.

Sorting by a chosen column uses sort's own key syntax:

awk '{ n[$1]++ } END { for (k in n) print n[k], k }' errors.txt | sort -k2,2

2 ASSERT
3 TIMEOUT
1 XCHECK

Alphabetical by signature rather than by count.

4. Sort Inside awk

Piping from within awk keeps everything in one command, which matters when the output is redirected to a file or when the program writes several sections:

awk '{ n[$1]++ } END { for (k in n) print n[k], k | "sort -rn" }' errors.txt

3 TIMEOUT
2 ASSERT
1 XCHECK

Same result. The cost is the ordering hazard from Chapter 6.2 section 10: a pipe's output appears only when the pipe is flushed, so anything written directly to stdout can overtake it.

awk '{ n[$1]++ } END { print "=== counts ==="; for (k in n) print n[k], k | "sort -rn" }' errors.txt

=== counts ===
3 TIMEOUT
2 ASSERT
1 XCHECK

That one happens to come out in the intended order because the header was written before the pipe received anything. Putting a direct write after the pipe is where it goes wrong, and close() is the fix:

awk '{ n[$1]++ } END { for (k in n) print n[k], k | "sort -rn"; close("sort -rn"); print "=== end ===" }' errors.txt

3 TIMEOUT
2 ASSERT
1 XCHECK
=== end ===

The close() flushes sort and waits for it before the trailer is written — Chapter 7.3 section 7.

5. An Index Array

Sometimes the order you want is not sorted at all: first appearance. That needs no external process and is fully reproducible.

awk '!($1 in n) { order[++seq] = $1 } { n[$1]++ } END { for (i = 1; i <= seq; i++) printf "%-8s %d\n", order[i], n[order[i]] }' errors.txt

TIMEOUT  3
ASSERT   2
XCHECK   1

The signatures in the order the log first mentioned them, which for a triage is often more useful than alphabetical — it preserves the chronology of the run.

The mechanism: !($1 in n) is true exactly once per signature, on its first record, before n[$1]++ creates the entry. So order[] gets each key once, numbered by ++seq. The END loop then walks order[] by integer index, which does have a defined order.

Two details that matter. The membership test must be !($1 in n) and not n[$1] == 0, because the comparison would create the key and the test would never fire again — Chapter 6.1 section 5. And the rules must be in this order; reversed, n[$1]++ creates the entry first and order[] stays empty, which Chapter 6.1's final exercise demonstrated.

This technique generalises: any array can be given a defined iteration order by keeping a parallel integer-indexed array of its keys. That is also the basis of section 6.

6. Sorting Keys Inside awk

When an external sort is unavailable or the sorted keys are needed during the program rather than on the way out, you sort them yourself:

awk '{ n[$1]++ } END { m = 0; for (k in n) key[++m] = k; for (i = 1; i <= m; i++) for (j = i + 1; j <= m; j++) if (key[j] < key[i]) { t = key[i]; key[i] = key[j]; key[j] = t } for (i = 1; i <= m; i++) printf "%-8s %d\n", key[i], n[key[i]] }' errors.txt

ASSERT   2
TIMEOUT  3
XCHECK   1

Alphabetical, computed entirely inside awk. Three steps: copy the keys into an integer-indexed array, sort that array with a selection sort, then walk it by position.

The comparison key[j] < key[i] is a string comparison here, because the keys are non-numeric strings. On numeric keys it would be numeric, per Chapter 4.2's rule — which is a trap if the keys are numeric strings and you wanted lexical order, or vice versa. Force the one you mean: key[j] + 0 < key[i] + 0 for numeric, key[j] "" < key[i] "" for lexical.

The sort is a selection sort — it compares every pair, so it is quadratic. Fine for the few hundred distinct keys a report usually has, and wrong for tens of thousands. Chapter 6.4 section 6 made the same point about computing a median: when the data is large, sort is the right tool and awk should hand the job over.

7. The gawk Features, and Why They Are Not Used Here

gawk offers asort(), asorti() and PROCINFO["sorted_in"], and they are the natural answer — on gawk.

None of them is available on this awk:

awk 'BEGIN { a[1] = "b"; a[2] = "a"; n = asort(a); print n }' 2>&1 | head -1

awk: calling undefined function asort

awk 'BEGIN { a[1] = "b"; n = asorti(a); print n }' 2>&1 | head -1

awk: calling undefined function asorti

A fatal error, not a fallback. A script using them runs on a workstation with gawk and dies on a compute farm with a POSIX awk — the portability failure Chapter 8.6 is about.

PROCINFO is worse, because it fails silently:

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; a["b"] = 1; a["a"] = 2; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'

a b

That looks like it worked — the keys came out sorted. It did not. With two keys there are only two possible orders and this awk happened to produce the sorted one anyway.

The honest test uses keys whose hash order is known to differ from sorted order, which section 2 supplied:

awk 'BEGIN { for (i = 1; i <= 8; i++) a["key" i] = i; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'

key3 key4 key5 key6 key7 key8 key1 key2

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; for (i = 1; i <= 8; i++) a["key" i] = i; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'

key3 key4 key5 key6 key7 key8 key1 key2

Identical. PROCINFO had no effect whatsoever.

What it actually did is create an ordinary array:

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; n = 0; for (k in PROCINFO) n++; printf "PROCINFO has %d element(s):\n", n; for (k in PROCINFO) printf "  [%s] = [%s]\n", k, PROCINFO[k] }'

PROCINFO has 1 element(s):
  [sorted_in] = [@ind_str_asc]

One element — the one that assignment created. On a POSIX awk, PROCINFO is not a special variable at all; it is a name like any other, and writing to it is as meaningful as writing to MYARRAY.

8. Choosing

The default is the first row. Most awk reports are one command in a pipeline, and | sort is shorter, faster and more capable than anything you will write inside the program.

9. Common Mistakes

Assuming for (k in a) is sorted because a test with short key names came out sorted — section 2.

Assuming it is insertion order. It is neither.

Inferring a rule from one key set. Three prefixes give three orders.

Using PROCINFO["sorted_in"] on a non-gawk awk. Silently inert — section 7.

Testing a feature on an input where broken and working look alike. Two keys cannot distinguish sorted from unsorted.

Using asort() in a portable script. A fatal error where gawk is absent.

Mixing a pipe to sort with direct output and no close(). The direct output can overtake it.

Testing n[$1] == 0 to detect a first appearance. It creates the key and never fires again.

Putting the index-array rule after the counting rule. The membership test is then never true.

Hand-rolling a sort for tens of thousands of keys. It is quadratic; hand the job to sort.

10. Exercises

Exercise 1 — Predict whether you can predict it

awk 'BEGIN { a["z"] = 1; a["y"] = 2; a["x"] = 3; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'

x y z

Sorted — and that is a coincidence of these three keys, not a rule. Section 2's point: a single observation of order is evidence of nothing.

Exercise 2 — Make the order change

Run one program with three key prefixes and compare.

for pre in k key sig_; do printf "%-5s -> " "$pre"; awk -v p="$pre" 'BEGIN { for (i = 1; i <= 8; i++) a[p i] = i; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'; done

k     -> k1 k2 k3 k4 k5 k6 k7 k8
key   -> key3 key4 key5 key6 key7 key8 key1 key2
sig_  -> sig_4 sig_5 sig_6 sig_7 sig_8 sig_1 sig_2 sig_3

Three orders. The first would have convinced you the order was numeric.

Exercise 3 — Sort outside awk

Report error counts, most frequent first.

TIMEOUT uart
ASSERT alu
TIMEOUT fifo
ASSERT alu
TIMEOUT spi
XCHECK ddr

awk '{ n[$1]++ } END { for (k in n) print n[k], k }' errors.txt | sort -rn

3 TIMEOUT
2 ASSERT
1 XCHECK

Exercise 4 — First-appearance order

Report the same counts in the order the signatures first appeared, without any external command.

awk '!($1 in n) { order[++seq] = $1 } { n[$1]++ } END { for (i = 1; i <= seq; i++) printf "%-8s %d\n", order[i], n[order[i]] }' errors.txt

TIMEOUT  3
ASSERT   2
XCHECK   1

Reproducible for a given input on any awk, because the END loop walks integer indices.

Exercise 5 — Debug a silently-inert feature

A colleague reports their sorted output is not sorted. Their command and the honest test:

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; for (i = 1; i <= 8; i++) a["key" i] = i; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'

key3 key4 key5 key6 key7 key8 key1 key2

Unsorted, despite the assignment. PROCINFO is a gawk variable; on this awk it is an ordinary array name.

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; n = 0; for (k in PROCINFO) n++; print "PROCINFO element count:", n }'

PROCINFO element count: 1

One element — the assignment created it and nothing reads it.

The reason this is worth an exercise is that the obvious test hides it:

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; a["b"] = 1; a["a"] = 2; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'

a b

Two keys, sorted output, feature apparently working. Choose a test input on which a broken mechanism would visibly fail.

Exercise 6 — Engineering application

Produce a triage table sorted by severity then frequency, with a trailer that must appear after the rows.

UVM_WARNING clk_gate_unused env
UVM_WARNING clk_gate_unused env
UVM_ERROR scoreboard_mismatch alu
UVM_FATAL reset_deadlock ddr

awk 'function rank(s) { return s == "UVM_FATAL" ? 1 : s == "UVM_ERROR" ? 2 : 3 } { n[$2]++; sev[$2] = $1 } END { for (k in n) printf "%d|%-11s %-22s %d\n", rank(sev[k]), sev[k], k, n[k] | "sort -t\"|\" -k1,1n"; close("sort -t\"|\" -k1,1n"); c = 0; for (k in n) c++; printf "(%d distinct signature(s) over %d record(s))\n", c, NR }' sim.log

1|UVM_FATAL   reset_deadlock         1
2|UVM_ERROR   scoreboard_mismatch    1
3|UVM_WARNING clk_gate_unused        2
(3 distinct signature(s) over 4 record(s))

Three mechanisms cooperating. The rank() function supplies a leading sort key so severity beats frequency — Chapter 6.2 section 6's argument that the noisiest row is rarely the most important. The pipe to sort -t"|" -k1,1n orders on that key numerically. And the close() is what puts the trailer after the rows; without it the direct printf would overtake the pipe, per section 4.

The signature count is computed with a loop rather than length(n). On this awk length() on an array does work — Chapter 6.1 section 12 measured it — but it is a gawk extension and not POSIX, so a counting loop is the portable choice for a script that leaves your machine. Two lines instead of one, and it runs anywhere.

In a real report the 1|, 2|, 3| prefix would be stripped, since it exists only to give sort something to order by. Keeping it here makes the ordering visible, which is the point of the exercise.

Exercise 7 — Challenge: prove a report is diff-stable

Write a check that runs a report twice and confirms the row order is identical, so a diff between two nights is meaningful.

awk '{ n[$1]++ } END { for (k in n) print n[k], k }' errors.txt > r1.txt
awk '{ n[$1]++ } END { for (k in n) print n[k], k }' errors.txt > r2.txt
if cmp -s r1.txt r2.txt; then echo "stable across runs"; else echo "ORDER VARIED"; fi

stable across runs

Stable — because the same awk on the same input produces the same hash order every time. That is worth knowing and is not the guarantee you need: it says nothing about a different awk, a different key set, or the same keys with one added.

The guarantee comes from imposing the order:

awk '{ n[$1]++ } END { for (k in n) print n[k], k }' errors.txt | sort -rn > s1.txt
awk '{ n[$1]++ } END { for (k in n) print n[k], k }' errors.txt | sort -rn > s2.txt
if cmp -s s1.txt s2.txt; then echo "stable, and stable by construction"; else echo "ORDER VARIED"; fi

stable, and stable by construction

Both pass, and only the second is stable for a reason rather than by accident. A property that happens to hold is not a property you can rely on — the distinction this chapter exists to make, and the same one section 7's PROCINFO test turned on.

11. Summary

• for (key in array) has no defined order, and the order differs for different key names in the same program.
• A test that comes out sorted proves nothing. k1–k8 iterate in numeric order and key1–key8 do not.
• Pipe through sort in the shell by default — shortest, fastest, most capable.
• Pipe to sort inside awk when output goes elsewhere, and close() it before writing anything directly.
• An index array gives first-appearance order, reproducibly and with no external process.
• Keep a parallel integer-indexed array to give any array a defined iteration order.
• A hand-rolled sort is quadratic — fine for hundreds of keys, wrong for thousands.
• Force the comparison you mean when sorting keys: + 0 for numeric, "" for lexical.
• asort() and asorti() are gawk-only and fail loudly.
• PROCINFO["sorted_in"] is gawk-only and fails silently — on a POSIX awk it just creates an array element.
• Test a mechanism on an input where a broken one would visibly fail.

Ordering is the last property of a single report. Chapter 7.8 — Merging Multiple Reports closes the module by combining two of them on a shared key — a timing report and a coverage report into one table — and by deciding deliberately what to print for a key that appears in only one of the two, which is where most merges quietly lose rows.