Redirecting Output

Part II · Module 7 · Chapter 7.3 · Redirecting Output

1. Why This Matters

A regression produces one merged log and somebody wants one file per test. A metrics script produces rows that must arrive sorted. A summary needs its errors on stderr so that $(…) in a shell script captures only the answer.

All three are redirection, all three are one clause of awk, and the first one is where the surprise lives.

2. Writing to a File

print > "file" sends output to a file instead of standard output:

alu PASS 120
fifo FAIL 95
uart PASS 88
alu FAIL 60

awk '{ print $1 > "names.out" }' merged.log
cat names.out

alu
fifo
uart
alu

Four lines. That is the fact to hold on to, because the shell's > would have left one.

In the shell, > truncates every time it is evaluated. In awk, > truncates when the file is first opened and appends for the rest of the run. The file is opened once, on the first write, and stays open.

awk '{ print $1 > "names.out" }' merged.log
wc -l < names.out

       4

Four again, not eight — the new run truncated the file once at its own first write.

3. The File Stays Open for the Whole Run

Because the file is opened once, a header, the records and a footer all land in one file in order:

awk 'BEGIN { print "== names ==" > "hdr.out" } { print $1 > "hdr.out" } END { print "== end ==" > "hdr.out" }' merged.log
cat hdr.out

== names ==
alu
fifo
uart
alu
== end ==

Three separate rules, three > clauses naming the same file, one open. Chapter 2.4's phase model and this chapter's open-once rule combine to make that work without any bookkeeping.

4. A Destination Computed per Record

The destination is an expression, which is what makes the log splitter a one-liner:

awk '{ print > ($1 ".out") }' merged.log
for f in alu.out fifo.out uart.out; do echo "--- $f ---"; cat "$f"; done

--- alu.out ---
alu PASS 120
alu FAIL 60
--- fifo.out ---
fifo FAIL 95
--- uart.out ---
uart PASS 88

One pass, three files, and alu.out holds both of its records — which only works because of section 2's open-once rule. Under shell semantics the second alu write would have truncated the first.

The parentheses are mandatory. Without them it is a syntax error:

awk '{ print $1 > $1 ".out3" }' merged.log 2>&1 | head -2

awk: syntax error at source line 1
 context is

awk parses > greedily as the redirection operator and then cannot make sense of the concatenation that follows. Parenthesise any computed destination — and while you are at it, parenthesise even a simple one, since it costs nothing and removes the question.

5. Appending with >>

>> never truncates, so it accumulates across runs:

rm -f acc.out
awk '{ print $1 >> "acc.out" }' merged.log
awk '{ print $1 >> "acc.out" }' merged.log
wc -l < acc.out

       8

Eight lines from two runs of a four-record file.

That makes >> right for a long-lived log and wrong for a report that should reflect only this run — where a stale >> file silently mixes last night's rows with tonight's. The choice is the same one as in the shell; only >'s meaning differs.

6. Piping to a Command

print | "command" sends output to a process:

awk '{ print $3, $1 | "sort -rn" }' merged.log

120 alu
95 fifo
88 uart
60 alu

Sorted by runtime, descending. The command is a string, so it can be built at run time, and everything written to the same string goes to the same process.

Chapter 6.2 section 10 covered the ordering trap this creates — a pipe's output appears only when the pipe is flushed, so mixing it with direct print reorders the report. The fix there was close(), which is the next section.

7. close() and the Truncation Trap

close() ends a destination. On a pipe that is exactly what you want — it flushes the process and waits for it, so the output arrives before anything written afterwards:

awk 'BEGIN { print "b" | "sort"; print "a" | "sort"; close("sort"); print "-- second batch --"; print "d" | "sort"; print "c" | "sort" }'

a
b
-- second batch --
c
d

Two separate sorted batches. After the close(), the same command string starts a fresh process, which is how you sort groups independently in one program.

On a file opened with >, close() does something most people do not expect:

rm -f t.out
awk '{ print $1 > "t.out"; close("t.out") }' merged.log
cat t.out

alu

One line. Three records are gone.

The safe form of the same tidying, using >>:

rm -f app.out
awk '{ print $1 >> "app.out"; close("app.out") }' merged.log
cat app.out

alu
fifo
uart
alu

All four records, because >> has nothing to truncate.

8. The Open-File Limit

Leaving files open is normally right, and it has a ceiling. awk keeps its own table of open destinations, and it is much smaller than the operating system's limit:

ulimit -n

1048576

A million descriptors available, and yet awk stops at about ten thousand output files:

rm -rf lim && mkdir lim
awk 'BEGIN { for (i = 1; i <= 12000; i++) print i > ("lim/f" i ".out") }' 2>&1 | head -2

awk: can't open file lim/f10238.out
 source line number 1

ls lim | wc -l

   10237

10,237 files created, then a fatal error at the 10,238th. The exit status is 2 and the run stops — so a splitter that works on a 500-test regression can die on a 20,000-instance report, with the first ten thousand files written and the rest missing.

The limit is on simultaneously open destinations, not on files written in total. close() releases a descriptor, which section 7 proved from the other direction: a closed > file re-truncates on the next write precisely because it really was closed and reopened.

So a program that must write more destinations than the table holds has to close as it goes — and, per section 7, that means >> rather than >:

rm -rf many && mkdir many
awk 'BEGIN { for (i = 1; i <= 3; i++) { f = "many/f" i ".out"; print "row for " i >> f; close(f) } }'
for f in many/f1.out many/f2.out many/f3.out; do printf "%s: " "$f"; cat "$f"; done

many/f1.out: row for 1
many/f2.out: row for 2
many/f3.out: row for 3

Three files, each written then closed, so at most one descriptor is held at a time. The same loop scales to any number of files.

In practice the better answer is usually to avoid thousands of output files. One delimited file with a key column, sorted afterwards, is easier to consume, easier to clean up, and does not hit any limit — which is Chapter 7.6's argument.

9. Writing to stderr

"/dev/stderr" is a destination like any other, and it is how a script separates its answer from its complaints:

awk '{ if ($2 == "FAIL") print "FAILED:", $1 > "/dev/stderr"; else print "ok:", $1 }' merged.log 2>/dev/null

ok: alu
ok: uart

awk '{ if ($2 == "FAIL") print "FAILED:", $1 > "/dev/stderr" }' merged.log 2>&1 >/dev/null

FAILED: fifo
FAILED: alu

The first command discarded stderr and shows only the passes; the second discarded stdout and shows only the failures. Two streams, separable by the caller.

This matters for the Chapter 5.6 pattern where a shell script captures awk's output:

count=$(awk '{ if ($2 == "FAIL") { n++; print "note: " $1 " failed" > "/dev/stderr" } } END { print n + 0 }' merged.log 2>/dev/null)
echo "captured: [$count]"

captured: [2]

$count is the number and nothing else. Had the notes gone to stdout, $count would have held three lines of mixed text and the arithmetic downstream would have failed — the usual reason a shell script reports an empty or malformed value.

/dev/stderr is widely supported and is not POSIX awk. Chapter 8.6 covers the portability question; the portable alternative is to let the shell do the splitting.

10. Common Mistakes

Expecting > to truncate per write. It truncates once, on first open — section 2.

Adding close() to a per-record > write. Every record then truncates; only the last survives — section 7.

Forgetting parentheses on a computed destination. A syntax error — section 4.

Using >> for a per-run report. Stale rows from previous runs accumulate silently.

Opening thousands of destinations. A fatal error at about ten thousand — section 8.

Mixing a pipe with direct output and no close(). The direct output appears first — Chapter 6.2 section 10.

Sending diagnostics to stdout. A shell $(…) capture then contains them — section 9.

Assuming awk's > inside a program is the shell's >. The shell never sees it; it is awk's own operator.

11. Exercises

Exercise 1 — Predict the line count

awk '{ print $1 > "e1.out" }' merged.log
wc -l < e1.out

       4

Four, not one. > truncated at the first write and appended for the rest.

Exercise 2 — Split a log

Write each record to a file named after its test.

alu PASS 120
fifo FAIL 95
uart PASS 88
alu FAIL 60

awk '{ print > ($1 ".split") }' merged.log
for f in alu.split fifo.split uart.split; do echo "--- $f ---"; cat "$f"; done

--- alu.split ---
alu PASS 120
alu FAIL 60
--- fifo.split ---
fifo FAIL 95
--- uart.split ---
uart PASS 88

Exercise 3 — Debug a splitter that keeps one record

A colleague's splitter produces the right files, each containing a single record.

rm -f d3.out
awk '{ print $1 > "d3.out"; close("d3.out") }' merged.log
cat d3.out

alu

The close() makes each write a first open, and > truncates on every first open.

Two repairs:

rm -f d3.out
awk '{ print $1 > "d3.out" }' merged.log
cat d3.out

alu
fifo
uart
alu

rm -f d3b.out
awk '{ print $1 >> "d3b.out"; close("d3b.out") }' merged.log
cat d3b.out

alu
fifo
uart
alu

Drop the close(), or keep it and use >>. The first is right for a handful of files; the second is required past the section 8 limit.

Exercise 4 — A pipe, closed and reused

Predict the output before running.

awk 'BEGIN { print "b" | "sort"; print "a" | "sort"; close("sort"); print "d" | "sort"; print "c" | "sort" }'

a
b
c
d

Two independent sorts, not one sort of four values — the close() ended the first process and the next write started a new one. Without the close() all four would have gone through one sort and still come out in this order, which is why this example uses values that make the two batches visible as a b then c d rather than interleaved.

Exercise 5 — Separate the answer from the notes

Produce a count on stdout and per-failure notes on stderr, then capture only the count.

count=$(awk '{ if ($2 == "FAIL") { n++; print "note: " $1 " failed" > "/dev/stderr" } } END { print n + 0 }' merged.log 2>/dev/null)
echo "captured: [$count]"

captured: [2]

awk '{ if ($2 == "FAIL") { n++; print "note: " $1 " failed" > "/dev/stderr" } } END { print n + 0 }' merged.log 2>&1 >/dev/null

note: fifo failed
note: alu failed

The same command, read two ways. A caller that wants the number gets exactly the number.

Exercise 6 — Engineering application

Split a merged regression log into per-test files, write a sorted summary index to its own file, and report the number of failing tests on stderr with a non-zero exit status.

rm -f index.txt *.tlog
awk '{ print > ($1 ".tlog"); seen[$1]++; if ($2 == "FAIL") bad[$1]++ } END { for (t in seen) printf "%-6s %d record(s), %d failure(s)\n", t, seen[t], bad[t] + 0 | "sort > index.txt"; close("sort > index.txt"); n = 0; for (t in bad) n++; printf "%d test(s) had failures\n", n > "/dev/stderr"; if (n) exit 1 }' merged.log 2>&1 >/dev/null

3 test(s) had failures

cat index.txt

alu    2 record(s), 1 failure(s)
fifo   1 record(s), 1 failure(s)
uart   1 record(s), 0 failure(s)

The index is right and the stderr count is wrong. Read the index: two tests had failures, alu and fifo. The summary says three.

The cause is Chapter 6.1 section 5. The printf loop evaluates bad[t] + 0 for every test in seen[], including uart — and reading bad["uart"] creates it. By the time for (t in bad) n++ runs, bad[] has gained a phantom key, so the count is the number of tests rather than the number of failing tests.

Note how this presents: the per-test column is correct (0 failure(s) for uart), and only the aggregate is wrong. The detailed rows exonerate the summary line, which is the opposite of where anyone looks first.

Two independent repairs, and the command needs only one:

rm -f index.txt
awk '{ print > ($1 ".tlog"); seen[$1]++; if ($2 == "FAIL") bad[$1]++ } END { n = 0; for (t in bad) n++; for (t in seen) printf "%-6s %d record(s), %d failure(s)\n", t, seen[t], (t in bad ? bad[t] : 0) | "sort > index.txt"; close("sort > index.txt"); printf "%d test(s) had failures\n", n > "/dev/stderr"; if (n) exit 1 }' merged.log 2>&1 >/dev/null

2 test(s) had failures

cat index.txt

alu    2 record(s), 1 failure(s)
fifo   1 record(s), 1 failure(s)
uart   1 record(s), 0 failure(s)

2, correctly. Both fixes are present: n is counted before anything reads bad[], and the lookup uses (t in bad ? bad[t] : 0) so no phantom is created at all. Either alone would be enough; together the command is robust to a later edit that reorders the END rule.

Two details worth naming. The pipe destination is the whole string "sort > index.txt" — that > is the shell's, inside the command awk runs, which is a different operator from awk's own > earlier in the same line. And the close() is required before END finishes, so sort completes and index.txt exists by the time anything reads it; without it this is Chapter 6.2 section 10's ordering problem.

The engineering caveat: 2 record(s), 1 failure(s) counts log records, not distinct defects, and a test that produced no record appears nowhere — Chapter 5.6's point, and the reason a real regression index is built from the list of expected tests rather than from the log.

Exercise 7 — Challenge: a splitter that survives any number of keys

Section 8's limit makes a naive splitter fail past about ten thousand destinations. Write one that cannot, and show it working.

rm -rf parts && mkdir parts
awk '{ f = "parts/" $1 ".part"; print >> f; close(f) }' merged.log
for f in parts/*.part; do echo "--- $f ---"; cat "$f"; done

--- parts/alu.part ---
alu PASS 120
alu FAIL 60
--- parts/fifo.part ---
fifo FAIL 95
--- parts/uart.part ---
uart PASS 88

Both alu records present, and at most one descriptor open at any moment. >> is essential — with > this is Exercise 3's bug.

The cost is one open-and-close per record rather than per file, which on a large log is a great deal of system call overhead. The usual engineering answer avoids the whole trade-off:

awk '{ print $1 "\t" $0 }' merged.log | sort -k1,1

alu	alu FAIL 60
alu	alu PASS 120
fifo	fifo FAIL 95
uart	uart PASS 88

One output stream with a key column, sorted — no per-key files, no descriptor limit, no cleanup, and a form Chapter 7.6 shows how to make directly consumable. Reach for many output files when a consumer genuinely needs separate files; otherwise emit one keyed stream.

12. Summary

• awk's > truncates on first open and appends thereafter. It is not the shell's >, and this is what makes a one-line splitter work.
• A destination is an expression, so one program writes to many files — parenthesise any computed name.
• >> never truncates, so it accumulates across runs.
• close() on a > file makes the next write truncate again. Adding it to a per-record loop keeps only the last record.
• close() on a pipe flushes and waits, and a later write to the same command string starts a fresh process.
• The open-destination limit is awk's own, far below the OS limit — a fatal error at about ten thousand files.
• Past that limit, close as you go and use >>.
• "/dev/stderr" separates diagnostics from the answer, which is what makes a shell $(…) capture usable.
• Prefer one keyed, sorted stream to thousands of files unless a consumer needs the files.

Output can now go wherever it is needed. The rest of this module turns to input: Chapter 7.4 — NR, FNR, and FILENAME covers telling the cumulative record number from the per-file one when awk is reading a whole directory of logs — the three variables every multi-file script depends on, and the per-file reset that silently does not happen for a log that is empty.