Script Files and the -f Option

Part II · Module 8 · Chapter 8.1 · Script Files and the -f Option

1. Why This Matters

A one-liner is the right form for a command you type once. It is the wrong form for anything that runs every night, and the reasons are specific rather than aesthetic.

Four things a quoted one-liner cannot do:

• Carry a comment. There is nowhere to say why the threshold is -0.05.
• Define a function readably. Chapter 6.5's functions are legal in a one-liner and unreadable in one.
• Produce a reviewable diff. A changed one-liner in a shell script shows as one modified line of 300 characters.
• Report an error usefully. awk cannot name a line in a program that has no lines.

This chapter is about moving the program into a file. Everything after it in Module 8 assumes you have.

2. The Problem With Quotes

Before the solution, the specific pain. A one-liner lives inside shell quotes, so two languages are parsing the same characters — and Chapter 12.1 is devoted to the consequences. The short version is that a single-quoted awk program cannot contain a single quote, which rules out an apostrophe in a message, and any shell variable spliced into it changes the program text rather than passing a value (Chapter 8.2 section 6 shows what that costs).

A -f file is parsed only by awk. The shell never sees it.

3. A Script File

A program in a file, with everything a one-liner cannot have:

# summarise.awk — reduce a regression log to a verdict.
# Usage: awk -f summarise.awk sim.log

function pct(a, b) { return b ? 100 * a / b : 0 }

BEGIN { threshold = 100 }

$2 == "FAIL" { fails++ }
$3 + 0 > threshold { slow++ }

END {
    if (!NR) { print "ERROR: no records" > "/dev/stderr"; exit 2 }
    printf "%d of %d failed (%.1f%%), %d over %ds\n", fails + 0, NR, pct(fails, NR), slow + 0, threshold
}

alu PASS 120
fifo FAIL 95
uart PASS 88

awk -f summarise.awk sim.log

1 of 3 failed (33.3%), 1 over 100s

Comments, blank lines, a function, a BEGIN block setting a named threshold, two rules and an END — all in a layout a reader can follow.

Three details worth noticing. Comments start with # and run to the end of the line; there is no block comment. Blank lines are free and are the cheapest readability improvement available. And the threshold has a name, so the comment explaining it has something to attach to — which is the whole argument in miniature.

4. What the Shell Still Does

The shell is not entirely out of the picture: it still expands the filenames and routes the output.

awk -f summarise.awk sim.log > report.txt
cat report.txt

1 of 3 failed (33.3%), 1 over 100s

That > is the shell's, and Chapter 7.3 established that awk's own > inside the program is a different operator. In a -f file the distinction is at least visible — the shell's redirections are on the command line and awk's are in the script.

5. Errors Name a File and a Line

The diagnostic benefit is immediate and is the reason to switch even for a program you will run twice.

{ print $1

awk -f broken.awk sim.log 2>&1 | head -3 | cat -e

awk: syntax error at source line 0 source file broken.awk$
 context is$
	{ print $1 >>> $

The file is named. On a script of two hundred lines that is the difference between a fix and a hunt.

The same error in a one-liner has no file to name and no line number that means anything, so awk can only echo the program text back with a marker — which for a long one-liner is a wall of characters you then have to read.

The >>> marks where awk stopped understanding. Reading it as "everything before this was fine" is the fastest way to locate an unbalanced brace or quote.

6. Several -f Files

-f may be given more than once, and the programs are concatenated. That makes a shared function library possible:

function pct2(a, b) { return b ? 100 * a / b : 0 }

{ n++ }
END { printf "%.1f%%\n", pct2(n, NR) }

awk -f lib.awk -f main.awk sim.log

100.0%

main.awk calls a function it does not define, and the call resolves because lib.awk was loaded first. Chapter 6.5 section 3 established that definition order does not matter, so the -f order is a convention rather than a requirement.

This is how a team shares awk code: one library of guarded helpers — a percentage that cannot divide by zero, a numeric validator, a quoting function for CSV — and a short program per job. It is also the only module system awk has.

7. A Self-Executing Script

A #! line makes the script a command in its own right:

printf '#!/usr/bin/awk -f\n{ n++ }\nEND { printf "%%d record(s)\\n", n }\n' > report
chmod +x report
cat report

#!/usr/bin/awk -f
{ n++ }
END { printf "%d record(s)\n", n }

./report sim.log

3 record(s)

The kernel runs /usr/bin/awk -f ./report sim.log, so the -f in the shebang line is doing the same job as before.

The neat part is that the same file still works the other way:

awk -f report sim.log

3 record(s)

#!/usr/bin/awk -f begins with #, which is a comment to awk. So a shebang script is also a valid -f file, and you can run it either way without maintaining two copies.

8. When to Stay a One-Liner

The chapter is not an argument that every awk program should be a file.

Stay a one-liner when the command is exploratory, when it is one stage of a pipeline you are composing interactively, or when it is short enough that a reader of the shell script can see what it does without opening anything. awk '{ print $3 }' in the middle of a pipeline is clearer inline than as a file called print_third_field.awk.

Move to a file when the program has a function, needs a comment, exceeds roughly one screen width, will be edited by somebody else, or is run by automation. The last one is the strongest: a nightly job's awk program is a maintained artefact whether or not anybody has treated it as one.

The honest middle case is a program of two or three rules with no functions, called from one shell script. Either form is defensible; the question to ask is whether the next person to change it will be able to see what it does.

9. Common Mistakes

Forgetting -f. awk summarise.awk sim.log treats the script as the program text and then as a filename.

Expecting a block comment. There is none; # to end of line only.

Putting -f after the program text. It is not recognised there — Chapter 8.2 section 8 shows the same hazard for -v.

Hard-coding /usr/bin/awk in a shebang. Use a shell wrapper for anything portable — section 7.

Leaving a one-liner in a nightly job. It is a maintained artefact; give it a file.

Assuming the -f order matters for functions. It does not; definition order is irrelevant.

10. Exercises

Exercise 1 — Predict the output

awk -f summarise.awk sim.log

1 of 3 failed (33.3%), 1 over 100s

One FAIL of three records, and one runtime above the 100-second threshold set in BEGIN.

Exercise 2 — Read the error

{ print $1

awk -f broken.awk sim.log 2>&1 | head -3 | cat -e

awk: syntax error at source line 0 source file broken.awk$
 context is$
	{ print $1 >>> $

The file is named and >>> marks where parsing stopped — an unclosed brace.

Exercise 3 — Combine two -f files

Call a function defined in one file from a program in another.

function pct2(a, b) { return b ? 100 * a / b : 0 }

{ n++ }
END { printf "%.1f%%\n", pct2(n, NR) }

awk -f lib.awk -f main.awk sim.log

100.0%

Exercise 4 — One file, two ways to run it

Show that a shebang script is also a valid -f file.

printf '#!/usr/bin/awk -f\n{ n++ }\nEND { printf "%%d record(s)\\n", n }\n' > report
chmod +x report
./report sim.log
awk -f report sim.log

3 record(s)
3 record(s)

Identical, because #! is a comment to awk.

Exercise 5 — Engineering application

Turn a Module 7 one-liner into a documented script: a merge of a timing and a coverage report with a guarded percentage, a stated threshold and a usable exit status.

# signoff.awk — merge timing and coverage on the block name.
#
# Usage:  awk -f signoff.awk timing.csv cov.csv
# Exit:   0 all blocks complete and meeting threshold
#         1 at least one block incomplete or below threshold
#         2 could not evaluate (empty or missing input)
#
# Both inputs are comma-separated: name,value

BEGIN { FS = ","; MIN_COV = 80 }

function pct(a, b) { return b ? 100 * a / b : 0 }

# Phase 1 — load the timing report, keyed on block name.
FILENAME == ARGV[1] { slack[$1] = $2; key[$1]; next }

# Phase 2 — load coverage and record the union of keys.
{ cov[$1] = $2; key[$1] }

END {
    if (!NR) { print "ERROR: no records in either input" > "/dev/stderr"; exit 2 }

    for (k in key) {
        n++
        have_t = (k in slack); have_c = (k in cov)
        if (!have_t || !have_c) { incomplete++; st = "INCOMPLETE" }
        else if (cov[k] + 0 < MIN_COV) { low++; st = "LOW_COV" }
        else st = "ok"
        printf "%-6s %8s %6s  %s\n", k, (have_t ? slack[k] : "-"), (have_c ? cov[k] : "-"), st | "sort"
    }
    close("sort")
    printf "(%d block(s); %d incomplete, %d below %d%% coverage)\n", n, incomplete + 0, low + 0, MIN_COV
    if (incomplete || low) exit 1
}

alu,0.5
fifo,-0.2
uart,0.1

alu,95
uart,88
ddr,70

awk -f signoff.awk timing.csv cov.csv
echo "status: $?"

alu         0.5     95  ok
ddr           -     70  INCOMPLETE
fifo       -0.2      -  INCOMPLETE
uart        0.1     88  ok
(4 block(s); 2 incomplete, 0 below 80% coverage)
status: 1

Everything the one-liner version of this did, plus three things it could not: a usage comment stating the exit codes, a named MIN_COV the comment can explain, and a layout in which the two phases are visibly separate.

Note FILENAME == ARGV[1] rather than FNR == NR for the phase split — Chapter 7.5 section 8's choice, made here because the two inputs are definitely different files and either could be empty.

Exercise 6 — Challenge: a portable wrapper

Package the script so a caller needs to know nothing about awk, and show it working.

printf '#!/bin/sh\n# signoff — wrapper around signoff.awk\nif [ $# -ne 2 ]; then\n  echo "usage: $0 timing.csv cov.csv" >&2\n  exit 2\nfi\nexec awk -f "$(dirname "$0")/signoff.awk" "$@"\n' > signoff
chmod +x signoff
./signoff timing.csv cov.csv
echo "status: $?"

alu         0.5     95  ok
ddr           -     70  INCOMPLETE
fifo       -0.2      -  INCOMPLETE
uart        0.1     88  ok
(4 block(s); 2 incomplete, 0 below 80% coverage)
status: 1

And the argument check refuses a bad call before awk runs:

./signoff timing.csv 2>&1
echo "status: $?"

usage: ./signoff timing.csv cov.csv
status: 2

Three properties worth having. exec replaces the shell rather than nesting a process, so the exit status passes through unchanged — which matters because Chapter 5.6 established that the status is how awk reports its findings. $(dirname "$0") finds the awk file beside the wrapper, so the pair can be installed anywhere. And "$@" passes the arguments through with quoting intact, so a path containing a space still works.

The argument count is checked in the shell rather than with awk's ARGC, which Chapter 7.5 section 7 used. Either works; doing it in the wrapper means the error message can name the wrapper rather than awk, which is what a caller expects to see.

11. Summary

• A -f file buys four things a one-liner cannot have: comments, readable functions, a reviewable diff, and errors that name a file.
• The shell never parses a -f file, which removes the quoting problems of a one-liner.
• # comments run to end of line; there is no block comment.
• A syntax error names the source file and marks where parsing stopped with >>>.
• Several -f files concatenate, which is awk's only module system — a shared library plus a short program per job.
• #!/usr/bin/awk -f works, and the same file still works via -f, because #! is a comment to awk.
• A shebang hard-codes an interpreter path. A three-line shell wrapper is the portable form.
• Stay a one-liner for exploration and short pipeline stages; move to a file for anything with a function, a comment, or a cron entry.

The program now has somewhere to live. Chapter 8.2 — Passing Variables With -v is about getting values into it from outside: why -v is visible in BEGIN while a file=value assignment is not, and why splicing a shell variable into the program text instead lets the value run as code.