awk Part II · Module 8 · Writing Reliable AWK Scripts

POSIX awk, gawk, and Portability

A script written on a workstation runs on a compute farm, and the two do not have the same awk. That matters more than it sounds, because the failure modes are not symmetric: a gawk function that is missing produces a fatal error naming it, while a gawk special variable that is missing is simply an ordinary variable your assignment creates and nothing reads. One costs a minute. The other produces a report in an arbitrary order while the code says it is sorted. This chapter sorts the features this track has used into three categories, and measures what the flag that is supposed to enforce the distinction actually does.

Advanced16 min readawkPart IIPortabilityPOSIXgawk

Part II · Module 8 · Chapter 8.6 · POSIX awk, gawk, and Portability

1. Why This Matters

"It works on my machine" is a portability problem in every language. In awk it has a specific shape, because the implementations differ in capability rather than in syntax — the same program is valid on all of them and does different things.

Three awks you will meet. one-true-awk (also called bwk or nawk) is the original, and is what macOS and the BSDs ship. gawk is GNU awk, the Linux default, with a large set of extensions. mawk is a fast minimal one, common on Debian-derived systems as /usr/bin/awk.

A script that uses a gawk extension works on a Linux developer machine and fails on a BSD build server — or, worse, on a farm node whose /usr/bin/awk is mawk.

Figure 1 — every awk feature falls into one of three categories, and the one in the middle is the one to be deliberate about. A POSIX feature is safe everywhere. A gawk-only feature fails somewhere, and how it fails depends on whether it is a function or a variable. A feature that works on your awk without being POSIX is the trap, because nothing on your machine will tell you it is not portable.

2. What You Are Running

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk --version

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk version 20200816

A one-true-awk version string — a date, with no product name. gawk prints a line beginning GNU Awk, which makes it the thing to test for:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if awk --version 2>/dev/null | grep -qi 'GNU Awk'; then echo "gawk"; else echo "NOT gawk"; fi

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

NOT gawk

Note --version and not -V:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -V 2>&1 | head -1

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk: unknown option -V ignored

Unknown, and ignored rather than refused — which is section 4's subject.

3. The POSIX Core

Everything in this list is POSIX and present on every awk. It is also almost everything this track has used:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

for fn in 'length("ab")' 'substr("abc",2)' 'index("abc","b")' 'split("a b",A)' 'sub(/a/,"b",S)' 'gsub(/a/,"b",S)' 'sprintf("%d",1)' 'int(1.9)' 'tolower("A")' 'toupper("a")' 'match("ab",/b/)' 'system("true")'; do
  printf "  %-22s " "$fn"
  awk "BEGIN { S=\"aaa\"; r = $fn; print \"ok\" }" </dev/null 2>&1 | head -1
done

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  length("ab")           ok
  substr("abc",2)        ok
  index("abc","b")       ok
  split("a b",A)         ok
  sub(/a/,"b",S)         ok
  gsub(/a/,"b",S)        ok
  sprintf("%d",1)        ok
  int(1.9)               ok
  tolower("A")           ok
  toupper("a")           ok
  match("ab",/b/)        ok
  system("true")         ok

The variables are likewise standard: NR, NF, FNR, FILENAME, FS, OFS, RS, ORS, SUBSEP, RSTART, RLENGTH, CONVFMT, OFMT, ENVIRON, ARGC, ARGV.

So a program built from Modules 1 to 8's core material is portable. The exceptions are specific and few, and they are sections 5 and 7.

4. `--posix` Does Nothing Here

The obvious way to check portability is to ask awk to enforce the standard. On this awk that does not work, and the way it fails is the point:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf 'alu 1\n' > t.txt
awk --posix '{ print "program ran:", $1 }' t.txt 2>&1
echo "rc=$?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk: unknown option --posix ignored

program ran: alu
rc=0

The warning is on stderr — followed by a blank line, which is why the output has a gap — the option is ignored, the program runs normally, and the status is 0.

With stderr discarded — which is most of a nightly job — there is nothing at all:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk --posix '{ print $1 }' t.txt 2>/dev/null
echo "rc=$?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

alu
rc=0

A portability flag that is itself unportable

This is a small joke with real consequences.

--posix is a gawk option. On gawk it does what you want: it disables gawk's own extensions so a script using one fails where it would otherwise work. On one-true-awk and mawk it is not an option at all, and this awk's response is to warn once and carry on.

So a build step written as awk --posix -f script.awk data is a portability check on gawk and a no-op everywhere else — and the machines where it is a no-op are exactly the ones whose awk is already restrictive. The check passes vacuously on the awk that needed no checking and is absent on the one that did.

There are two things to do instead, and neither is a flag.

Run the script under more than one awk. If gawk is available alongside the system awk, running both is a real test:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

for a in awk gawk mawk; do
  printf "%-6s " "$a"
  if command -v "$a" >/dev/null 2>&1; then awk_bin=$a; "$awk_bin" 'BEGIN { print "available" }' </dev/null; else echo "not installed"; fi
done

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk    available
gawk   not installed
mawk   not installed

Only one awk exists on this machine, so that test cannot run here — which is itself the useful information, and the reason section 7's list had to be built by probing rather than by comparison.

Keep a list of the extensions you knowingly use, as section 7 does, and check it by eye at review time. That is less satisfying than a flag and it is what actually works across implementations.

5. Present Here, Not POSIX

The dangerous category: features this awk supports that the standard does not require. Nothing on this machine will warn you.

length() on an array returns the element count:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { a[1] = 1; a[2] = 2; print "length(a) =", length(a) }' </dev/null

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

length(a) = 2

Chapter 6.1 section 12 measured this and recommended a counting loop for anything portable. It is the extension most likely to appear by accident, because length() is itself POSIX and applying it to an array looks natural.

A regular expression as RS:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf 'aXbXXc\n' | awk 'BEGIN { RS = "X+" } { printf "%d:[%s]\n", NR, $0 }'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

1:[a]
2:[b]
3:[c
]

Three records, split at both the single X and the double XX — which proves it is a regex rather than a literal string, since a literal X+ would match neither. POSIX specifies only a single-character RS (plus the RS="" paragraph mode of Chapter 3.4), so this is an extension on any awk that offers it.

Hexadecimal string-to-number conversion, from Chapter 5.3 section 9:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { print ("0x10" + 0) }' </dev/null

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

"/dev/stderr" as an output destination, from Chapter 7.3 section 9 — widely supported and not in POSIX awk. It is used throughout Module 8 because the alternative is worse, and the portable fallback is to let the shell split the streams.

6. gawk-Only, and Absent Here

The loud failures. Every one of these is fatal:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

for fn in 'asort(a)' 'asorti(a)' 'gensub(/x/,"y","g","x")' 'systime()' 'strftime("%Y")'; do
  printf "  %-26s " "$fn"
  awk "BEGIN { a[1]=1; r = $fn; print \"ok\" }" </dev/null 2>&1 | head -1
done

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  asort(a)                   awk: calling undefined function asort
  asorti(a)                  awk: calling undefined function asorti
  gensub(/x/,"y","g","x")    awk: calling undefined function gensub
  systime()                  awk: calling undefined function systime
  strftime("%Y")             awk: calling undefined function strftime

Five functions, five immediate errors naming the function. That is the good failure mode: the script stops, the message is specific, and the fix takes a minute.

systime() and strftime() are worth noting because their absence has no workaround inside awk — there is no POSIX way to get the current time. A script needing a timestamp has to receive one:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -v now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" 'BEGIN { print "stamped:", (now != "" ? "yes" : "no") }' </dev/null

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

stamped: yes

The shell supplies it with -v, which is Chapter 8.2's mechanism and the portable answer.

7. The Silent Failure

Now the category that costs real time. PROCINFO["sorted_in"] is gawk's way to order array iteration, and on this awk it is an ordinary array:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { for (i = 1; i <= 8; i++) a["key" i] = i; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

key3 key4 key5 key6 key7 key8 key1 key2

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; for (i = 1; i <= 8; i++) a["key" i] = i; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

key3 key4 key5 key6 key7 key8 key1 key2

Identical. The assignment had no effect, and awk reported nothing.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; n = 0; for (k in PROCINFO) n++; printf "PROCINFO holds %d element(s)\n", n }' </dev/null

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

PROCINFO holds 1 element(s)

One element — the one that assignment created. PROCINFO is not special here; it is a name like MYARRAY.

A missing function is fatal; a missing special variable is a no-op

This asymmetry is the most useful thing in the chapter.

A function that does not exist is a syntax-level failure. awk can see that asort is called and not defined, so it refuses. You find out on the first run.

A variable that does not exist cannot fail, because every awk variable springs into existence on first use — Chapter 5.1 section 2's rule. Assigning to PROCINFO["sorted_in"] is a perfectly valid thing to do on any awk; it just means nothing unless the implementation happens to read it.

So the extensions to fear are the variables, not the functions. gawk's IGNORECASE, FIELDWIDTHS, FPAT, RT and PROCINFO all behave this way: on gawk they change the program's behaviour, and elsewhere they are dead assignments that produce a plausible wrong result.

IGNORECASE is the most dangerous of them, because a case-insensitive match silently becoming case-sensitive changes which records a filter selects:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { IGNORECASE = 1 } /uvm_error/ { n++ } END { printf "matched %d of %d\n", n + 0, NR }' case.log

Azvya Education Pvt. Ltd.VLSI Mentor

case.log — synthetic. Mixed-case severities.

UVM_ERROR mismatch
uvm_error other
UVM_INFO fine

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

matched 1 of 3

One match of the two error lines. On gawk the same program matches both, so a filter developed on Linux silently halves its results on a BSD node — with no error, no warning and a plausible count.

The portable form states the cases in the pattern:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk '/[Uu][Vv][Mm]_[Ee][Rr][Rr][Oo][Rr]/ { n++ } END { printf "matched %d of %d\n", n + 0, NR }' case.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

matched 2 of 3

Verbose and correct everywhere. tolower($0) ~ /uvm_error/ is the readable alternative and is equally portable.

8. The Checklist

The features this track has used that are not POSIX, collected so a reviewer can check a script against a list:

Feature	Status	Portable alternative
`length(array)`	works here, not POSIX	count with a `for (k in a)` loop
regex `RS`	works here, not POSIX	single-char `RS`, or carry state across records
`"0x10" + 0` → 16	implementation-specific	validate with a positive pattern — Chapter 5.3 §11
`"/dev/stderr"`	widely supported, not POSIX	let the shell split the streams
`asort`, `asorti`	gawk only, fatal	hand-rolled sort or pipe to `sort` — Chapter 7.7
`gensub`	gawk only, fatal	`gsub` into a copy
`systime`, `strftime`	gawk only, fatal	pass a timestamp with `-v`
`PROCINFO`	gawk only, silent	index array or pipe to `sort` — Chapter 7.7
`IGNORECASE`	gawk only, silent	bracket the cases, or `tolower($0) ~ /…/`
`FIELDWIDTHS`, `FPAT`	gawk only, silent	`substr()` for fixed columns
`--posix`	gawk only, ignored	run under several awks; keep this list

The rule of thumb that follows: if a feature is not in section 3's core list, write down why you used it. A comment naming the extension is what makes a review possible, and Chapter 8.1's -f file is where the comment goes.

9. Common Mistakes

Using --posix as a portability check. It is a gawk option and is ignored elsewhere — section 4.

Assuming a gawk-only feature will fail loudly. Functions do; variables do not — section 7.

Setting PROCINFO or IGNORECASE and trusting it. On a non-gawk awk they are ordinary variables.

length(array) in a portable script. Count with a loop — Chapter 6.1 §12.

A regex RS. POSIX allows one character plus paragraph mode.

Relying on hex conversion. Implementation-specific.

Expecting systime(). There is no POSIX clock; pass the time in with -v.

Testing on one awk and calling it portable. One implementation cannot reveal an extension it supports.

awk -V for the version. It is --version; -V is ignored.

10. Exercises

Exercise 1 — Identify your awk

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk --version
if awk --version 2>/dev/null | grep -qi 'GNU Awk'; then echo "gawk"; else echo "NOT gawk"; fi

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk version 20200816
NOT gawk

A date-only version string and no GNU Awk line — one-true-awk.

Exercise 2 — Watch `--posix` do nothing

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf 'alu 1\n' > t.txt
awk --posix '{ print "ran:", $1 }' t.txt 2>&1
echo "rc=$?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk: unknown option --posix ignored

ran: alu
rc=0

Warned, ignored, ran, status 0.

Exercise 3 — A loud failure

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { a[1] = 1; n = asort(a); print n }' 2>&1 | head -1

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk: calling undefined function asort

Fatal and specific — the good failure mode.

Exercise 4 — Debug a silent one

A script sets PROCINFO to sort its output and the output is not sorted.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; for (i = 1; i <= 8; i++) a["key" i] = i; for (k in a) { printf "%s%s", sep, k; sep = " " }; print "" }'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

key3 key4 key5 key6 key7 key8 key1 key2

PROCINFO is a gawk variable. Here the assignment created an ordinary array element that nothing reads:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc"; n = 0; for (k in PROCINFO) n++; printf "PROCINFO holds %d element(s)\n", n }' </dev/null

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

PROCINFO holds 1 element(s)

The portable fix is Chapter 7.7's: pipe to sort, or keep an index array.

Exercise 5 — Debug a halved match count

A filter developed on Linux matches fewer records on a BSD node.

Azvya Education Pvt. Ltd.VLSI Mentor

case.log

UVM_ERROR mismatch
uvm_error other
UVM_INFO fine

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { IGNORECASE = 1 } /uvm_error/ { n++ } END { printf "matched %d of %d\n", n + 0, NR }' case.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

matched 1 of 3

One of the two error lines. IGNORECASE is gawk-only, so the match stayed case-sensitive.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'tolower($0) ~ /uvm_error/ { n++ } END { printf "matched %d of %d\n", n + 0, NR }' case.log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

matched 2 of 3

tolower() is POSIX, so this is portable and readable — better than bracketing every character.

Exercise 6 — Engineering application

Write a portability auditor: scan an awk script for the extensions in section 8's table and report them.

Azvya Education Pvt. Ltd.VLSI Mentor

audit.awk — synthetic teaching script.

# audit.awk — report non-POSIX awk features used by a script.
# Usage: awk -f audit.awk script-to-check.awk
# Exit:  0 nothing non-POSIX found
#        1 extensions found (listed on stdout)

BEGIN {
    # feature -> why it matters
    why["asort"]     = "gawk only, fatal elsewhere"
    why["asorti"]    = "gawk only, fatal elsewhere"
    why["gensub"]    = "gawk only, fatal elsewhere"
    why["systime"]   = "gawk only, fatal elsewhere"
    why["strftime"]  = "gawk only, fatal elsewhere"
    why["PROCINFO"]  = "gawk only, SILENTLY INERT elsewhere"
    why["IGNORECASE"]= "gawk only, SILENTLY INERT elsewhere"
    why["FIELDWIDTHS"]="gawk only, SILENTLY INERT elsewhere"
    why["FPAT"]      = "gawk only, SILENTLY INERT elsewhere"
}

# Skip comment lines so a mention in a comment is not a finding.
/^[ \t]*#/ { next }

{
    for (f in why)
        if (index($0, f)) printf "%s:%d: %s — %s\n", FILENAME, FNR, f, why[f]
}

{ if (index($0, "/dev/stderr")) printf "%s:%d: /dev/stderr — widely supported, not POSIX\n", FILENAME, FNR }

END {
    # A second pass is not possible here, so count findings as we go would
    # need a counter; instead re-report the total from the caller's view.
    if (!NR) { print "ERROR: script is empty" > "/dev/stderr"; exit 2 }
}

Azvya Education Pvt. Ltd.VLSI Mentor

uses_gawk.awk — synthetic. A script with two extensions.

# a script that will not port
BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc" }
{ n[$1]++ }
END { k = asort(n); print k > "/dev/stderr" }

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f audit.awk uses_gawk.awk
echo "status: $?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

uses_gawk.awk:2: PROCINFO — gawk only, SILENTLY INERT elsewhere
uses_gawk.awk:4: asort — gawk only, fatal elsewhere
uses_gawk.awk:4: /dev/stderr — widely supported, not POSIX
status: 0

Three findings on the right lines, and the comment on line 1 correctly produced none.

The status is 0, which the usage comment says should be 1 when extensions are found — so the script's own contract is not met. Adding the counter the END comment admits is missing:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -f audit.awk uses_gawk.awk >/dev/null 2>&1; echo "before: $?"
awk 'BEGIN { why["asort"]=1; why["PROCINFO"]=1 } /^[ \t]*#/ { next } { for (f in why) if (index($0, f)) { printf "%s:%d: %s\n", FILENAME, FNR, f; found++ } } END { if (found) exit 1 }' uses_gawk.awk
echo "after: $?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

before: 0
uses_gawk.awk:2: PROCINFO
uses_gawk.awk:4: asort
after: 1

The second version counts its findings and exits 1, which is what Chapter 8.3's convention requires — a tool that reports problems and exits 0 is the failure mode Chapter 5.6 section 9 warned about, and my own first draft of the auditor had it.

Exercise 7 — Challenge: prove a check cannot run

Section 4 claimed a multi-awk test is the real portability check. Show whether it is possible here.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

for a in awk gawk mawk busybox; do
  printf "%-9s " "$a"
  if command -v "$a" >/dev/null 2>&1; then echo "available"; else echo "not installed"; fi
done

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk       available
gawk      not installed
mawk      not installed
busybox   not installed

One awk. So the multi-implementation test cannot be run on this machine, and every claim in this chapter about what gawk does is from its documentation rather than from measurement here.

That distinction is worth making explicit, because it bounds what the chapter proves. What is measured here is what this awk does — that asort is fatal, that PROCINFO is inert, that --posix is ignored, that a regex RS works. What is not measured is gawk's behaviour, and the section 8 table marks those rows as "gawk only" on that basis.

The practical consequence for a reader: if gawk is available to you, run the probes in sections 5 to 7 on it. The ones that differ are exactly the portability risks in your own scripts, and you will have measured them rather than read them.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk 'BEGIN { print "this machine:", "one-true-awk" }' </dev/null
awk --version

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

this machine: one-true-awk
awk version 20200816

11. Summary

Three awks are common: one-true-awk (macOS, BSD), gawk (Linux default), mawk (minimal, Debian-derived).
awk --version identifies it; a line containing GNU Awk means gawk. -V is ignored.
The POSIX core is almost everything this track uses — the standard functions and variables are present everywhere.
--posix is a gawk option and is warned-about-then-ignored here, so it is not a portability check.
A missing gawk function is fatal and specific — asort, asorti, gensub, systime, strftime.
A missing gawk special variable is a no-op, because every awk variable is created on first use. PROCINFO, IGNORECASE, FIELDWIDTHS, FPAT.
The silent category is the dangerous one: a case-insensitive filter silently becoming case-sensitive halves its results with no error.
Features that work here and are not POSIX — length(array), a regex RS, hex conversion, "/dev/stderr" — are the trap, because nothing local warns you.
There is no POSIX clock. Pass a timestamp in with -v.
Test on more than one awk if you can, and keep a written list of the extensions you chose.

Reliability now covers input, semantics and implementation. Chapter 8.7 — Testing an awk Script closes Module 8 and Part II with the thing that catches all of it: a sample input and its expected output kept beside the script, so a format change or a careless edit is caught by a diff rather than by a wrong nightly report.

Where this fits

Part of the awk curriculum.

1. Why This Matters

2. What You Are Running

3. The POSIX Core

4. --posix Does Nothing Here

5. Present Here, Not POSIX

6. gawk-Only, and Absent Here

7. The Silent Failure

8. The Checklist

9. Common Mistakes

10. Exercises

Exercise 1 — Identify your awk

Exercise 2 — Watch --posix do nothing

Exercise 3 — A loud failure

Exercise 4 — Debug a silent one

Exercise 5 — Debug a halved match count

Exercise 6 — Engineering application

Exercise 7 — Challenge: prove a check cannot run

11. Summary

Where this fits

4. `--posix` Does Nothing Here

Exercise 2 — Watch `--posix` do nothing