User-Defined Functions

Part II · Module 6 · Chapter 6.5 · User-Defined Functions

1. Why This Matters

Chapter 6.4's final exercise was one line holding an accumulation, a seeding guard, a threshold test, two printf calls and an exit status. It worked, and it is at the limit of what a line should be asked to carry.

A function fixes three things at once. The percentage calculation appears once instead of three times. The guard has a name. And the top-level program reads as a description of what it does rather than how.

awk's functions are ordinary in syntax and genuinely unusual in semantics, and the unusual parts are not obscure corners — they are the first three things you will hit. Sections 5 to 9 are about them, and section 8's pair of commands is the one to remember.

2. Defining and Calling

awk 'function pct(a, b) { return b ? 100 * a / b : 0 } BEGIN { print pct(3, 4) }'

75

The parts: the keyword function, a name, a parenthesised parameter list, and a braced body. return sends a value back.

The guard b ? … : 0 is there because Chapter 5.3 established that division by zero is fatal. Putting it inside the function means every caller gets it for free, which is the first real argument for writing one.

Used on real input:

alu,95,100
fifo,72,100
uart,88,100
spi,45,100

awk -F, 'function pct(a, b) { return b ? 100 * a / b : 0 } { p += $2; t += $3 } END { printf "%.1f%%\n", pct(p, t) }' cov.csv

75.0%

That is Chapter 6.4's weighted aggregation with the division named and guarded in one place.

3. Order Does Not Matter

A function may be called before it is defined — awk reads the whole program before running any of it:

awk 'BEGIN { print pct(3, 4) } function pct(a, b) { return b ? 100 * a / b : 0 }'

75

No forward declaration and no prototype. Put functions wherever they read best — conventionally after the rules, or in their own -f file, which is Chapter 8.1's subject.

One syntactic requirement: a function definition cannot be inside a rule. It is a top-level construct, alongside BEGIN, END and the pattern-action rules.

4. Scalars Pass by Value

A scalar argument is copied. The function cannot change the caller's variable:

awk 'function tryset(x) { x = "changed" } BEGIN { v = "original"; tryset(v); print "v =", v }'

v = original

tryset assigned to its parameter and the caller's v is untouched. This is what most languages do and it is the half of awk's parameter model that will not surprise you.

5. Arrays Pass by Reference

An array argument is not copied. The function and the caller share one array:

awk 'function addone(arr) { arr["from_function"] = 1 } BEGIN { a["from_main"] = 1; addone(a); n = 0; for (k in a) n++; print "a has", n, "keys:"; for (k in a) print "  ", k }'

a has 2 keys:
   from_function
   from_main

The key the function added is visible to the caller. One array, two names for it.

This is deliberate and it is the only way to get structured data out of an awk function, because return can only return a scalar. So the idiom for a function that produces a collection is to pass it an array to fill:

u_alu comb 1820
u_regfile seq 2410
u_alu comb 640

awk 'function tally(rec, out) { out[rec]++ } { tally($1, byname) } END { for (k in byname) print k, byname[k] | "sort" }' area.rpt

u_alu 2
u_regfile 1

byname was never assigned in the main program — it came into existence as an argument and the function filled it.

6. There Is No Local Scope

Here is where awk departs from everything else. A variable used inside a function, which is not a parameter, is global:

awk 'function f() { i = 999 } BEGIN { i = 1; f(); print "i after calling f():", i }'

i after calling f(): 999

The caller set i to 1, called a function that mentions i, and i is now 999.

There is no local, no var, no my, and no block scope. Every name in an awk program refers to one variable for the whole program — which Chapter 5.1 section 11 noted about the three phases and which is now a correctness problem rather than a convenience.

7. Extra Parameters Are the Only Locals

The workaround is a convention rather than a feature: declare more parameters than you pass. Unpassed parameters are unset, and they are local to the call.

awk 'function f(   i) { i = 999 } BEGIN { i = 1; f(); print "i after calling f() with a local i:", i }'

i after calling f() with a local i: 1

i is now a parameter of f. The caller passed nothing for it, so inside f it starts unset and is independent of the global i.

An unpassed parameter really is unset, not zero-filled or inherited:

awk 'function f(a, b, c) { return "[" a "][" b "][" c "]" } BEGIN { print f("x") }'

[x][][]

b and c are empty, with the usual dual nature from Chapter 5.1 — empty string and 0.

The convention is the run of extra spaces before the locals:

awk does not care about the spacing — function stats(arr, n, i, s, mx) is the identical program. The gap is purely for human readers, telling them where the arguments stop and the locals begin. It is universal in awk code and worth following exactly because nothing in the language enforces it.

8. The Leak, and Why It Survives

The same function written both ways. This is the pair to remember.

With locals declared:

awk 'function stats(arr, n,    i, s, mx) { for (i = 1; i <= n; i++) { s += arr[i]; if (i == 1 || arr[i] > mx) mx = arr[i] } return s "/" n " max=" mx } BEGIN { v[1]=10; v[2]=30; v[3]=20; print stats(v, 3); print "i is still [" i "], s is [" s "], mx is [" mx "]" }'

60/3 max=30
i is still [], s is [], mx is []

Without them:

awk 'function stats(arr, n) { for (i = 1; i <= n; i++) { s += arr[i]; if (i == 1 || arr[i] > mx) mx = arr[i] } return s "/" n " max=" mx } BEGIN { v[1]=10; v[2]=30; v[3]=20; print stats(v, 3); print "i is now [" i "], s is [" s "], mx is [" mx "]" }'

60/3 max=30
i is now [4], s is [60], mx is [30]

9. What the Leak Breaks

The leak is real; what it damages is worth being precise about, because the obvious guess is wrong.

A for (key in array) loop keeps its iteration count even when a called function clobbers the loop variable:

awk 'function helper() { k = "CLOBBERED"; return 1 } BEGIN { a["x"]=1; a["y"]=2; a["z"]=3; for (k in a) { helper(); print "loop sees k =", k } }'

loop sees k = CLOBBERED
loop sees k = CLOBBERED
loop sees k = CLOBBERED

Three iterations, which is right — awk tracks its position in the array internally, not through the variable. But the value is destroyed, so the loop body cannot tell which key it is on, and a[k] would read a non-existent element (creating it, by Chapter 6.1 section 5).

A counting for loop does not survive. It becomes infinite:

awk 'function helper() { for (i = 1; i <= 3; i++) ; return 1 } BEGIN { n = 0; for (i = 1; i <= 5; i++) { helper(); n++; if (n > 20) { print "BAILED - the loop never advances"; break } } print "iterations:", n }'

BAILED - the loop never advances
iterations: 21

The helper leaves i at 4 every time, the outer loop's i++ makes it 5, the test passes, and it runs forever. The bail-out counter is the only reason that command terminates — Chapter 5.5 section 12's technique.

With the local declared, the same program is correct:

awk 'function helper(   i) { for (i = 1; i <= 3; i++) ; return 1 } BEGIN { n = 0; for (i = 1; i <= 5; i++) { helper(); n++; if (n > 20) { print "BAILED"; break } } print "iterations:", n }'

iterations: 5

An accumulator is silently zeroed. The quietest of the three:

a 1
b 2
c 3

awk 'function fmt(v) { total = 0; return "<" v ">" } { total += $2; print fmt($1) } END { print "total =", total + 0 }' t.txt

<a>
<b>
<c>
total = 0

The formatting worked on every record and the total is 0, because fmt resets it each call. No error, no warning, and a summary line that looks like a legitimate result.

10. Calling Pitfalls

No space before the parenthesis. A function call must have the name touching the (:

awk 'function f(x) { return x } BEGIN { print f (5) }' 2>&1 | head -2

awk: can't read value of f; it's a function.
 source line number 1

With a space, awk reads f as a variable, and concatenation (Chapter 5.3, section 12) would have joined it to (5). Since f is a function name, that is an error — which is fortunate, because the alternative would be a silent concatenation.

Built-in functions are more relaxed about this, which makes the rule easy to forget. For your own functions it is strict.

Too many arguments is a warning, not an error. The program continues:

awk 'function f(a) { return a } BEGIN { print f(1, 2, 3) }' 2>&1

awk: function f called with 3 args, uses only 1
 source line number 1
1

The message goes to stderr and 1 is printed. So in a pipeline where stderr is discarded — which is most nightly jobs — passing too many arguments is completely silent, and the extra values are dropped.

That is worth a moment, because the usual cause is adding a parameter to a function and missing one of its call sites, or the reverse. The arity mismatch does not stop the run; it just loses data.

Too few arguments is not even a warning, as section 7 showed — the missing parameters are simply unset, which is the same mechanism the locals convention depends on. awk cannot distinguish "you forgot an argument" from "that is a declared local", so it never complains.

11. Returning a Value

return expr sends a scalar back. A bare return yields the empty string:

awk 'function f() { return } BEGIN { print "[" f() "]", f() + 0 }'

[] 0

The usual unset duality. A function that falls off its end without return behaves the same way.

You cannot return an array. Fill one passed in as a parameter — section 5.

Recursion works:

awk 'function fact(n) { return n <= 1 ? 1 : n * fact(n - 1) } BEGIN { print fact(5), fact(10) }'

120 3628800

Correct. Recursion is rare in awk report-processing but genuinely useful for walking a hierarchy — and the locals convention becomes mandatory rather than advisable, since every recursive call shares every global.

12. A Function as a Pattern

A function returning a truth value can be the pattern of a rule, which is where functions start improving readability rather than just removing duplication:

awk -F, 'function low(p, t) { return t ? 100 * p / t < 50 : 0 } low($2, $3) { print $1, "below 50%" }' cov.csv

spi below 50%

The rule reads as "for low blocks, print them" — the threshold, the division and its zero guard are all named and out of the way. Compare with the same test inline, where the reader has to decode the arithmetic to find out what the rule selects.

This is the strongest argument for functions in awk. Chapter 4.3 built compound conditions out of operators; a named predicate does the same job and says what it means.

13. Common Mistakes

Using a working variable without declaring it as an extra parameter. It is global, and the function still returns the right answer — section 8.

Expecting a changed scalar parameter to reach the caller. Scalars are copied.

Expecting an array parameter to be a copy. It is not; the function shares it.

Trying to return an array. Pass one in and fill it.

A space between a function name and (. An error for your own functions.

Passing too many arguments. A stderr warning the program ignores — silent in any job that discards stderr.

Reusing i, s, n, k, tmp or max as an undeclared function variable. These are precisely the names the caller is using.

Defining a function inside a rule. It is a top-level construct.

Assuming recursion is safe without locals. Every global is shared across every frame.

14. Exercises

Exercise 1 — Predict the output

awk 'function double(x) { return x * 2 } BEGIN { print double(21) }'

42

Exercise 2 — Write a guarded percentage

Write a function that returns a percentage and never divides by zero, then use it on the coverage file.

alu,95,100
fifo,72,100
uart,88,100
spi,45,100

awk -F, 'function pct(a, b) { return b ? 100 * a / b : 0 } { printf "%-6s %5.1f%%\n", $1, pct($2, $3) }' cov.csv

alu     95.0%
fifo    72.0%
uart    88.0%
spi     45.0%

The guard is written once and applies to every call — including, had the data contained one, a row with a zero total.

Exercise 3 — Scalars against arrays

Predict both before running.

awk 'function f(s, arr) { s = "set by f"; arr["set"] = "by f" } BEGIN { v = "untouched"; f(v, a); print "scalar v =", v; print "array a[set] =", a["set"] }'

scalar v = untouched
array a[set] = by f

Identical call syntax, opposite behaviour. The scalar was copied; the array was shared.

Exercise 4 — Debug a leaked loop counter

A script hangs. The helper and the loop are below.

awk 'function helper() { for (i = 1; i <= 3; i++) ; return 1 } BEGIN { n = 0; for (i = 1; i <= 5; i++) { helper(); n++; if (n > 20) { print "BAILED - the loop never advances"; break } } print "iterations:", n }'

BAILED - the loop never advances
iterations: 21

helper uses i without declaring it, so it is the same i the outer loop is counting on. Every call leaves it at 4; the outer i++ makes 5; the test i <= 5 passes; forever.

The bail-out counter is what makes this diagnosable rather than just a hang — note that it also tells you the loop ran 21 times rather than 5, which is the evidence.

awk 'function helper(   i) { for (i = 1; i <= 3; i++) ; return 1 } BEGIN { n = 0; for (i = 1; i <= 5; i++) { helper(); n++; if (n > 20) { print "BAILED"; break } } print "iterations:", n }'

iterations: 5

One declared parameter.

Exercise 5 — Debug a zeroed total

A per-record formatter works and the summary reports zero.

a 1
b 2
c 3

awk 'function fmt(v) { total = 0; return "<" v ">" } { total += $2; print fmt($1) } END { print "total =", total + 0 }' t.txt

<a>
<b>
<c>
total = 0

fmt assigns to total, which is the caller's accumulator. Each record adds its value and then the formatter wipes it, so the final value is whatever the last call left — zero.

Both fixes work and they are not equivalent:

awk 'function fmt(v,   total) { total = 0; return "<" v ">" } { total += $2; print fmt($1) } END { print "total =", total + 0 }' t.txt

<a>
<b>
<c>
total = 6

Declaring total local to fmt repairs it without touching the caller. Removing the pointless assignment is better still, since fmt never needed a total at all:

awk 'function fmt(v) { return "<" v ">" } { total += $2; print fmt($1) } END { print "total =", total + 0 }' t.txt

<a>
<b>
<c>
total = 6

The second is the real fix. The first is what you reach for when the function's internals are genuinely needed and merely named badly.

Exercise 6 — Engineering application

Refactor Chapter 6.4's coverage gate into functions: a guarded percentage, a named below-threshold predicate, and a summary that reports the weighted figure with both counts and an exit status.

awk -F, '
function pct(a, b)      { return b ? 100 * a / b : 0 }
function low(p, t, lim) { return pct(p, t) < lim }
function report(h, g, blocks, nlow,   w) {
  w = pct(h, g)
  printf "coverage %.2f%% (%d of %d bins over %d blocks)\n", w, h, g, blocks
  if (nlow) printf "%d block(s) below threshold\n", nlow
  return nlow ? 1 : 0
}
{ hit += $2; tot += $3; n++; if (low($2, $3, 50)) nlow++ }
END {
  if (!tot) { print "ERROR: no bins"; exit 2 }
  exit report(hit, tot, n, nlow)
}' bins.csv
echo "status: $?"

blockA,180,200
blockB,5,500
blockC,95,100

coverage 35.00% (280 of 800 bins over 3 blocks)
1 block(s) below threshold
status: 1

Three things improved over the one-liner. pct guards the division once and low calls it, so the threshold logic cannot disagree with the headline figure. report declares w as a local, so it cannot collide with anything in the rules. And the main program is four lines that say what happens, with the arithmetic named.

The same program on uniformly-covered data:

awk -F, '
function pct(a, b)      { return b ? 100 * a / b : 0 }
function low(p, t, lim) { return pct(p, t) < lim }
function report(h, g, blocks, nlow,   w) {
  w = pct(h, g)
  printf "coverage %.2f%% (%d of %d bins over %d blocks)\n", w, h, g, blocks
  if (nlow) printf "%d block(s) below threshold\n", nlow
  return nlow ? 1 : 0
}
{ hit += $2; tot += $3; n++; if (low($2, $3, 50)) nlow++ }
END {
  if (!tot) { print "ERROR: no bins"; exit 2 }
  exit report(hit, tot, n, nlow)
}' cov.csv
echo "status: $?"

coverage 75.00% (300 of 400 bins over 4 blocks)
1 block(s) below threshold
status: 1

75% overall and still a failing block — spi at 45%. Chapter 6.4's point that a headline figure and a per-block minimum are different facts, now enforced by a gate that checks both.

Exercise 7 — Challenge: prove a function does not leak

Write a check that detects whether a function has clobbered any of its caller's variables, and show it catching a real leak.

awk 'function leaky(arr, n) { for (i = 1; i <= n; i++) s += arr[i]; return s } BEGIN { v[1]=10; v[2]=20; i = "CALLER_I"; s = "CALLER_S"; r = leaky(v, 2); bad = 0; if (i != "CALLER_I") { print "LEAK: i was changed to [" i "]"; bad++ } if (s != "CALLER_S") { print "LEAK: s was changed to [" s "]"; bad++ } printf "result %s, %d leak(s)\n", r, bad; exit bad ? 1 : 0 }'
echo "status: $?"

LEAK: i was changed to [3]
LEAK: s was changed to [30]
result 30, 2 leak(s)
status: 1

Two leaks caught, and a non-zero status so a test harness notices. i ended at 3 — one past the last value its loop body saw, Chapter 5.5 section 3's rule — and s holds 30, the function's accumulated sum, sitting where the caller's string used to be.

What += did to that string on the way is worth seeing on its own:

awk 'BEGIN { s = "CALLER_S"; s += 10; print "[" s "]" }'

[10]

+= forces a numeric context, so Chapter 5.3's conversion rule turned "CALLER_S" into 0 before adding. The caller lost its value and its type, silently.

The same function with locals declared passes:

awk 'function clean(arr, n,   i, s) { for (i = 1; i <= n; i++) s += arr[i]; return s } BEGIN { v[1]=10; v[2]=20; i = "CALLER_I"; s = "CALLER_S"; r = clean(v, 2); bad = 0; if (i != "CALLER_I") { print "LEAK: i was changed to [" i "]"; bad++ } if (s != "CALLER_S") { print "LEAK: s was changed to [" s "]"; bad++ } printf "result %s, %d leak(s)\n", r, bad; exit bad ? 1 : 0 }'
echo "status: $?"

result 30, 0 leak(s)
status: 0

Now compare the two results carefully, because this is the whole point of the exercise: both versions returned 30. The leaky one produced the correct answer and destroyed two of its caller's variables. The clean one produced the correct answer and touched nothing.

The function's output is identical, so no test of its return value can tell them apart. A leak check has to examine the caller's state — which is what the i != "CALLER_I" and s != "CALLER_S" assertions do. Seed the caller's variables with recognisable sentinels, call the function, and assert they survived; it is the only cheap way to detect this class of bug, and worth doing once for any function a team will reuse.

15. Summary

• function name(params) { … } at top level, callable from anywhere, definable in any order.
• A guard written inside a function applies to every caller — the first real reason to write one.
• Scalars pass by value, so a function cannot change the caller's scalar.
• Arrays pass by reference, which is the only way to get a collection out of a function since return yields a scalar.
• The call syntax is identical for both, so you cannot tell from a call whether it modifies its arguments.
• There is no local scope. A non-parameter variable in a function is global.
• Extra parameters are the only locals, separated by a run of spaces that matters to readers and not to awk.
• The leak does not change the function's result, which is precisely why it survives testing.
• It breaks a counting loop into an infinite one, destroys a for-in loop variable's value while keeping its iteration count, and silently zeroes an accumulator.
• No space before the ( in a call to your own function.
• Too many arguments is a stderr warning the program ignores — silent wherever stderr is discarded.
• Declare every working variable. i, s, n, k, tmp and max are the names both sides will use.

That completes Module 6 and Part II's data-structure half. You can hold one value per key, count and group without knowing the keys in advance, turn the result into defensible numbers, and factor the whole thing into named functions that do not sabotage their caller.

Module 7 — Input, Output, and Multi-File Processing turns to getting data in and out: print against printf, formatting a table that stays aligned, redirecting records to per-test files, reading many logs in one run, writing CSV that survives a field containing your delimiter, and imposing an output order on the array iteration that Chapter 6.1 left deliberately unspecified.