awk vs Python

Part I · Module 1 · Chapter 1.5 · awk vs Python

1. The Question This Chapter Answers

Chapters 1.3 and 1.4 compared awk with two other small commands. This one compares it with a full programming language, and the shape of the question changes.

grep and sed cannot do some of what awk does — field positions, arithmetic. Python can do everything awk can do, and a great deal awk cannot. So "which is more capable" has an obvious and useless answer.

The useful question is different:

How much program does this job deserve?

Some jobs deserve one line and no files. Some deserve a program with functions, validation and a test. Getting that judgement right is a real engineering skill, and getting it wrong is expensive in both directions — a 200-line Python program where one line would do is waste, and a clever one-liner holding up a nightly regression is a liability.

2. What Each One Is

awk is a small programming language whose entire design assumes your input is records and fields. That assumption is built in: by the time your instruction runs, the line is already split. It has variables, arithmetic, conditions, loops, arrays and functions — it is a real language — but it has no modules, no data structures beyond the associative array, and no ecosystem.

Python is a general-purpose language that assumes nothing about your input. Reading a file, splitting a line and converting text to a number are things you ask for explicitly. In exchange you get data structures, a large standard library, exceptions, testing frameworks, and the ability to grow a script into a program other people maintain.

The asymmetry is deliberate on both sides. awk is small because it gave up generality; Python is general because it gave up assumptions.

3. The Same Three Jobs, Both Ways

Here is a small file. Four names and scores — nothing engineering-specific yet, because the point is the languages.

alice 72
bob 91
carol 58
erin 100

Job one: print the second column.

awk '{ print $2 }' scores.txt

72
91
58
100

with open("scores.txt") as f:
    for line in f:
        print(line.split()[1])

python3 column.py

72
91
58
100

Job two: print the names scoring above 80.

awk '$2 > 80 { print $1 }' scores.txt

bob
erin

with open("scores.txt") as f:
    for line in f:
        name, score = line.split()
        if int(score) > 80:
            print(name)

python3 filter.py

bob
erin

Job three: total the scores.

awk '{ total += $2 } END { print total }' scores.txt

321

total = 0
with open("scores.txt") as f:
    for line in f:
        total += int(line.split()[1])
print(total)

python3 total.py

321

Byte-identical output in all three pairs.

4. Reading Both Line by Line

Take job three, since it has the most going on.

The awk version, { total += $2 } END { print total }:

• { total += $2 } has no pattern in front of it, so it runs for every record. total += $2 adds the second field to a variable called total. You never declared total and never set it to zero — awk creates it on first use and treats it as 0 in arithmetic.
• END { print total } runs once, after the last record, and prints the accumulated value. Chapter 1.1 introduced END; Chapter 2.4 covers the execution model properly.

The Python version:

• total = 0 — you must create the variable explicitly. Python has no implicit zero.
• with open("scores.txt") as f: — opens the file and guarantees it is closed when the block ends. awk opened the file for you because the filename was an argument.
• for line in f: — iterates the lines. awk's record loop is implicit.
• line.split() — splits on whitespace into a list. awk did this before your code ran. [1] takes the second element, because Python lists start at index 0 while awk's fields start at $1.
• int(...) — converts the text to a number. awk converted automatically, which is convenient and is also what caused the subtle trap in Chapter 1.2.
• print(total) — after the loop, which is what awk's END block expresses.

Every line of the Python is doing something awk also does. The difference is that in awk four of those steps are assumptions and in Python they are statements.

5. What That Table Does and Does Not Prove

It proves that for a job matching awk's assumptions, awk needs less typing. That is real and it is why one-liners exist.

It does not prove awk is better. Every one of those assumptions is also a constraint:

• Input must be records-and-fields. Give awk nested JSON and the assumption is simply wrong.
• Whitespace-separated by default. Changing that is Module 3.
• Automatic numeric conversion is convenient and occasionally silent, as Chapter 1.2's unit-suffix trap showed.
• One implicit loop over one stream. Several files with per-file state needs Module 7; Python has for path in ... and a dictionary.

Python's extra four lines buy the right to not hold those assumptions. Whether that is a good trade depends entirely on the job, which is what the next two sections are really about.

6. The Difference That Actually Matters: Bad Input

Here is the comparison worth remembering, and it is not about line count.

A file where one row is missing its score — which is exactly what a truncated or interrupted report looks like:

alice 72
bob
carol 58

awk:

awk '{ total += $2 } END { print total }' ragged.txt

130

Python, written the way you would actually write it once you know input can be malformed:

total = 0
with open("ragged.txt") as f:
    for n, line in enumerate(f, 1):
        parts = line.split()
        try:
            total += int(parts[1])
        except (IndexError, ValueError):
            print(f"line {n}: cannot read a score from {line.strip()!r}")
print("total:", total)

python3 robust_total.py

line 2: cannot read a score from 'bob'
total: 130

Read both outputs carefully, because the interesting thing is easy to miss.

The totals are the same. 130 both times. awk did not get the arithmetic wrong — 72 + 58 is 130, and the missing field contributed nothing, exactly as Chapter 1.2 said it would.

The difference is that Python told you row 2 was unreadable and awk did not. awk's 130 is indistinguishable from the 130 you would get from a file with no problems. Python's 130 arrives with a note saying one row did not count.

7. Missing Files

The same pattern, one level up. Ask each for a file that is not there.

awk '{ print }' nosuch.txt

awk: can't open file nosuch.txt
 source line number 1

awk reports the problem and exits with status 2. That is good behaviour — it is loud, and a shell script checking the exit status will catch it.

Two details worth noticing. The message goes to standard error, not standard output, which is why capturing the command's output with $(...) or redirecting it to a data file gives you an empty result rather than an error mixed into your numbers. And the exact wording differs between awk implementations, so match on the exit status, never on the message text.

try:
    open("nosuch.txt")
except FileNotFoundError as e:
    print(type(e).__name__)

python3 missing.py

FileNotFoundError

Python raises an exception, which by default stops the program with a traceback, and which you can catch and decide about — skip the file, warn, substitute a default, or stop. That ability to decide, in the middle of a larger program, is most of what "Python scales better" actually means.

8. Startup and Workflow

There is a real convenience difference here, and it is worth stating carefully because it is usually overstated.

awk needs nothing to exist first. You type a command with the program inside quotes and it runs against the file you name. There is no file to create, no imports, no scaffolding. That makes it well suited to a question you are asking right now, at a prompt, about a file in front of you — and to being embedded in a shell script, where it is one more command among many.

Python's smallest useful unit is a little larger. Even the five-line versions above are more comfortable in a file than typed at a prompt, and that file is a thing to name, save and keep.

9. Choosing by Task Complexity

Practical signals that you have left awk's regime:

• You need several files with per-file results, not one aggregate.
• You need to validate input and report what was wrong, as in section 6.
• You need to join two sources of data on a key.
• The output is structured — nested, or needing a real format.
• Someone else will maintain it, so it needs a name, comments and tests.
• The exit status has to express something more than "the command ran".

Signals that you are still comfortably inside it:

• One file, or one stream from a pipe.
• One question, whose answer is a column, a count or a number.
• You will read the answer once and act on it.
• It belongs inside a shell script as one step.

10. A Semiconductor Example

A synthetic regression summary — one row per test, as Chapter 1.2 described.

test_a PASS
test_b FAIL
test_c PASS
test_d FAIL

The small question: which tests failed? This is squarely awk's regime, and it does not deserve a Python file:

awk '$2 == "FAIL" { print $1 }' regression.txt

test_b
test_d

Now the job grows, the way real jobs do. There is a directory of per-block summaries rather than one file. You want pass and fail counts per block, you want to be told about rows that do not parse, and you want the command's exit status to say whether the regression was clean, so a nightly job can act on it.

test_alu_smoke PASS
test_alu_rand FAIL
test_fifo_burst PASS

test_uart_tx PASS
test_uart_rx FAIL
test_uart_parity

import sys
from pathlib import Path

def read_summary(path):
    """Return (passes, failures, bad_rows) for one regression summary file."""
    passes = failures = 0
    bad_rows = []
    for n, line in enumerate(path.read_text().splitlines(), 1):
        fields = line.split()
        if len(fields) != 2:
            bad_rows.append(n)
            continue
        name, result = fields
        if result == "PASS":
            passes += 1
        elif result == "FAIL":
            failures += 1
        else:
            bad_rows.append(n)
    return passes, failures, bad_rows

total_fail = 0
for path in sorted(Path("runs").glob("*.txt")):
    passes, failures, bad_rows = read_summary(path)
    total_fail += failures
    note = f"  (unreadable rows: {bad_rows})" if bad_rows else ""
    print(f"{path.name:18} pass={passes} fail={failures}{note}")

print(f"total failures: {total_fail}")
sys.exit(1 if total_fail else 0)

python3 summarise.py

block_alu.txt      pass=2 fail=1
block_uart.txt     pass=1 fail=1  (unreadable rows: [3])
total failures: 2

Four things in that output are things the one-liner could not give you: a result per file, an explicit note that block_uart.txt row 3 did not parse, an aggregate across files, and an exit status of 1 that a nightly job can test. The validation is not decoration — row 3 is a real malformed row, and silence about it is how a regression gets reported as healthier than it is.

11. Portability and Dependencies

awk is present on essentially every Unix-like system, including minimal containers and compute-farm nodes you do not control. What varies is which awk — POSIX awk, gawk, mawk, BusyBox awk — and therefore which features are available. Chapter 8.6 covers that split.

Python is usually present but its version is not guaranteed, and a script that imports anything outside the standard library adds an installation requirement. The script in section 10 uses only sys and pathlib, which is a deliberate constraint worth keeping for anything that has to run on a machine you do not administer.

A practical rule: the fewer assumptions a script makes about the machine, the more places it runs. A POSIX awk one-liner makes almost none. A Python script using only the standard library makes few. A Python script with a dependency list makes a request of whoever runs it.

12. Common Mistakes

Reaching for Python by default because it is the language you know. A one-line question does not need a file, a name and a place to live. Chapter 1.2's point about proportional effort applies.

Reaching for awk by default because it is shorter. If the job needs validation or per-file state, the one-liner is not shorter, it is just missing those things.

Comparing a careful Python program with a careless awk one-liner, or the reverse. Compare equivalents, as section 3 did.

Off-by-one between the two languages. awk's first field is $1; Python's first list element is [0]. This catches people switching between them in one afternoon.

Expecting Python to convert text to numbers automatically. "72" + 1 is an error in Python, where awk would give 73. The explicitness is the feature.

Believing a performance claim you did not measure. Section 8.

13. Exercises

Exercise 1 — Translate in both directions

Using scores.txt, print the name and score of everyone scoring 90 or above — first in awk, then in Python.

alice 72
bob 91
carol 58
erin 100

awk '$2 >= 90 { print $1, $2 }' scores.txt

bob 91
erin 100

with open("scores.txt") as f:
    for line in f:
        name, score = line.split()
        if int(score) >= 90:
            print(name, score)

python3 atleast90.py

bob 91
erin 100

Same answer, and two details worth noticing. awk's comma in print $1, $2 inserts the separating space; Python's print(name, score) does the same thing by default, which is a rare case of the two languages agreeing on a convenience. And the Python needed int() — comparing the string "91" with the number 90 raises a TypeError rather than guessing, which is section 12's last mistake in miniature.

Exercise 2 — Modify a demonstrated command

Report the average score rather than the total, in awk.

awk '{ total += $2; n++ } END { if (n) print total / n }' scores.txt

80.25

Two accumulators and a guard. The if (n) matters: on an empty file n would be 0 and dividing by it is a mistake awk will not stop you from making. Chapter 6.4 returns to this.

Exercise 3 — Reason about silent behaviour

Before running anything, predict what each of these prints for ragged.txt, then check. Explain the difference in one sentence.

alice 72
bob
carol 58

awk '{ n++ } END { print n }' ragged.txt

3

awk '$2 != "" { n++ } END { print n }' ragged.txt

2

The first counts records — three lines, including the malformed one. The second counts records that actually have a second field, which is two. Neither number is wrong; they answer different questions, and reporting the first while meaning the second is how a bad row becomes invisible. This is section 6's lesson without any Python.

Exercise 4 — An engineering decision

You need a nightly job that reads every regression summary under a directory, reports failures per block, flags malformed rows, and exits non-zero if anything failed, so the CI system can act on it.

Which tool, and why?

Python, on the evidence of section 10 — four of those five requirements are about per-file state, validation, structured reporting and exit semantics, which is the regime the bottom row of Figure 1 describes. It will also be read and modified by people who are not you, which favours a named file with functions over a command buried in a shell script.

The defensible alternative: if the nightly job already exists as a shell script and you only need to add "how many failures in total", that is one awk command appended to something that works, and rewriting the whole thing in Python to add one number would be the mistake in the other direction.

Exercise 5 — Challenge: say what tips the balance

For each pair, both tools would work. Say which you would choose and name the single factor that decides it.

1. Print the worst slack from a two-column table, once, while debugging.
2. Produce the coverage report that goes to the project lead every Monday.
3. Count error lines in a log inside an existing shell script.
4. Join a timing table and a coverage table on instance name and emit a combined file.

Answers. 1 — awk; the factor is that you will read it once and throw it away. 2 — Python; the factor is that it is a recurring deliverable someone else depends on, so it needs a name, a home and a test. 3 — awk; the factor is that it is one step in something that already exists, and awk fits a shell script as one command. 4 — Python; the factor is the join, which needs two sources held in memory at once — awk can do this with arrays from Module 6 and Chapter 7.8, but a join is the point where structured data starts paying for itself.

If you chose on "which language do I prefer" for any of them, that is the habit this chapter is trying to replace.

14. Summary

• The question is how much program the job deserves, not which language is more capable. Python can do everything awk can.
• For a job matching awk's assumptions, awk needs less typing — one line against five, because opening, looping, splitting and converting are assumptions rather than statements.
• Every one of those assumptions is also a constraint: records and fields, whitespace, automatic conversion, one implicit loop.
• The difference that matters is noticing. On a ragged file both printed 130. awk reached it silently; Python reached it having named the unreadable row. Python makes it natural to notice and awk makes it natural not to.
• awk is loud about a missing file and exits 2 — match the exit status, not the message, because the wording differs between implementations. Python raises an exception you can catch and decide about.
• awk needs nothing to exist first, which suits a question asked at a prompt or a step inside a shell script. This is a claim about typing and scaffolding, never about speed.
• Escalate to Python for several files with per-file results, validation, joins, structured output, or code someone else maintains.
• Compare equivalents. A careful program against a careless one-liner proves nothing.

You now have the full set of comparisons: grep for lines, sed for rewriting, Python for programs, and awk for the field-aware middle ground. What you have not yet done is sit down and drive awk yourself from an empty directory.

When the answer to that trade-off is Python — a join, a second data source, a rendered report — the Python track covers it from the same engineering starting point.

That is the last chapter of this module. Chapter 1.6 — Your First awk Command is deliberately hands-on: checking that awk is there, making a file, running the command, reading the errors when it goes wrong, and ending on a small engineering summary you build yourself.