Your First awk Command

Part I · Module 1 · Chapter 1.6 · Your First awk Command

1. What You Will Do Here

By the end of this chapter you will have run roughly a dozen awk commands, seen two of them fail on purpose, and built a small summary of a synthetic cell-count file.

You need a terminal and a directory you can write files into. You do not need to have read a simulation log, written a script, or used a regular expression. If you have read Chapters 1.1 to 1.5 you will recognise some of the commands; if you arrived here directly, everything is explained from scratch.

One reminder that the rest of the chapter leans on. awk reads input one record at a time — a line, by default — and splits each record into fields on whitespace. $1 is the first field, $2 the second. That is the whole model, and Chapter 1.1 is where it is argued for properly.

2. The Shape of an awk Command

Every awk command you will write has the same three parts:

awk '{ print $2 }' cells.txt

The middle part is the one that takes getting used to, because it is a small program written inside a shell command. It has its own structure:

• The braces hold an action — something to do.
• Anything before the braces is a pattern — a condition deciding which records the action runs on. With no pattern, the action runs on every record.
• $2 inside the program means awk's second field, not anything the shell knows about.

That last point is why the quotes matter, and section 7 is entirely about it.

3. Setting Up

First check that awk is installed. It almost certainly is — awk ships with essentially every Unix-like system, including macOS and Linux.

awk --version

On the machine these examples were verified on, that prints:

awk version 20200816

Your output will probably differ, and that is fine and expected. You might see GNU Awk 5.1.0 if your system provides gawk, or a one-line BusyBox banner on a minimal container. Any of those means awk is present. Chapter 8.6 covers why the differences matter later; for now you only need it to run.

If you want to know which program you are actually getting:

which awk

/usr/bin/awk

Now make a file to work on. This is a synthetic summary of cell counts from a synthesis run — a cell type and how many of them the tool used:

DFF 120
AND 45
OR 30
MUX 18

Create it however you like — a text editor, or a shell heredoc. Four lines, two fields each, separated by a single space.

4. Your First Command

Print everything:

awk '{ print }' cells.txt

DFF 120
AND 45
OR 30
MUX 18

That looks like it did nothing, and in a sense it did nothing interesting — but a lot happened that is worth naming.

awk opened the file. It read the first line, split it into the fields DFF and 120, and ran your action. Your action was print with nothing after it, which means "print the current record". Then it did the same for the second line, the third and the fourth, and stopped at the end of the file.

You never wrote a loop. The loop is awk's, and it is the reason awk programs are short.

5. Printing Fields

Now ask for part of each record. The first field:

awk '{ print $1 }' cells.txt

DFF
AND
OR
MUX

The second field:

awk '{ print $2 }' cells.txt

120
45
30
18

Both fields, in order:

awk '{ print $1, $2 }' cells.txt

DFF 120
AND 45
OR 30
MUX 18

The comma between them is doing real work: it tells awk to put a separator — a space, by default — between the two values. Leave it out and awk joins them with nothing at all, which is occasionally what you want and usually a surprise.

Both fields, reversed, which is something no amount of grep will do for you:

awk '{ print $2, $1 }' cells.txt

120 DFF
45 AND
30 OR
18 MUX

Three more names are worth knowing now, because they appear constantly.

$0 is the whole record:

awk '{ print $0 }' cells.txt

DFF 120
AND 45
OR 30
MUX 18

That is the same output as plain print, because printing the whole record is what print does by default.

NF is the number of fields on this record:

awk '{ print NF }' cells.txt

2
2
2
2

Two fields on every line, as expected. Notice NF has no dollar sign — it is a plain variable holding a count.

$NF is the last field, because putting $ in front of a number gives you that field, and NF is the number of the last one:

awk '{ print $NF }' cells.txt

120
45
30
18

On this file $NF and $2 are the same thing. They stop being the same the moment a line has a different number of fields — which is exactly why $NF is useful: it means "the last column" without you having to count.

6. Asking for a Field That Is Not There

This file has two fields per line. Ask for a third:

awk '{ print $3 }' cells.txt

Four empty lines, no error, exit status 0.

That is worth sitting with, because it is the shape of a whole family of real bugs. awk did not complain that you asked for something that does not exist. A field that is not there is the empty string, and printing the empty string prints an empty line.

It gets quieter still in arithmetic. An empty field counts as zero:

awk '{ total += $3 } END { print total }' cells.txt

0

A total of zero from a file full of numbers, with nothing to tell you that you asked for the wrong column. Chapters 8.3 and 8.4 are about defending against this properly; what you need now is the awareness that awk's silence is not the same as agreement.

7. Why the Quotes Are Single

This is the most common way a first awk session goes wrong, and it fails quietly, which makes it worth doing deliberately.

The right way, with single quotes:

awk '{ print $2 }' cells.txt

120
45
30
18

Now the same command with double quotes:

awk "{ print $2 }" cells.txt

DFF 120
AND 45
OR 30
MUX 18

Whole lines, not the second column. No error, no warning.

Here is what happened. The shell processes your command line before awk ever sees it. Inside double quotes, the shell treats $2 as one of its own variables — the second argument to the script — which is empty. So the shell replaced $2 with nothing, and what actually reached awk was { print }. awk did exactly what it was told: print the whole record.

Inside single quotes, the shell leaves the text completely alone. awk receives { print $2 } as written, and $2 means what awk means by it.

8. Filtering Records

So far every action ran on every record. Put a condition in front of the braces and the action only runs when it is true.

Only the cells with a count above 100:

awk '$2 > 100 { print $1 }' cells.txt

DFF

Read it as English: for each record, if the second field is greater than 100, print the first field. Only DFF, at 120, qualifies.

The comparison here is numeric — awk compared the number 120 against the number 100. This is the thing Chapter 1.3 showed grep cannot do.

A text comparison instead, using == for "is exactly equal to":

awk '$1 == "MUX" { print $2 }' cells.txt

18

The quotes around MUX are needed because it is a text value. Without them awk would read MUX as a variable name, find it empty, and compare the first field against the empty string — which matches nothing, prints nothing, and reports no error. Another quiet one.

9. A Number From the Whole File

Every command so far produced one line of output per matching record. This one produces a single number from the whole file:

awk '{ total += $2 } END { print total }' cells.txt

213

Two parts:

• { total += $2 } has no pattern, so it runs on every record. total += $2 means "add the second field to total". You never created total and never set it to zero — awk does both for you on first use.
• END { print total } is special. END is not a condition about a record; it means after the last record has been read. So the printing happens once, at the end, when total holds the sum of all four.

213 is 120 + 45 + 30 + 18. Chapter 2.4 covers BEGIN and END as part of the execution model.

10. When It Goes Wrong: Reading awk's Errors

Two failures you will meet in your first hour.

A program awk cannot parse. Leave off the closing brace:

awk '{ print $2 ' cells.txt

awk refuses to run and prints a diagnostic whose first line is awk: syntax error at source line 1. Below that it shows the piece of your program it had reached, marked between >>> and <<<, and then a second line explaining what it expected — here, a missing }.

Those >>> and <<< markers are the useful part: they point at where awk got confused, which is usually just after the real mistake. Read the marker first, then look at what comes before it.

The exact wording of this message differs between awk implementations, so learn the structure rather than the string. What is reliable is that awk exits with a non-zero status — 2 on the awk used here — and prints nothing to standard output.

A file that is not there. Misspell the filename:

awk '{ print }' nosuch.txt

awk: can't open file nosuch.txt
 source line number 1

awk names the file it could not open and exits with status 2. Two details matter. The message goes to standard error, not standard output — so if you capture the command's output with $(...) or redirect it into a data file, you get an empty result and the error goes to your screen separately. And again the wording varies between implementations, so a script should test the exit status rather than match the text.

11. Exit Status, and What It Does Not Prove

This is the most important section in the chapter, and it is the one most first-time users never encounter.

Every command leaves behind an exit status: a number saying how it went. Zero conventionally means success. Your shell keeps the last one in $?.

Ask for cells with a count above 1000. There are none:

awk '$2 > 1000 { print $1 }' cells.txt
echo $?

0

Read that carefully. awk printed no cell names, because none matched — and then reported exit status 0, meaning success.

Both of those are correct. awk's job was to read the file and apply the program, and it did that perfectly. Finding no matching records is not a failure; it is an answer.

But now consider a shell script that does this:

run the awk command
if the exit status is 0, report that the check passed

That script will report a pass whether there were no violations or no matching records for any other reason — including a misspelled field number, a report whose columns moved, or a file that was legitimately empty. The exit status cannot tell those apart, because from awk's point of view they are all the same thing: a successful run that matched nothing.

The honest version of the check, which reports a number rather than relying on silence:

awk '$2 > 1000 { n++ } END { print n + 0 }' cells.txt

0

Now the zero is a measurement — awk looked and found none — rather than an absence of output that could have meant anything. The + 0 matters: without it, n was never created and print n would output an empty line instead of 0.

12. A Small Engineering Summary

Put the chapter together on cells.txt. Four questions a synthesis review might actually ask, each one command.

Which cell types are in the report:

awk '{ print $1 }' cells.txt

DFF
AND
OR
MUX

What the counts are:

awk '{ print $2 }' cells.txt

120
45
30
18

Which types exceed 40 instances:

awk '$2 > 40 { print $1, $2 }' cells.txt

DFF 120
AND 45

How many types are listed, and the sum of the counted column:

awk '{ n++; total += $2 } END { print n, total }' cells.txt

4 213

Four cell types, 213 cells counted in total. Two accumulators in a single pass over the file — n counting records and total adding the second field — which is the pattern almost every awk summary script is built from.

And the honest caveats, all of which you have now met: 213 is a sum of counts and not an area; the numbers are synthetic, so they say nothing about implementation quality; and a real synthesis report has headers, banners and more columns, so none of these commands would survive contact with one unchanged. Module 11 is where real reports are handled.

13. Exercises

Create cells.txt as shown and work through these in order.

DFF 120
AND 45
OR 30
MUX 18

Exercise 1 — Print the last field two ways

Print the count column twice: once by its position, once as the last field. Confirm the outputs match, then say when they would stop matching.

awk '{ print $2 }' cells.txt

120
45
30
18

awk '{ print $NF }' cells.txt

120
45
30
18

Identical, because every record here has exactly two fields. They would differ the moment a line had three — $2 would still be the second column while $NF followed the last one. $NF is the safer choice when you mean "the final column" and the column count is not guaranteed.

Exercise 2 — Modify a demonstrated command

Print the cell types with a count below 40.

awk '$2 < 40 { print $1 }' cells.txt

OR
MUX

One character changed from section 8. OR at 30 and MUX at 18 qualify; AND at 45 does not.

Exercise 3 — Debug a command that prints nothing

An engineer runs this and gets no output at all. The file is correct and awk reports no error. Find the bug and fix it.

awk '$1 == MUX { print $2 }' cells.txt

The problem is the missing quotes around MUX. Without them awk reads MUX as a variable, which has never been assigned and is therefore empty, so the test becomes "is the first field equal to the empty string" — true for no record here. Nothing matches, nothing prints, and awk exits 0.

awk '$1 == "MUX" { print $2 }' cells.txt

18

This is section 11's lesson in practice: silence plus a successful exit status told the engineer nothing was wrong, when something was.

Exercise 4 — An engineering summary of your own

Report, in one command, the number of cell types whose count is above 20 and the total of just those counts.

awk '$2 > 20 { n++; total += $2 } END { print n, total }' cells.txt

3 195

Three types — DFF, AND and OR — totalling 195. The pattern applies to both accumulators at once because they are in the same action, which only runs on matching records. Compare with section 12's version, where the accumulators had no pattern and so counted everything.

Exercise 5 — Challenge: make the silence speak

You need a command that reports whether any cell type exceeds 500 instances, in a way a nightly script could act on. Explain why awk '$2 > 500 { print $1 }' cells.txt is not good enough, then write something better.

It is not good enough because, on this file, it prints nothing and exits 0 — and as section 11 showed, that is indistinguishable from a misspelled field, a changed column layout or an empty file. A script testing only the exit status would report a pass in all four cases.

awk '$2 > 500 { n++ } END { print n + 0 }' cells.txt

0

Now the output is a number the script can test, and 0 means "awk looked and found none" rather than "awk said nothing". A fuller version would also check that the file had records at all — END { print NR, n + 0 } would report 4 0, proving awk read four records before finding no matches. That distinction, between a measured zero and an absent answer, is what Chapters 8.3 and 14.1 build on.

14. Summary

• An awk command has three parts: the command, the program in single quotes, and the input file.
• The loop is awk's. An action with no pattern runs on every record; you never write the loop.
• $1, $2, $0, NF, $NF. $0 is the whole record, NF is how many fields it has, $NF is the last one. NF is a count, $NF is a value.
• A field that is not there is the empty string, which is 0 in arithmetic — so asking for the wrong column gives you empty lines or a zero total, with no error.
• Single-quote the program, always. In double quotes the shell replaces $2 before awk sees it, and the command prints whole lines with no warning.
• A pattern before the braces filters records. Numeric comparison works; quote text values, or awk reads them as empty variables and matches nothing.
• END runs after the last record, which is how a per-record accumulator becomes one number.
• Read errors by their structure, not their wording. The >>> markers point at where awk got confused; the wording differs between implementations, so test the exit status.
• A zero exit status means awk ran, not that your condition held. Make awk report a number instead of relying on silence.

You have now run awk, broken it twice on purpose, read its errors, and produced a small summary with honest caveats attached. That completes Module 1: you know what awk is, where it belongs in a chip flow, how it compares with grep, sed and Python, and how to drive it.

What you have been taking on trust throughout is the order in which awk does things — why END runs last, what happens before the first record is read, and how a pattern and an action are actually evaluated. Module 2, Understanding the awk Execution Model, makes that explicit, and it is the foundation every later module depends on.