Thinking in Lines and Fields

Part I · Module 13 · Chapter 13.1 · Thinking in Lines and Fields

Module 12 built the plumbing. This module is the tools that were designed to go in it.

1. One Model, Not Six Tools

$ grep -h ERROR *.log | sed 's/.*ERROR: //' | sort | uniq -c | sort -rn | head -10

Five tools nobody coordinated, chained without adapters. That works because every one of them agrees on what text is:

2. A Line Is a Record

alu,7,PASS,1200
fifo,9,FAIL,340
cache,3,PASS,98000

One line is one record. Three tests, three lines — not a block of text that happens to contain newlines.

That assumption runs through everything. wc -l counts records. head -5 takes the first five records. sort reorders records. grep keeps the records that match. When you ask "how many tests failed?", the answer is a count of lines, and that is not a coincidence — the tools were built around the expectation that a line is a unit of meaning.

3. A Line Is a Row of Fields

Within a line, a separator divides the record into fields:

alu,7,PASS,1200
 │  │  │    │
 1  2  3    4

Fields are numbered from 1, which is worth stating because almost everything else in the shell counts from zero — array indices, substring offsets, PIPESTATUS. Field numbering is the exception, and it is consistent across cut, sort, awk and join.

The separator is a choice, and the tools disagree about the default.

4. The Defaults Differ, and It Matters

This table is the practically useful part of the chapter. Every row was measured.

Two differences in that table cause most of the trouble.

cut defaults to a tab, not a space. So cut -f2 on space-separated text gives you the whole line, because there is no tab to split on and the tool sees one field:

$ printf 'a\tb\tc\n' | cut -f2
b

$ printf 'a b c\n' | cut -f2
a b c

No error. The second looks like it ignored you.

cut treats every separator as a boundary; awk treats a run as one.

$ printf 'alu    7   PASS\n' > spaced.txt

$ cut -d' ' -f2 spaced.txt
            ← empty

$ awk '{print $2}' spaced.txt
7

cut saw alu, then empty, empty, empty, then 7 — field 2 is the first of those empties. awk saw two fields with whitespace between them.

5. The Tools Are Verbs

Each tool does one thing to the stream. Knowing which verb you want is most of choosing the tool:

You have already used the last three. They belong to the same family and obey the same model — which is why they were usable in pipelines before this module explained why.

6. Reading a Pipeline

With the model in place, a long pipeline reads as a sequence of transformations of one stream:

grep -h ERROR *.log \
  | sed 's/.*ERROR: //' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -10

Each row's output is the next row's input, and each stage does exactly one thing. That is the shape to aim for when writing one, and the way to read one somebody else wrote.

Chapter 13.9 is about constructing these without getting lost.

7. What Makes Text Easy or Hard

A practical observation that will save you more time than any individual tool.

Easy text has one record per line, one separator between fields, no separator inside a field, and a consistent number of fields:

alu,7,PASS,1200
fifo,9,FAIL,340

Hard text has aligned columns, variable field counts, separators inside quoted values, headers and footers mixed with data, or records spanning lines:

Test Results Summary
====================
Test      Seed   Result   Cycles
alu          7   PASS       1200
fifo         9   FAIL        340

Total: 2 tests

That second block is designed for a person. Every feature that makes it readable — the alignment, the underline, the blank line, the summary — is an obstacle to a tool.

Practice

mkdir -p /tmp/lf && cd /tmp/lf
cat > results.csv <<'EOF'
test,seed,result,cycles
alu,7,PASS,1200
fifo,9,FAIL,340
cache,3,PASS,98000
EOF
printf 'alu    7   PASS\nfifo   9   FAIL\n' > spaced.txt
printf 'a\tb\tc\n' > tabbed.txt

1 — Confirm a line is a record.

wc -l results.csv
head -2 results.csv
tail -n +2 results.csv | wc -l

2 — Find cut's default separator.

cut -f2 tabbed.txt
cut -f2 results.csv
cut -d, -f2 results.csv

The second gives you the whole line. Work out why before reading §4 again.

3 — Watch the two separator models differ.

cut -d' ' -f2 spaced.txt
awk '{print $2}' spaced.txt

4 — Count the fields each tool sees.

awk '{print NF}' spaced.txt
awk -F' ' '{print NF}' spaced.txt
awk -F, '{print NF}' results.csv

5 — Do one job four ways.

grep PASS results.csv
awk '/PASS/' results.csv
sed -n '/PASS/p' results.csv
awk -F, '$3=="PASS"' results.csv

The last is the only one that would not also match a test called pass.

6 — Read a pipeline stage by stage.

tail -n +2 results.csv
tail -n +2 results.csv | cut -d, -f3
tail -n +2 results.csv | cut -d, -f3 | sort
tail -n +2 results.csv | cut -d, -f3 | sort | uniq -c

7 — Compare human and machine formats.

df -h | head -3
df -P | head -3

Count the separators in each.

8 — Clean up. cd /tmp && rm -rf /tmp/lf

Check Your Understanding

1. What is the shared model, and why does it matter?

2. cut -f2 data.csv returns whole lines. What is wrong?

3. cut -d' ' -f2 on space-aligned output gives nothing, while awk '{print $2}' works. Why?

4. Are fields numbered from 0 or 1?

5. Why are these tools a poor choice for JSON?

6. A tool offers both an aligned and a machine-readable output format. Which should a script use, and why does it matter beyond convenience?

8. Remember

• One model, not six tools: input is lines, a line is fields, output is lines.
• That shared shape is the only interface, which is why unrelated tools chain and why your own script joins the toolkit for free.
• A line is a record, which is why wc -l counts things and sort reorders them.
• The model breaks on JSON, XML, real CSV and wrapped prose. Know which kind of text you have.
• Fields are numbered from 1 — the exception in a shell that otherwise counts from zero.
• cut defaults to a tab; awk and sort default to whitespace. A cut -f2 that returns whole lines is this.
• cut splits on every separator; awk treats a run as one. Aligned columns need awk.
• Each tool is a verb: select, reorder, collapse, translate, substitute, accumulate, combine.
• Overlap is fine; prefer the simplest tool — and prefer one that replaces two stages, like grep -c or sort -u.
• Easy text has one record per line and one separator per field. Everything that makes output readable makes it harder to parse.
• Ask for the machine-readable format. It is a documented interface; the aligned one is presentation and will change.
• A log is the opposite problem from source: uniform, mostly noise, and you count before you search. Chapter 24.4 is that inversion.

The first verb is the simplest one, and the chapter about it is mostly about the point at which it stops being enough.