Linux & Bash · Part I · Module 13 · Text Processing on the Command Line

Joining and Reshaping: paste and join

Every tool so far has transformed one stream. These two combine two of them — one by position and one by a shared key. The first is trivial and underused; the second is genuinely useful and has the nastiest failure mode in the module.

Intermediate16 min readLinuxPart IFundamentalsText Processing

Part I · Module 13 · Chapter 13.8 · Joining and Reshaping: paste and join

Seven chapters of transforming one stream. This one puts two together.

1. paste: Side by Side

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat names.txt
alu
fifo
cache

$ cat times.txt
1200
340
98000

$ paste names.txt times.txt
alu	1200
fifo	340
cache	98000

paste glues lines together by position — line 1 with line 1, line 2 with line 2 — separated by a tab.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ paste -d, names.txt times.txt
alu,1200
fifo,340
cache,98000

-d sets the separator. With several characters it cycles through them, which is occasionally useful for three or more files:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ paste -d',;' a.txt b.txt c.txt

a,b;c — comma between the first pair, semicolon between the second.

paste matches by position and checks nothing

There is no key and no validation. If the two files are in different orders, you will silently pair the wrong rows:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ paste names.txt shuffled_times.txt
alu	340
fifo	98000
cache	1200

Three correct values, three correct names, every pairing wrong — and nothing to indicate it.

paste is right when the two files were produced together, in the same order, by the same process. It is wrong for anything you received separately. Use join, which matches on a value rather than a position.

Mismatched lengths are also silent: the shorter file simply contributes empty fields once it runs out.

Transposing a column to a row

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ paste -s -d, names.txt
alu,fifo,cache

-s reads each file serially — all its lines onto one line — rather than pairing files side by side.

This is the thing people most often reinvent with tr '\n' ',', and paste -s does it better: it puts the separator between items rather than after each one, so there is no trailing comma to strip.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ printf 'a\nb\nc\n' | tr '\n' ','
a,b,c,
$ printf 'a\nb\nc\n' | paste -s -d, -
a,b,c

(The - means "read standard input", which both tools accept where a filename is expected.)

It is the natural way to build a comma-separated list for a tool that wants one:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

tests=$(cut -d, -f1 -s tests.csv | paste -s -d, -)
./run_regression.sh --tests "$tests"

2. join: On a Shared Key

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat owners.csv
alu,owner1
cache,owner3
fifo,owner2

$ cat status.csv
alu,PASS
cache,PASS
fifo,FAIL

$ join -t, owners.csv status.csv
alu,owner1,PASS
cache,owner3,PASS
fifo,owner2,FAIL

join matches lines from two files on a shared field and emits the key followed by the remaining fields of each.

-t sets the separator — and the default is whitespace, not comma, as with sort and awk (Chapter 13.1).

By default both files are joined on field 1. To use a different one:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

join -t, -1 2 -2 1 left.csv right.csv     # field 2 of the left, field 1 of the right

3. The Sorting Requirement

This is the thing to take away from the chapter.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat uns.csv
fifo,owner2
alu,owner1
cache,owner3

$ join -t, uns.csv status.csv
fifo,owner2,FAIL

$ echo "exit status: $?"
exit status: 0

One line of output where three were expected, and a successful exit status.

join works like a merge: it walks both files in step, assuming both are sorted on the key. When the left file jumps from fifo back to alu, the algorithm has already passed alu in the right file and cannot go back. Two of the three matches are simply missed.

It produces a wrong answer, not an error

This is worse than Chapter 13.4's uniq problem, because the output is partially correct. You get a table. It has the right columns and plausible rows. It is just missing most of them.

GNU join sometimes prints join: input is not sorted to the error stream — but it still exits zero and still emits its partial output, so a script redirecting errors will never notice. BSD join is quieter still.

Sort both inputs on the join field, every time:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

join -t, <(sort -t, -k1,1 left.csv) <(sort -t, -k1,1 right.csv)

Those are the process substitutions from Chapter 12.5, sorting both files without temporary files.

Two details matter in that sort:

-k1,1, not -k1 — Chapter 13.3 showed that -k1 runs to the end of the line, so two rows with the same key but different second fields sort in an order that depends on data join does not care about.
The same locale on both. Sort order is locale-dependent, so LC_ALL=C on both sides removes a difference that is very hard to see.

And the cheap assertion before trusting a result:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

sort -c -t, -k1,1 left.csv || echo "left file is not sorted" >&2

4. Unmatched Lines

By default join emits only the lines that matched. Rows present in one file and not the other vanish silently, which is its own quiet data loss.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ join -t, -a1 owners.csv status.csv

Flag	Also emits
`-a1`	Unmatched lines from the first file
`-a2`	Unmatched lines from the second
`-a1 -a2`	Both — a full outer join
`-v1`	Only the unmatched lines from the first file
`-e TEXT`	Fill missing fields with `TEXT`
`-o FORMAT`	Choose which fields to output, in what order

-v1 is the useful one for checking data:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

join -t, -v1 <(sort -t, -k1,1 expected.csv) <(sort -t, -k1,1 actual.csv)

"Which tests were expected but have no result?" — a question that comes up after every regression, and one line answers it.

Combining -a with -e and -o produces a filled table:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

join -t, -a1 -e 'MISSING' -o '0,1.2,2.2' \
  <(sort -t, -k1,1 owners.csv) <(sort -t, -k1,1 status.csv)

-o '0,1.2,2.2' means: the key, then field 2 of file 1, then field 2 of file 2. The syntax is terse and it is the point at which most people switch to awk.

5. When to Switch to awk

join has a narrow sweet spot: two files, one key, both sortable, and you want the straightforward combination. Outside that, awk is clearer.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -F, 'NR==FNR {owner[$1]=$2; next} $1 in owner {print $1 "," owner[$1] "," $2}' \
  owners.csv status.csv

That reads the first file into an array and then processes the second. Two pieces of awk vocabulary make it work:

FNR is the record number within the current file, while NR counts across all of them. So NR==FNR is true only while reading the first file.
next skips to the next line without running the rest of the program.

No sorting required, any number of keys, and the output format is whatever you write. The cost is memory — the whole first file is held — and a line that takes a moment to read.

The decision

Situation	Use
Two files produced together, same order	`paste`
A column that must become a row	`paste -s`
Two sorted files, one key, simple output	`join`
"Which of these are missing from that?"	`join -v1`
Unsorted, or several keys, or custom output	`awk`
Comparing two sorted lists for differences	`comm`

comm is the third tool in this family — it takes two sorted files and reports what is unique to each and common to both, in three columns. It has the same sorting requirement and the same silent failure, and comm -23 a b ("lines only in a") is worth knowing.

6. In Practice

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# attach the owner of each test to its result
join -t, \
  <(LC_ALL=C sort -t, -k1,1 owners.csv) \
  <(LC_ALL=C sort -t, -k1,1 status.csv) \
  > annotated.csv

# which expected tests produced no result at all
join -t, -v1 \
  <(LC_ALL=C sort -t, -k1,1 expected.csv) \
  <(LC_ALL=C sort -t, -k1,1 results.csv)

# build a single comma-separated argument from a column
tests=$(cut -d, -f1 -s tests.csv | paste -s -d, -)

# pair two measurements taken by the same loop, in order
paste -d, names.txt cycles.txt > combined.csv

# the same join, done in awk because the output format is custom
awk -F, 'NR==FNR {o[$1]=$2; next} {printf "%-10s %-10s %s\n", $1, (o[$1]?o[$1]:"-"), $2}' \
  owners.csv status.csv

The last one is worth noting: (o[$1]?o[$1]:"-") fills in a dash where there is no owner, which is join -a1 -e expressed in a way you can read.

Practice

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

mkdir -p /tmp/jn && cd /tmp/jn
printf 'alu\nfifo\ncache\n' > names.txt
printf '1200\n340\n98000\n' > times.txt
printf 'alu,owner1\ncache,owner3\nfifo,owner2\n' > owners.csv
printf 'alu,PASS\ncache,PASS\nfifo,FAIL\n' > status.csv
printf 'fifo,owner2\nalu,owner1\ncache,owner3\n' > uns.csv

1 — Glue two columns.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

paste names.txt times.txt
paste -d, names.txt times.txt

2 — Transpose a column, two ways.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

paste -s -d, names.txt
printf 'a\nb\nc\n' | tr '\n' ','; echo
printf 'a\nb\nc\n' | paste -s -d, -

Note the trailing comma on the middle one.

3 — Watch paste pair the wrong rows.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf '340\n98000\n1200\n' > shuffled.txt
paste names.txt shuffled.txt

Every value is correct and every pairing is wrong.

4 — Join on a key.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

join -t, owners.csv status.csv

5 — Break it with unsorted input.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

join -t, uns.csv status.csv
echo "  lines: $(join -t, uns.csv status.csv | wc -l | tr -d ' '), status: $?"

One line, status zero.

6 — Fix it.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

join -t, <(sort -t, -k1,1 uns.csv) <(sort -t, -k1,1 status.csv)

7 — Check before trusting.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

sort -c -t, -k1,1 uns.csv    && echo "sorted" || echo "NOT sorted"
sort -c -t, -k1,1 owners.csv && echo "sorted" || echo "NOT sorted"

8 — Find what is missing.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf 'alu,x\nfifo,x\ncache,x\nuart,x\n' > expected.csv
join -t, -v1 <(sort -t, -k1,1 expected.csv) <(sort -t, -k1,1 status.csv)

9 — Do the same join in awk.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

awk -F, 'NR==FNR {o[$1]=$2; next} $1 in o {print $1 "," o[$1] "," $2}' owners.csv status.csv

No sorting needed.

10 — Clean up. cd /tmp && rm -rf /tmp/jn

Check Your Understanding

1. What does paste match on, and what does it check?

2. Why is paste -s -d, better than tr '\n' ','?

3. join returns one row where you expected twenty, and exits zero. What is wrong?

4. Why is this failure worse than uniq on unsorted input?

5. What does join -v1 answer?

6. When should a join be written in awk instead?

Answers

1. It matches on position — line 1 with line 1, line 2 with line 2 — and it checks nothing. Two files in different orders are paired wrongly with no indication, and a shorter file silently contributes empty fields.

2. Because paste -s puts the separator between items, while tr replaces every newline including the last — leaving a trailing comma you then have to strip.

3. The inputs are not sorted on the join field. join walks both files in step assuming sorted order, so once the key goes backwards it cannot recover and silently misses the rest. Sort both with -k1,1 and the same locale.

4. Because the output is partially correct. uniq on unsorted data at least produces something obviously unchanged; join produces a real table with the right columns and plausible rows, just missing most of them. GNU join may warn on the error stream but still exits zero.

5. "Which lines in the first file have no match in the second?" — after a regression, that is "which expected tests produced no result", answered in one line.

6. When the inputs are not sorted and sorting is inconvenient, when there are several keys, when you need a custom output format, or when you want to fill in missing values readably. awk's NR==FNR idiom reads the first file into an array and needs no sort.

7. Remember

join is what makes two nights of regression results comparable — Chapter 32.4 builds the tab-separated file it joins, and Chapter 32.5 measures what a silently dropped key costs.
paste combines by position — line n with line n — and validates nothing.
-d sets the separator and cycles through several characters for three or more files.
Different orderings pair wrongly and silently. Use paste only for files produced together.
paste -s -d, turns a column into a row with no trailing separator, which tr '\n' ',' leaves behind.
join matches on a shared key, defaulting to field 1 of both files and a whitespace separator.
join requires both inputs sorted on the key, and gives a partial, plausible, wrong answer when they are not — with exit status zero.
Sort with -k1,1 and a fixed locale, and use sort -c as a cheap assertion first.
Unmatched lines are dropped by default. -a1 keeps them, -v1 shows only them.
join -v1 answers "what is missing from the other file?" — the standard post-regression question.
Switch to awk with NR==FNR for unsorted input, several keys, or a custom output format.
comm is the sibling for comparing two sorted lists, with the same sorting requirement.
comm -12 and comm -23 answer a real engineering question: Chapter 23.3 uses them to decide which of three copies of a source file is the one being compiled.
Filelists compose out of per-block lists — Chapter 26.6 measures a shared list keeping two flows in step while two copies silently diverge.

One chapter left, and it is not about a tool. Eight chapters have introduced pieces; the last is about putting several of them together without getting lost.

Where this fits

Part of the Linux & Bash curriculum.