Finding and Counting Duplicates: uniq

Part I · Module 13 · Chapter 13.4 · Finding and Counting Duplicates: uniq

Chapter 13.3 brought identical lines together. This is the tool that does something with them.

1. It Only Looks Backwards One Line

$ cat dup.txt
a
b
a
c
b
a

$ uniq dup.txt
a
b
a
c
b
a

Nothing happened. Six lines in, six lines out, with three as still there.

uniq compares each line with the line immediately before it and nothing else. No two identical lines in that file are adjacent, so nothing was collapsed.

$ sort dup.txt | uniq
a
b
c

2. Counting

$ sort dup.txt | uniq -c
   3 a
   2 b
   1 c

-c prefixes each line with how many times it occurred.

The count is right-aligned in a fixed-width field, which matters for two reasons: the output lines up neatly for a human, and the leading whitespace means a later sort needs -n to handle it (Chapter 13.3 — -n skips leading blanks, a plain text sort does not).

$ sort dup.txt | uniq -c | sort -rn
   3 a
   2 b
   1 c

3. Only the Duplicates, or Only the Singletons

$ sort dup.txt | uniq -d
a
b

$ sort dup.txt | uniq -u
c

-d shows only values that appeared more than once. -u shows only values that appeared exactly once.

They partition the input, and each answers a question the other cannot:

# which test names appear twice in the list? (a configuration error)
cut -d, -f1 tests.csv | sort | uniq -d

# which modules appear in the filelist but only once? (possibly a typo)
sort filelist.f | uniq -u

-d is the one to reach for when "this should be unique and I want to know if it is not" — a duplicate entry in a filelist, a test registered twice, two config lines setting the same key.

Combining them with -c works as expected:

$ sort dup.txt | uniq -cd
   3 a
   2 b

4. Ignoring Part of the Line

Three flags narrow what counts as "the same":

-f is the useful one, because log lines usually start with something that differs on every line:

$ cat timed.log
10:01:22 disk full
10:02:04 disk full
10:05:51 link down

$ sort -k2 timed.log | uniq -f1 -c
   2 10:01:22 disk full
   1 10:05:51 link down

-f1 skipped the timestamp, so the two "disk full" lines counted as the same despite different times. Note that the surviving line keeps the first timestamp of its group — uniq prints a representative line, not a merged one.

The alternative is to remove the varying part first, which is usually clearer:

cut -d' ' -f2- timed.log | sort | uniq -c

Now the output has no misleading timestamp in it. Prefer stripping to ignoring when the result will be read by a person.

5. The Legitimate Use Without sort

There is one, and it is worth knowing so the rule in §1 does not become superstition.

Collapsing consecutive repeats is a different job from deduplication, and for that uniq alone is correct:

$ printf 'connecting\nconnecting\nconnecting\nconnected\nconnecting\n' | uniq
connecting
connected
connecting

A log where a retry prints the same line forty times in a row becomes readable, and the structure — that it reconnected afterwards — is preserved. Sorting first would destroy exactly the information you wanted.

tail -f run.log | uniq        # live log with repeat-spam collapsed

The test: are you removing duplicates, or compressing runs? The first needs sort; the second must not have it.

6. When to Use awk Instead

uniq counts lines. awk can count by field, without sorting:

$ awk -F, 'NR>1 {t[$3]++} END {for (k in t) print t[k], k}' results.csv | sort -rn
3 PASS
2 FAIL

t[$3]++ keeps a running tally in an associative array keyed by field 3 (Chapter 13.7).

Use sort | uniq -c by default. It is clearer, it is what everyone recognises, and constant memory means it never surprises you on a large input.

Reach for awk when you need to count by a field rather than a whole line, when the input is enormous and sorting it is the expensive part, or when you are already using awk in that pipeline.

7. In Practice

#!/bin/bash
set -uo pipefail

logs="${1:?usage: $0 <log-directory>}"
[ -d "$logs" ] || { echo "no such directory: $logs" >&2; exit 2; }

echo "=== most frequent errors ==="
grep -h 'ERROR' "$logs"/*.log 2>/dev/null \
  | sed 's/.*ERROR: //' \
  | sed 's/[0-9][0-9]*/N/g' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -10

echo
echo "=== tests that appear more than once in the list ==="
cut -d, -f1 -s "$logs/../tests.csv" 2>/dev/null | sort | uniq -d

One line in that pipeline is doing more work than it looks:

sed 's/[0-9][0-9]*/N/g'

Replacing every run of digits with N before counting. Without it, timeout after 1203 cycles and timeout after 1204 cycles are two distinct messages and each gets a count of 1 — so the most common error in your regression never reaches the top ten. With it they collapse to timeout after N cycles with a count of 400.

That normalisation step is what separates a useful frequency table from a useless one, and it is the thing people most often leave out. Chapter 13.6 is the tool for it.

Practice

mkdir -p /tmp/unq && cd /tmp/unq
printf 'a\nb\na\nc\nb\na\n' > dup.txt
printf '10:01:22 disk full\n10:02:04 disk full\n10:05:51 link down\n' > timed.log
printf 'connecting\nconnecting\nconnecting\nconnected\nconnecting\n' > retry.log

1 — Watch it do nothing.

uniq dup.txt | tr '\n' ' '; echo
sort dup.txt | uniq | tr '\n' ' '; echo

2 — Count, then rank.

sort dup.txt | uniq -c
echo "---"
sort dup.txt | uniq -c | sort -rn

3 — Split duplicates from singletons.

sort dup.txt | uniq -d | tr '\n' ' '; echo "  <- appeared more than once"
sort dup.txt | uniq -u | tr '\n' ' '; echo "  <- appeared exactly once"
sort dup.txt | uniq -cd

4 — Ignore a leading field.

sort -k2 timed.log | uniq -f1 -c
echo "--- or strip it first ---"
cut -d' ' -f2- timed.log | sort | uniq -c

5 — Collapse runs without sorting.

uniq retry.log
echo "--- sorted first, which destroys the structure ---"
sort retry.log | uniq

6 — Build the four-stage pipeline on real data.

ps -eo user= | sort | uniq -c | sort -rn | head -5
find /usr/bin -type f 2>/dev/null | sed 's/.*\///' | sed 's/[0-9].*//' | sort | uniq -c | sort -rn | head -5

7 — See why normalisation matters.

printf 'timeout after 1203 cycles\ntimeout after 1204 cycles\ntimeout after 1205 cycles\n' > t.log
sort t.log | uniq -c
echo "--- with digits normalised ---"
sed 's/[0-9][0-9]*/N/g' t.log | sort | uniq -c

8 — Compare with the awk approach.

sort dup.txt | uniq -c | sort -rn
awk '{t[$0]++} END {for (k in t) print t[k], k}' dup.txt | sort -rn

9 — Clean up. cd /tmp && rm -rf /tmp/unq

Check Your Understanding

1. uniq data.txt returns the file unchanged even though it clearly has duplicates. Why?

2. Why is uniq on unsorted log data worse than useless?

3. What do the two sorts in sort | uniq -c | sort -rn each accomplish?

4. What is the difference between -d and -u?

5. When is uniq without sort the correct thing to write?

6. A frequency count of error messages shows hundreds of distinct messages each with a count of 1. What step is missing?

Chapter 29.4 runs this pipeline over a regression and finds that nothing groups until the varying parts are normalised — and that the obvious normalisation eats the message.

8. Remember

• uniq compares each line only with the one before it. It collapses adjacent duplicates, not all of them.
• So it requires sorted input, and that requirement is what keeps its memory constant whatever the input size.
• Unsorted input fails silently and partially, which is worse than failing outright.
• -c counts occurrences, right-aligned — so a following sort needs -n.
• sort | uniq -c | sort -rn | head is the most reusable pipeline in the toolkit: group, count, rank, truncate.
• Both sorts are doing different jobs. The first enables uniq; the second ranks the result.
• -d shows only repeated values; -u only unique ones. -d is the duplicate-detector for filelists and config.
• -i, -f N and -s N narrow what counts as the same. Prefer stripping the varying part to ignoring it.
• uniq without sort is correct for compressing runs — a retry storm in a log — where sorting would destroy the structure.
• awk with an array counts by field and needs no sort, at the cost of memory and readability.
• Normalise before counting. Messages differing only by an embedded number produce hundreds of counts of one.

The last point needed a tool to do the normalising. Two chapters give you one: the next is the cheapest, for when the change is character-by-character.