Replacing and Squeezing Characters: tr

Part I · Module 13 · Chapter 13.5 · Replacing and Squeezing Characters: tr

Chapter 13.4 ended needing something to normalise text before counting it. There are two tools for that; this is the cheap one.

1. One Character for One Character

$ echo 'Hello World' | tr 'a-z' 'A-Z'
HELLO WORLD

tr maps every character in the first set to the character at the same position in the second. a becomes A, b becomes B, and so on through the ranges.

That is the whole model. It works character by character, across the whole stream, with no notion of lines, words, positions or patterns.

tr 'a-z' 'A-Z'          # upper-case everything
tr ',' '\t'             # commas to tabs
tr '/' '_'              # slashes to underscores
tr -d '\r'              # delete carriage returns
tr -s ' '               # squeeze runs of spaces into one

2. Deleting and Squeezing

$ echo 'a1b2c3' | tr -d '0-9'
abc

-d deletes every character in the set rather than translating it. With -d the second set is not used at all.

$ printf 'a    b\t\tc\n' | tr -s ' \t' ' '
a b c

-s squeezes runs of repeated characters into a single one. Here it collapses runs of spaces or tabs and emits a single space for each.

-s is what makes aligned output usable by cut (Chapter 13.2):

$ printf 'alu    7   PASS\n' | tr -s ' ' | cut -d' ' -f2
7

Note the two forms. tr -s ' ' with one set squeezes runs of spaces and leaves them as spaces. tr -s ' \t' ' ' with two sets translates tabs to spaces and squeezes the result — which is what you want when the alignment mixes both.

Complementing a set

$ echo 'ab1cd2' | tr -cd '0-9'
12

-c complements the first set — "every character except these". Combined with -d it reads as "delete everything that is not a digit", which is a compact way to extract numbers.

It is also the robust way to strip anything unexpected:

tr -cd '[:print:]\n' < messy.log     # keep printable characters and newlines

A log with stray control characters — common when a tool writes progress bars into a redirected file — becomes readable.

3. Character Classes

Ranges like a-z are fine for ASCII. Named classes are clearer and handle more:

$ echo 'Hello World' | tr '[:upper:]' '[:lower:]'
hello world

tr -s '[:space:]' ' '        # any run of whitespace becomes one space
tr -cd '[:alnum:]_-'         # keep only safe filename characters
tr -d '[:cntrl:]'            # strip control characters

4. The Three Jobs It Is Best At

Case normalisation

tr '[:upper:]' '[:lower:]' < input.txt

For comparing or counting values whose case is inconsistent. Chapter 8.3 noted that Bash's own ${v^^} and ${v,,} need Bash 4 and fail on the 3.2 that macOS ships — tr works everywhere, which makes it the portable answer.

Whitespace cleanup

tr -s ' '                    # collapse runs of spaces
tr -s '[:space:]' ' '        # any whitespace run becomes one space
tr '\n' ' '                  # flatten a stream onto one line
tr -s '\n'                   # remove blank lines

tr '\n' ' ' is the one to remember: it turns a column of values into a space-separated list, which is how you get a list of files onto one command line.

$ printf 'a\nb\nc\n' | tr '\n' ' '
a b c

(It leaves no trailing newline, which is why the shell prompt often appears on the same line afterwards — Chapter 4.5 met that effect.)

Line endings

tr -d '\r' < windows.txt > unix.txt

Chapter 4.7 covered why a file written on Windows breaks a script with bad interpreter: /bin/bash^M. This is the one-command fix, and it is the most common real use of tr in engineering work — filelists and configs arrive from everywhere.

$ printf 'line\r\n' | tr -d '\r' | od -c | head -1
0000000    l   i   n   e  \n

5. What It Cannot Do

The limits are sharp and worth stating explicitly, because tr looks more capable than it is.

It cannot replace a string with a string.

$ echo 'cat' | tr 'cat' 'dog'
dog

$ echo 'concat' | tr 'cat' 'dog'
dongod

The first looks like it worked and did not. tr mapped c→d, a→o, t→g individually; cat becoming dog was a coincidence of the letters lining up. The second shows what actually happened.

For string replacement, use sed:

$ echo 'concat' | sed 's/cat/dog/'
condog

It cannot use patterns. No wildcards, no regular expressions, no anchors. The sets are literal character lists.

It cannot insert. The two sets must correspond one-to-one; there is no way to turn one character into two.

It has no idea what a line is. tr will happily delete every newline in a file, which is occasionally what you want and usually a mistake.

6. When the Sets Differ in Length

$ echo 'abcde' | tr 'abcde' 'xy'
xyyyy

A short second set is padded by repeating its last character. Every one of c, d, e became y.

That is deliberate and occasionally useful — tr 'a-z' 'x' replaces every lowercase letter with x — but it means a mismatched pair of sets produces no error, just an unexpected mapping. Count them when it matters.

GNU tr has -t to truncate the first set to the length of the second instead; it is not in the BSD version.

7. In Practice

# normalise a filelist that came from Windows, with inconsistent spacing
tr -d '\r' < filelist.f | tr -s '[:space:]' '\n' | grep -v '^$' > clean.f

# make a safe filename out of a test description
echo "$description" | tr -cs '[:alnum:]' '_' | tr '[:upper:]' '[:lower:]'

# get a column of file names onto one command line
find . -name '*.sv' | tr '\n' ' '

# count distinct words in a log, case-insensitively
tr -cs '[:alpha:]' '\n' < run.log | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head

That last one is worth reading closely. tr -cs '[:alpha:]' '\n' means complement, squeeze, and translate to newline — every run of non-letters becomes a single newline, which turns prose into one word per line. Then the pipeline from Chapter 13.4 counts them.

Three flags combining into a word-splitter is tr at its best: unreadable until you know it, and then obviously right.

Practice

mkdir -p /tmp/trd && cd /tmp/trd
printf 'Hello World\n' > case.txt
printf 'alu    7   PASS\nfifo   9   FAIL\n' > spaced.txt
printf 'line one\r\nline two\r\n' > windows.txt

1 — Meet the interface quirk.

tr 'a-z' 'A-Z' case.txt
tr 'a-z' 'A-Z' < case.txt

2 — Translate, delete, squeeze.

echo 'a1b2c3' | tr -d '0-9'
printf 'a    b\t\tc\n' | tr -s ' \t' ' '
echo 'ab1cd2' | tr -cd '0-9'; echo

3 — Use classes.

tr '[:upper:]' '[:lower:]' < case.txt
tr -s '[:space:]' ' ' < spaced.txt; echo

4 — Fix line endings.

od -c windows.txt | head -2
tr -d '\r' < windows.txt | od -c | head -2

5 — Make cut work on aligned columns.

cut -d' ' -f2 spaced.txt
tr -s ' ' < spaced.txt | cut -d' ' -f2

6 — Watch the string-replacement trap.

echo 'cat'    | tr 'cat' 'dog'
echo 'concat' | tr 'cat' 'dog'
echo 'concat' | sed 's/cat/dog/'

7 — See set-length padding.

echo 'abcde' | tr 'abcde' 'xy'
echo 'abcde' | tr 'a-z' 'x'

8 — Build the word counter.

printf 'The cat sat. The CAT sat again!\n' > prose.txt
tr -cs '[:alpha:]' '\n' < prose.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn

9 — Flatten a column.

printf 'a\nb\nc\n' | tr '\n' ' '; echo

10 — Clean up. cd /tmp && rm -rf /tmp/trd

Check Your Understanding

1. tr 'a-z' 'A-Z' file.txt prints a usage message. Why?

2. What is the difference between tr -d and tr -s?

3. echo 'cat' | tr 'cat' 'dog' prints dog. Did it replace the word?

4. What does tr -cd '0-9' do, read aloud?

5. echo 'abcde' | tr 'abcde' 'xy' gives xyyyy. Why no error?

6. When should you reach for sed instead?

8. Remember

• tr maps characters to characters across the whole stream, with no notion of lines, words or patterns.
• It takes no file arguments. Redirect or pipe into it — this catches everybody once.
• -d deletes a set; -s squeezes runs; -c complements the set.
• -cd together means "delete everything except these", which is the compact way to extract or sanitise.
• Named classes are clearer than ranges — [:upper:], [:space:], [:alnum:] — and the brackets are part of the name.
• Its three best jobs are case, whitespace and line endings, all character-level problems with one-line answers.
• tr -d '\r' is the fix for a file written on Windows.
• tr -s ' ' makes aligned columns usable by cut.
• tr '\n' ' ' flattens a column onto one line.
• It cannot replace a string with a string. tr 'cat' 'dog' maps three characters individually, and the plausible output is the trap.
• A short second set is padded with its last character — no error, just an unexpected mapping.
• It is byte-oriented, so avoid it on non-ASCII text.
• If what you are replacing is longer than one character, it is a sed job.

Which is the next chapter.