Pipes: One Program's Output as Another's Input

Part I · Module 12 · Chapter 12.5 · Pipes

Three chapters have moved streams between a process and a file. This one connects two processes to each other.

1. The Operator

$ ps -ef | grep sim | wc -l
      4

| connects one command's stream two to the next command's stream one. No file is involved. The first command's output becomes the second's input directly.

cat run.log | grep ERROR            # only the error lines
ls -lt | head -5                    # the five newest entries
ps -ef | grep vcs | wc -l           # how many simulators are running

This is the construct the Unix toolkit exists for. Each tool does one thing to a stream of text, and the pipe is what lets you build the thing you actually wanted out of them. Module 13 is a whole module of tools designed to be pipeline stages.

2. The Stages Run at the Same Time

Most people picture a pipeline as a sequence: the first command finishes, its output is handed over, the second starts. That is not what happens. Every stage starts immediately and they all run concurrently, with the pipe acting as a small buffer between them.

Measured, with timestamps:

$ produce() { for i in 1 2 3; do echo "[$(date +%S)] produced $i"; sleep 1; done; }
$ consume() { while read -r l; do echo "$l -> consumed at $(date +%S)"; done; }

$ produce | consume
[39] produced 1 -> consumed at 39
[40] produced 2 -> consumed at 40
[41] produced 3 -> consumed at 41

Each line was consumed in the same second it was produced. The consumer did not wait for the producer to finish — it was running the whole time, blocking until data arrived and processing it as it came.

The buffer is finite — typically 64 kilobytes. When it fills, the writer blocks until the reader catches up; when it empties, the reader blocks until the writer produces more. The two ends throttle each other automatically, which is why a fast producer feeding a slow consumer does not consume unbounded memory.

3. Only Ordinary Output Travels

A pipe carries stream two and nothing else. Error output goes straight to the terminal:

$ { echo "normal"; echo "error" >&2; } | sed 's/^/  piped: /'
error
  piped: normal

Only normal was transformed. The error bypassed sed entirely and arrived unmodified.

This is the composability point from Chapter 12.1, working:

make | grep -i warning

Build output is filtered; compiler errors still appear on your screen in full. If both streams went down the pipe you would see only the lines matching "warning" and the actual error would be filtered away.

To send both:

$ { echo "normal"; echo "error" >&2; } 2>&1 | sed 's/^/  piped: /'
  piped: normal
  piped: error

2>&1 must come before the |. The ordering rule from Chapter 12.3 applies unchanged — the redirect belongs to the command on the left of the pipe, and it has to happen before the pipe is set up.

make 2>&1 | tee build.log        # both streams into tee
make | tee build.log 2>&1        # WRONG — redirects tee's own output

4. Each Stage Is a Subshell

Chapter 11.4 listed pipelines among the constructs that create subshells without parentheses. This is where it costs you:

$ n=0; cat data.txt | while read -r l; do n=$(( n + 1 )); done; echo "after pipe:     n=[$n]"
after pipe:     n=[0]

$ n=0; while read -r l; do n=$(( n + 1 )); done < data.txt;  echo "after redirect: n=[$n]"
after redirect: n=[3]

The loop on the right of the pipe ran in a child process, counted correctly, and discarded the answer when it exited.

Redirect instead of piping whenever a loop needs to set variables, as Chapter 8.6 and Chapter 12.4 both showed. When the input genuinely comes from a command, use process substitution:

while IFS= read -r line; do
  n=$(( n + 1 ))
done < <(grep ERROR *.log)

< <(command) presents a command's output where a file is expected. The space between the two < matters — without it you have a here-document. The loop is not part of a pipeline, so no subshell, so the counter survives.

5. When the Reader Stops Early

$ yes | head -3
y
y
y

yes prints y forever. head -3 takes three lines and exits. So why does yes stop?

When the reading end of a pipe closes, the writer receives SIGPIPE (Chapter 7.6), whose default action is to terminate. head exited, the pipe closed, and yes was killed by the kernel on its next write.

This is a feature, and it is how | head is efficient rather than wasteful:

grep ERROR enormous.log | head -5

grep stops as soon as head has its five lines. It does not read the remaining four gigabytes.

The status is worth knowing:

$ yes | head -3 >/dev/null; echo "  status: $?"
  status: 0

$ ( set -o pipefail; yes | head -3 >/dev/null ); echo "  with pipefail: $?"
  with pipefail: 141

141 is 128 + 13, and signal 13 is SIGPIPE. With pipefail on, a pipeline whose first stage was killed this way reports failure — even though nothing went wrong. It is the one case where pipefail produces a misleading answer, and Chapter 12.8 covers what to do about it.

6. Building a Pipeline

The practical skill is constructing one incrementally rather than writing it all and debugging it.

Start with the first stage and look at the output:

$ ps -ef | head -3

Add one stage, look again:

$ ps -ef | grep vcs | head -3

Keep going until it does what you want. At every point you can see what the next stage will receive, which is far easier than reasoning about a six-stage pipeline that produced nothing.

A real one, built up:

grep ERROR *.log                            # every error line
grep -h ERROR *.log                         # without the filename prefix
grep -h ERROR *.log | sed 's/.*ERROR: //'   # just the message
grep -h ERROR *.log | sed 's/.*ERROR: //' | sort | uniq -c | sort -rn

The last line gives the distinct error messages with counts, most common first. Five tools, each doing one thing. (Module 13 teaches them properly; the shape is this chapter's.)

7. Ordering and Efficiency

Stages run concurrently, so a pipeline is roughly as slow as its slowest stage rather than the sum of all of them. But what each stage receives depends on the order, and that changes how much work there is:

sort enormous.log | grep ERROR     # sorts four gigabytes, then filters
grep ERROR enormous.log | sort     # filters first, sorts a few hundred lines

Both give the same lines in the same order. The second does a fraction of the work.

Filter early. Put grep, head and anything else that reduces the stream as close to the front as it can correctly go. A sort or a uniq that has to see everything belongs after the filtering, not before.

8. A Realistic Pipeline

Finding which tests failed most often across a regression:

grep -l 'TEST FAILED' runs/*/sim.log \
  | sed 's|runs/||; s|/sim.log||' \
  | sed 's/_seed[0-9]*$//' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -10

Read it as six transformations of one stream:

1. grep -l — the names of logs containing a failure, not the lines.
2. sed — strip the directory and the filename, leaving the run name.
3. sed — strip the seed suffix, so the same test with different seeds collapses together.
4. sort — group identical names, because uniq needs them adjacent.
5. uniq -c — count each distinct name.
6. sort -rn — most failures first.

Each line is one idea, and the backslashes from Chapter 9.3 keep it readable. Built one stage at a time, checking the output at each step, this takes two minutes. Written all at once it takes twenty.

Practice

mkdir -p /tmp/pipes && cd /tmp/pipes
printf 'a\nb\na\nc\nb\na\n' > letters.txt
printf 'x\ny\nz\n' > three.txt

1 — Build one up.

ps -ef | head -3
ps -ef | grep bash | head -3
ps -ef | grep bash | wc -l

2 — Prove the stages run concurrently.

produce() { for i in 1 2 3; do echo "[$(date +%S)] produced $i"; sleep 1; done; }
consume() { while read -r l; do echo "$l -> consumed at $(date +%S)"; done; }
produce | consume

3 — Prove only one stream travels.

{ echo normal; echo error >&2; } | sed 's/^/  piped: /'
{ echo normal; echo error >&2; } 2>&1 | sed 's/^/  piped: /'

4 — Lose a counter, then rescue it.

n=0; cat three.txt | while read -r l; do n=$(( n + 1 )); done; echo "piped:    $n"
n=0; while read -r l; do n=$(( n + 1 )); done < three.txt;     echo "redirect: $n"
n=0; while read -r l; do n=$(( n + 1 )); done < <(cat three.txt); echo "procsub:  $n"

5 — Watch SIGPIPE.

yes | head -3
yes | head -3 >/dev/null; echo "  status: $?"
( set -o pipefail; yes | head -3 >/dev/null ); echo "  pipefail: $?"

6 — Catch the uniq trap.

uniq       < letters.txt
sort       < letters.txt | uniq
sort -u    < letters.txt
sort       < letters.txt | uniq -c | sort -rn

7 — See why order matters.

time ( sort letters.txt | grep a > /dev/null )
time ( grep a letters.txt | sort > /dev/null )

The file is tiny so the times will match; read the two commands and say which does less work on a four-gigabyte file.

8 — Remove a useless cat.

cat letters.txt | grep a | wc -l
grep -c a letters.txt

9 — Clean up. cd /tmp && rm -rf /tmp/pipes

Check Your Understanding

1. Does the first stage of a pipeline finish before the second starts?

2. make | grep -i error shows nothing, yet the build clearly failed. Why?

3. How do you send both streams into a pipeline, and where does the operator go?

4. cat f | while read -r l; do n=$((n+1)); done leaves n at zero. Why, and what are two fixes?

5. Why does yes | head -3 terminate rather than running forever?

6. sort huge.log | grep ERROR and grep ERROR huge.log | sort give the same answer. Which is better?

9. Remember

• | connects one command's stream two to the next command's stream one, with no file involved.
• All stages run concurrently, with the pipe as a buffer — not one after another.
• So a pipeline can process input larger than memory, and produce output as it goes.
• The buffer is finite and the ends throttle each other, so a fast producer cannot run away.
• Only stream two travels. Errors bypass the pipe and reach your terminal, which is the point.
• 2>&1 | sends both, and the redirect goes before the pipe. |& is Bash 4 only.
• Every stage is a subshell, so a loop on the right of a pipe loses its variables.
• Redirect, or use < <(command), when a loop must set variables. The space between the two < matters.
• A closed reader sends SIGPIPE to the writer, which is why | head stops early rather than wasting work.
• pipefail turns that into status 141 — the one case where it reports a failure that is not one.
• Build pipelines one stage at a time, looking at the output after each.
• Filter early. grep before sort, not after.
• uniq only removes adjacent duplicates. sort | uniq or sort -u.
• cat file | is usually a wasted process, though cat several_files | genuinely earns its place.

A pipeline shows you the result and keeps nothing. The next chapter is the tool that lets you watch a long run and have a log of it afterwards, without choosing between the two.