The Subshell Trap: Why a Loop in a Pipeline Loses Its Variables

Part I · Module 17 · Chapter 17.7 · The Subshell Trap

Chapters 17.2 and 17.5 both told you not to pipe into a loop and deferred the reason. Here it is.

1. The Symptom

$ printf 'a\nb\nc\n' > three.txt
$ n=0
$ cat three.txt | while IFS= read -r l; do n=$(( n + 1 )); done
$ echo "n=$n"
n=0

Three lines of input, a counter that is incremented once per line, and a final value of zero.

The first thing everyone does is check whether the increment runs. It does:

$ n=0
$ cat three.txt | while IFS= read -r l; do n=$(( n + 1 )); echo "  inside: n=$n"; done
  inside: n=1
  inside: n=2
  inside: n=3
$ echo "after: n=$n"
after: n=0

1, 2, 3, then 0. The arithmetic is right, the loop ran the right number of times, and the value is gone.

2. The Cause

Chapter 11.4 established it and this is where it bites: every stage of a pipeline runs in its own child process.

A child process gets a copy of the parent's variables. It can change its copy as much as it likes; the parent's is untouched, because that is what separate processes means. The loop above incremented a variable in a child, printed it from the child, and then the child exited — taking its copy with it. The n that echo printed afterwards was the parent's, which nothing ever modified.

Two measurements nail the cause down, and most write-ups skip both.

It is not specific to the right-hand side. Both ends of a pipe are children:

$ x=outer; echo hi | { x=inner; };           echo "right side: x=$x"
right side: x=outer
$ x=outer; { x=inner; echo hi; } | cat > /dev/null; echo "left side:  x=$x"
left side:  x=outer

And it is not the braces. Give the same block a redirection instead of a pipe and the assignment sticks:

$ x=outer; { x=inner; } < /dev/null; echo "redirect:   x=$x"
redirect:   x=inner

So the braces are innocent and the pipe is the whole cause. { ...; } groups commands in the current shell — Chapter 11.4's distinction between { } and ( ) — and a redirection attaches a file descriptor without forking anything.

3. It Is Not Only Variables

This is where the trap stops being an annoyance and becomes a correctness bug.

A directory change is lost:

$ cd /tmp
$ echo x | while read -r l; do cd /; done
$ echo "pwd=$PWD"
pwd=/tmp

And exit does not exit your script:

$ echo x | while read -r l; do echo "about to exit"; exit 9; done
about to exit
$ echo "STILL RUNNING, pipeline status=$?"
STILL RUNNING, pipeline status=9

Read that again. The loop ran exit 9. The script kept going. exit ends the process it runs in, and the process it ran in was the subshell — so the subshell exited with 9, which became the pipeline's status, and the parent shell carried on to the next line.

Compare the redirect-fed version, where exit means what you think:

$ bash -c 'while read -r l; do echo "about to exit"; exit 9; done < three.txt; echo "NOT REACHED"'
about to exit
$ echo $?
9

NOT REACHED never printed, and the script's status is 9.

4. The Three Fixes

All three measured on the same loop and input, all three giving 3:

Redirect a file. The simplest, when the input is a file:

$ n=0; while IFS= read -r l; do n=$(( n + 1 )); done < three.txt; echo "n=$n"
n=3

Process substitution, when the input comes from a command:

$ n=0; while IFS= read -r l; do n=$(( n + 1 )); done < <(cat three.txt); echo "n=$n"
n=3

< <(cmd) runs cmd in a child — it has to — but the loop stays in your shell, which is the half that matters. Note the space between < and <(: < <( is a redirection whose source is a process substitution, and <<( is a syntax error.

This is the general-purpose answer. Anywhere you would have written cmd | while read, write:

while IFS= read -r line; do ...; done < <(cmd)

A here-string, when the output is small:

$ n=0; while IFS= read -r l; do n=$(( n + 1 )); done <<< "$(cat three.txt)"; echo "n=$n"
n=3

<<< is Chapter 12.4's here-string. It buffers the entire output into memory first, so it is wrong for a large stream, but for a handful of lines it is the shortest correct form — and it is what Chapter 17.5's read -r name seed <<< "$line" was doing.

5. When You Genuinely Have a Pipeline

Sometimes the data really does arrive through a pipe you want — several transformations chained, the way Module 13 taught. You have two honest options.

Restructure so the pipeline feeds a redirection:

while IFS= read -r l; do
  n=$(( n + 1 ))
done < <(grep UVM_ERROR run.log | sort -u)

The whole pipeline sits inside the process substitution. It still runs in children — it always did — but the loop does not.

Or send the answer out of the subshell as output and capture it:

$ n=$(cat three.txt | { c=0; while read -r l; do c=$(( c + 1 )); done; echo "$c"; })
$ echo "n=$n"
n=3

The subshell cannot hand you a variable, but it can hand you output, and command substitution (Chapter 8.4) catches it. This is worth knowing because it generalises — it is how you get anything out of any subshell — but for a count specifically there is usually a tool that already does it:

$ n=$(cat three.txt | wc -l | tr -d ' ')
$ echo "n=$n"
n=3

6. A Complete Script

A log summariser that must report totals — exactly the shape that gets this wrong:

#!/usr/bin/env bash
# summarise-tree.sh — count errors across every log in a tree
# usage: summarise-tree.sh <dir>
#
# exit: 0 no errors found · 1 errors found · 2 unusable arguments

set -o nounset

dir=${1:-}

if [ -z "$dir" ]; then
  echo "usage: ${0##*/} <dir>" >&2
  exit 2
fi
if ! [ -d "$dir" ] || ! [ -r "$dir" ]; then
  echo "${0##*/}: $dir is not a readable directory" >&2
  exit 2
fi

logs=0
empty=0
errors=0
worst=""
worst_n=0

# < <(find ... -print0) — the loop stays in this shell, so the five
# counters above are still set when the loop ends. A pipe would lose
# all of them, and the exit below would not exit.
while IFS= read -r -d '' f; do

  [ -f "$f" ] || continue
  logs=$(( logs + 1 ))

  if ! [ -s "$f" ]; then
    empty=$(( empty + 1 ))
    continue
  fi

  if ! [ -r "$f" ]; then
    printf '%s: not readable: %s\n' "${0##*/}" "$f" >&2
    exit 2                      # really exits — the loop is not in a subshell
  fi

  n=$(grep -c 'UVM_ERROR' "$f")
  errors=$(( errors + n ))

  if (( n > worst_n )); then
    worst_n=$n
    worst=$f
  fi
done < <(find "$dir" -name '*.log' -print0)

printf 'logs      %d\n' "$logs"
printf 'empty     %d\n' "$empty"
printf 'errors    %d\n' "$errors"
if [ -n "$worst" ]; then
  printf 'worst     %s (%d)\n' "$worst" "$worst_n"
fi

[ "$errors" -eq 0 ]

$ mkdir -p tree/sub
$ printf 'UVM_ERROR : a\nUVM_ERROR : b\n' > tree/one.log
$ printf 'TEST PASSED\n'                  > tree/two.log
$ printf 'UVM_ERROR : c\n'                > tree/sub/three.log
$ : >                                       tree/sub/empty.log

$ ./summarise-tree.sh tree
logs      4
empty     1
errors    3
worst     tree/one.log (2)
$ echo $?
1

$ rm tree/one.log tree/sub/three.log
$ ./summarise-tree.sh tree
logs      2
empty     1
errors    0
$ echo $?
0

Every number there would be zero if the loop were fed by a pipe. Three details:

< <(find ... -print0) is the two lessons of this module in one construct: -print0 with read -d '' because a file name can contain a newline (Chapter 17.2), and process substitution rather than a pipe because five counters have to survive (this chapter).

exit 2 inside the loop really exits, and the comment says so. In the piped version it would print the message, exit the subshell, and let the script print a confident summary of a tree it failed to read.

worst and worst_n carry state across iterations — the running maximum. That is the clearest possible demonstration of why the subshell matters: a reduction over the input is impossible if the accumulator dies when the loop does.

Practice

1. Reproduce the symptom, then add echo "inside: $n" to the body. Watch it count 1, 2, 3 and then read 0. Explain in one sentence where those three values went.
2. Run both echo hi | { x=inner; } and { x=inner; echo hi; } | cat > /dev/null, printing $x. What do the two results together prove about which side of a pipe is special?
3. Run { x=inner; } < /dev/null and print $x. What does this rule out as the cause?
4. Run the exit 9 experiment both ways — piped and redirect-fed. Write down which line prints and what $? is in each case.
5. Take the broken validation loop from the §3 callout and run it on a directory with an unreadable file. Confirm it reaches the line after the loop. Then fix it with one character's worth of change in structure and confirm it does not.
6. Convert cat three.txt | while read -r l; do ...; done to all three fixes and verify each gives 3. Then try <<( instead of < <( and report the error.
7. Try shopt -s lastpipe on your machine. If it fails, say what that means for a script that relies on it.
8. Get a count out of a subshell by echo-ing it and capturing with $( ). Then do the same thing with wc -l and say which you would ship.
9. Run false | true; echo "$? ${PIPESTATUS[*]}". Explain both numbers.
10. Change summarise-tree.sh to use find ... | while read and run it on the tree. Report every number that changes and say which single line of output proves the loop ran at all.

Check Your Understanding

1. A counter prints 1, 2, 3 inside a piped loop and 0 after it. Where did the increments go?
2. Which side of a pipe runs in a subshell?
3. { x=inner; } < /dev/null sets x. What does that tell you about { }?
4. What does exit 2 do inside a loop on the right of a pipe?
5. Name the three portable fixes.
6. Why is lastpipe not one of them?
7. The input is a multi-stage pipeline you want to keep. What do you do?

7. Remember

• Every stage of a pipeline runs in its own child process, and a child gets a copy of your variables.
• So a loop fed by a pipe sets variables that die when it ends. The increments are real; they happen somewhere else.
• Both sides of the pipe are subshells, not just the right-hand one.
• A redirection is not a subshell. { x=1; } < file sets x; the braces were never the problem.
• It is not only variables. A cd is lost, and exit exits the subshell, not your script — so a guard inside a piped loop does not stop anything.
• Three portable fixes: < file, < <(cmd), <<< "$(cmd)". The middle one is the general answer.
• < <( needs the space. <<( is a syntax error.
• lastpipe is Bash 4.2+ and absent on macOS, and needs job control off. Do not build on it.
• Keep a real pipeline by putting it inside the process substitution, so only the loop stays in your shell.
• A subshell can return output even though it cannot return a variable. n=$( ... echo "$c" ) is the general escape hatch.
• $? on a pipeline is the last stage's. PIPESTATUS has them all.

8. The Module, in One Page

Seven chapters, and they reduce to two sentences and their consequences.

A for loop walks a finished list that expansion already produced; a while loop re-runs a command and reads its status. Everything else is downstream:

• The list is frozen before iteration one (17.1), which is why a loop cannot change its own list, why {1..$n} is impossible, and why an unmatched glob runs the body once on a pattern.
• A file name is not a word (17.2). for f in $(ls) turns two files into three iterations, a file called -n disappears into echo, and a newline in a name defeats every text stream — so loop over globs, write ./*, and use find -print0 with read -d ''.
• When the number is the point, use arithmetic (17.3) — and i=$(( i + 1 )) rather than let, which dies silently under set -e.
• When you do not know the count, use while (17.4) — and never poll without a pause, which measured 107,635 pointless tests and 1.36 seconds of CPU against 9 tests and 0.07.
• while IFS= read -r line || [ -n "$line" ] (17.5), because a bare read trims your whitespace, joins two lines when one ends in a backslash, and silently drops a last line nobody terminated.
• break controls flow but carries no meaning (17.6) — a loop that broke reports success, so an outcome has to live in a variable.
• And a pipe into a loop puts the loop in another process (17.7), where its variables, its cd and even its exit do not reach you.

A script can now decide and repeat. What it cannot do is be read by anyone — including you, in six months. Every complete script in this module has been one flat file, and the longer ones are at the point where a reader has to hold too much in their head at once.

Module 18 is functions: naming a block of work, giving it its own arguments and its own return, keeping its variables out of everyone else's way with local, and splitting a script into pieces you can test separately. It is also the proper answer to two things this module worked around — break 3, and getting a value out of a block.