Redirecting Input, and Here Documents

Part I · Module 12 · Chapter 12.4 · Redirecting Input, and Here Documents

Chapter 12.1 said entry 0 is where a command reads from, and that it points at your keyboard by default. This chapter points it somewhere else.

1. Reading From a File

$ sort < names.txt
alpha
bravo
charlie

< makes the command read the file instead of the keyboard. 0< is the explicit spelling, and nobody writes it, for the same reason nobody writes 1>.

For most commands this looks identical to passing a filename:

$ sort names.txt
alpha
bravo
charlie

Identical output, genuinely different mechanism — and the difference shows the moment a command says anything about where its data came from.

2. The Difference It Makes

$ wc -l names.txt
       3 names.txt

$ wc -l < names.txt
       3

Given a filename, wc knows the name and prints it. Reading from stream one, there is no name to print — the data simply arrives, and wc has no way to ask where from.

Chapter 8.4 hit this when capturing a line count:

count=$(wc -l data.txt)      # "      42 data.txt" — useless as a number
count=$(wc -l < data.txt)    # "      42"          — usable

There is a second, subtler difference: with <, the shell opens the file, not the command. So the shell reports the error:

$ wc -l < nofile
bash: nofile: No such file or directory

$ wc -l nofile
wc: nofile: No such file or directory

Two different programs complaining. It matters when you are reading a log and wondering which component failed, and it matters for permissions — the shell opens the file as you, which is the same thing the command would have done, so in practice the outcome is the same and only the message differs.

3. Empty Input

/dev/null is a file that is always empty (Chapter 12.7 covers it properly). Reading from it gives a command immediate end-of-input:

$ wc -l < /dev/null
       0

That is more useful than it looks. A command that reads stream one and finds a terminal will wait forever — and in an unattended job, forever means until somebody notices:

# in a cron job or a regression harness
./possibly_interactive_tool < /dev/null

Now any attempt to read input gets end-of-file immediately and the tool either proceeds with defaults or exits, rather than hanging.

4. Writing the Input In Place

Often the input is a few lines that belong with the script rather than in a separate file. A here document supplies them inline:

$ sort <<EOF
charlie
alpha
bravo
EOF
alpha
bravo
charlie

<<WORD reads everything up to a line containing exactly WORD and feeds it to the command as stream one. The word is yours to choose — EOF is conventional, and anything that cannot appear in the text works.

Chapter 9.4 introduced here documents as a quoting tool, because the delimiter decides expansion. Here they are as a redirection tool, which is what they structurally are. The expansion rule is worth restating because it is the thing people get wrong:

$ v=world
$ cat <<EOF
unquoted: $v and $(echo sub)
EOF
unquoted: world and sub

$ cat <<'EOF'
quoted:   $v and $(echo sub)
EOF
quoted:   $v and $(echo sub)

Quote the delimiter when the body is code. An unquoted delimiter in a block that generates a script expands its variables now, when they are empty, rather than leaving them for later.

Indenting

<<- strips leading tabs so the body can be indented with the code around it:

if [ "$mode" = "batch" ]; then
	cat <<-EOF
	running in batch mode
	log: $logfile
	EOF
fi

Tabs only — spaces are never stripped, which is a reliable source of confusion in an editor that converts one to the other.

5. A Single Line

$ IFS=: read -r a b <<< "x:y"
$ echo "a=[$a] b=[$b]"
a=[x] b=[y]

<<< feeds one string as input. It is the natural partner for read from Chapter 8.6, and it has a property that matters:

$ echo "a:b" | { IFS=: read -r x y; echo "piped:  x=$x"; }
piped:  x=a
$ IFS=: read -r x y <<< "a:b";      echo "string: x=$x"
string: x=a

Both work here, but the here-string creates no subshell while the pipe does. Chapter 11.4 showed what that costs — a variable set inside a piped loop is lost. With <<<, the read happens in your own shell and the variables survive.

# counts correctly
while IFS=, read -r name result; do
  [ "$result" = PASS ] && pass=$(( pass + 1 ))
done <<< "$csv_data"

6. Combining Input and Output

Redirects for different streams combine freely:

sort < names.txt > sorted.txt
./tool < input.dat > output.dat 2> errors.log

Order among different streams does not matter — < in > out and > out < in are identical, because each names its own entry in the table. Order only matters when one redirect refers to another, which is 2>&1 from Chapter 12.3.

And the group form from Chapter 11.4 works on input too:

{
  read -r header
  while IFS=, read -r a b c; do
    echo "$a -> $c"
  done
} < results.csv

One < on the whole block, so the read of the header line and the loop that follows share a single stream — the loop picks up where the header read left off. That is the standard way to skip a header row, and it only works because both reads are drawing from the same redirect.

7. Putting It Together

A script that reads a test list, runs each entry, and writes a report:

#!/bin/bash
set -uo pipefail

testlist="${1:?usage: $0 <testlist.csv>}"
[ -r "$testlist" ] || { echo "cannot read $testlist" >&2; exit 2; }

pass=0; fail=0

# One redirect on the whole block: the header read and the loop share a stream.
{
  read -r _header
  while IFS=, read -r name seed || [ -n "$name" ]; do
    [ -n "$name" ] || continue
    if ./run_sim.sh "$name" "$seed" < /dev/null > "logs/$name.log" 2>&1; then
      pass=$(( pass + 1 ))
    else
      fail=$(( fail + 1 ))
      echo "FAILED: $name (seed $seed)" >&2
    fi
  done
} < "$testlist"

# A here document for the report: unquoted, because the values must be filled in.
cat > summary.txt <<EOF
generated : $(date +%F' '%T)
testlist  : ${testlist##*/}
passed    : $pass
failed    : $fail
EOF

echo "wrote summary.txt"
[ "$fail" -eq 0 ]

Five decisions worth naming:

• < "$testlist" on the closing brace, so the header read and the loop draw from one stream.
• || [ -n "$name" ] rescues a final line with no newline (Chapter 8.6).
• < /dev/null on the simulation, so a tool that asks a question cannot hang the whole list.
• > "logs/$name.log" 2>&1 for one complete log per test.
• An unquoted heredoc delimiter for the report, because it is data with values substituted — the opposite choice from generating a script.

Practice

mkdir -p /tmp/inp && cd /tmp/inp
printf 'charlie\nalpha\nbravo\n' > names.txt
printf 'name,seed\nalu,7\nfifo,9\n' > tests.csv

1 — Read from a file two ways.

sort < names.txt
sort names.txt

2 — See what the command knows.

wc -l names.txt
wc -l < names.txt

3 — See who reports a missing file.

wc -l < nofile
wc -l nofile

4 — Give a command empty input.

wc -l < /dev/null

Then run bare wc -l, notice it waiting, and press Ctrl-D.

5 — Compare the two heredoc forms.

v=world
cat <<EOF
unquoted: $v and $(echo sub)
EOF
cat <<'EOF'
quoted:   $v and $(echo sub)
EOF

6 — Use a here-string with read.

IFS=: read -r a b <<< "x:y"; echo "a=[$a] b=[$b]"

7 — Skip a header with one redirect on a block.

{
  read -r header
  echo "header was: $header"
  while IFS=, read -r name seed; do echo "  $name (seed $seed)"; done
} < tests.csv

8 — Create the infinite loop, carefully. Run this with a hard limit so it cannot run away:

n=0
while IFS= read -r line < names.txt; do
  n=$(( n + 1 )); echo "  iteration $n saw: $line"
  [ "$n" -ge 3 ] && break
done

Three iterations, the same line every time. Now move the redirect to the done and watch it behave.

9 — Clean up. cd /tmp && rm -rf /tmp/inp

Check Your Understanding

1. wc -l f.txt and wc -l < f.txt print different things. Why?

2. An overnight job produced nothing and was still running in the morning. What is one likely cause and the one-flag fix?

3. When should a here document's delimiter be quoted?

4. What does <<- strip, and what does it not?

5. Why does while read -r line < f; do …; done loop forever?

6. Why is read … <<< "$s" better than echo "$s" | read …?

8. Remember

• < makes a command read a file instead of the keyboard. 0< is the explicit spelling nobody writes.
• With < there is no filename for the command to print — which is why wc -l < f gives a bare number.
• With < the shell opens the file, so the shell reports a missing one.
• < /dev/null gives immediate end-of-input and is the standard guard against an unattended job hanging on a prompt.
• <<WORD is a here document — everything up to a line containing exactly WORD.
• An unquoted delimiter expands; a quoted one does not. Quote it when the body is code.
• <<- strips leading tabs only, never spaces.
• <<< feeds one string and creates no subshell, unlike a pipe — so read keeps its variables.
• Redirects for different streams combine in any order. Only 2>&1 cares, because it refers to another entry.
• One < on a { …; } block is how you skip a header row and keep reading from the same stream.
• A loop's redirect goes on the done. On the read it reopens the file every iteration and never terminates.

Input and output are now both under control, one file at a time. The next chapter connects two commands directly, with no file in between — which is the construct the whole of Unix tooling is built around.