read: Taking a Value from Input

Part I · Module 8 · Chapter 8.6 · read: Taking a Value from Input

Values have come from assignments (8.1), defaults (8.2), other values (8.3), commands (8.4) and arithmetic (8.5). The last source is input — a person typing, or a file read a line at a time.

1. The Simple Form

$ read -r -p "test name: " name
test name: alu_smoke

$ echo "you entered: $name"
you entered: alu_smoke

read takes one line of input and stores it in a variable.

• -p prints a prompt first. It goes to error output rather than normal output, so it still appears on your screen when the script's normal output is being redirected to a log — which is exactly right for a prompt.
• -r is the subject of §2, and should be considered part of the command's spelling.

With no variable name, the line lands in REPLY:

$ read -r
hello
$ echo "$REPLY"
hello

2. Why -r Is Not Optional

Without -r, read treats a backslash as an escape character and removes it.

$ printf 'C:\\Users\\path\n' > back.txt

$ read line < back.txt
$ echo "[$line]"
[C:Userspath]

$ read -r line < back.txt
$ echo "[$line]"
[C:\Users\path]

The data was silently corrupted. No error, no warning — the backslashes are simply gone.

It gets worse when a line ends in a backslash, because that is read as a line continuation:

$ printf 'first\\\nsecond\nthird\n' > cont.txt

$ read line < cont.txt
$ echo "[$line]"
[firstsecond]

$ read -r line < cont.txt
$ echo "[$line]"
[first\]

Without -r, one call to read consumed two lines of the file and glued them together. A loop over that file processes the wrong number of records, and nothing anywhere reports a problem.

3. The Whitespace That Disappears

The second wrong default:

$ printf '   padded value   \n' > pad.txt

$ read -r l < pad.txt
$ echo "[$l]"
[padded value]

$ IFS= read -r l < pad.txt
$ echo "[$l]"
[   padded value   ]

read splits its input using IFS, the internal field separator, which by default is space, tab and newline — and splitting means leading and trailing whitespace is discarded.

Setting IFS to nothing for the duration of the command turns the splitting off, so the line arrives exactly as it was.

IFS= read -r line

The assignment before the command is the prefix form from Chapter 8.1: it applies to that one command only, and your shell's IFS is untouched.

4. Several Variables at Once

Give read more than one name and the line is split across them:

$ echo "alu  pass  12 extra bits" | { read -r name result rest; echo "name=[$name] result=[$result] rest=[$rest]"; }
name=[alu] result=[pass] rest=[12 extra bits]

The last variable receives everything left over, including the spaces inside it. That is useful rather than accidental: it lets you pull two fields off the front and keep the remainder of the line whole.

Too few fields and the extra variables are empty:

$ echo "a b" | { read -r x y z; echo "x=[$x] y=[$y] z=[$z]"; }
x=[a] y=[b] z=[]

Splitting on something other than whitespace

Set IFS to the separator you actually have:

while IFS=: read -r user _ uid _ _ home _; do
  echo "$user ($uid) -> $home"
done < /etc/passwd

IFS=: splits on colons; _ is a conventional name for a field you are discarding. The same technique handles comma-separated data:

while IFS=, read -r test seed result; do
  echo "$test (seed $seed): $result"
done < results.csv

Into an array

$ echo "alu pass 12" | { read -r -a f; echo "count=${#f[@]}  first=${f[0]}  third=${f[2]}"; }
count=3  first=alu  third=12

-a splits the line into an array of however many fields there are, which is what you want when the count varies. Arrays are Chapter 19.2, and Chapter 19.4 uses this exact form to split a delimited string; this is the one form worth knowing early.

5. Reading a File, Line by Line

while IFS= read -r line; do
  echo "got: $line"
done < input.txt

This is the standard shape, and two details make it work.

read returns a non-zero exit status at end of input, which is what stops the loop:

$ read -r l < /dev/null
$ echo "status at EOF: $?"
status at EOF: 1

The redirection is on the done, attaching the file to the whole loop. Which brings us to the mistake everyone makes once.

The pipe that loses your work

$ n=0
$ cat data.txt | while read -r l; do n=$(( n + 1 )); done
$ echo "after pipe:     n=[$n]"
after pipe:     n=[0]

$ n=0
$ while read -r l; do n=$(( n + 1 )); done < data.txt
$ echo "after redirect: n=[$n]"
after redirect: n=[3]

Zero, from a three-line file. The loop ran, the counter incremented three times, and then the answer vanished.

Each part of a pipeline runs in its own subshell — a child process (Chapter 7.4). The while loop on the right-hand side of the pipe was a child, so n was its own copy, and it died with the subshell. The parent's n was never touched.

This is the same one-way inheritance rule for the fourth time in two modules, and it is the most expensive instance of it, because the symptom is a variable that is empty or zero for no visible reason.

The last line that disappears

$ printf 'one\ntwo\nthree' > nonl.txt     # note: no trailing newline

$ while read -r l; do echo "  got [$l]"; done < nonl.txt
  got [one]
  got [two]

$ echo "lines read: 2 — but 'three' is in the file"

The last line is silently dropped. read returns a non-zero status when it hits end of input without a newline — and it has stored the partial line in the variable, but the loop condition has already decided to stop.

A file with no trailing newline is not exotic. It is what you get from a program that was interrupted, from some editors, and from a printf without a \n — and Chapter 4.5 showed the same cause making wc -l report one fewer line than you can see.

The fix is to continue when the status is non-zero but something was read:

$ while read -r l || [ -n "$l" ]; do echo "  got [$l]"; done < nonl.txt
  got [one]
  got [two]
  got [three]

|| [ -n "$l" ] means "or the variable is not empty". (The test syntax is Module 16; the idiom is worth adopting now.)

Combined with everything else, the fully defensive line-reading loop is:

while IFS= read -r line || [ -n "$line" ]; do
  ...
done < input.txt

Every character of that has been earned by a specific failure in this chapter.

6. The Other Flags

-t is the one that matters for anything unattended. A script that stops at a prompt in an overnight regression waits until somebody notices:

if read -r -t 10 -p "continue? [y/N] " answer; then
  echo "you said: ${answer:-N}"
else
  echo "no answer in 10 seconds — stopping"
  exit 1
fi

Note ${answer:-N} doing exactly the job Chapter 8.2 described: somebody who presses Enter without typing leaves the variable empty, and the default turns that into a definite N.

-s for a password:

read -r -s -p "password: " pw
echo            # -s swallows the newline too, so print one

7. The Chapter, and the Module, in One Script

#!/bin/bash
set -euo pipefail

results="${1:?usage: summarise.sh <results-file>}"

pass=0
fail=0

while IFS=, read -r test seed result || [ -n "$test" ]; do
  case "$result" in
    PASS) pass=$(( pass + 1 )) ;;
    *)    fail=$(( fail + 1 ))
          echo "FAILED: ${test} (seed ${seed:-unknown})" ;;
  esac
done < "$results"

total=$(( pass + fail ))
(( total > 0 )) || { echo "no results in ${results}"; exit 1; }

report="${results%.csv}.summary"
{
  echo "generated : $(date +%F' '%T)"
  echo "source    : ${results##*/}"
  echo "total     : ${total}"
  echo "passed    : ${pass}  ($(( pass * 100 / total ))%)"
  echo "failed    : ${fail}"
} > "$report"

echo "wrote ${report}"

Every chapter of the module is in there:

• 8.1 — variables, and every expansion quoted.
• 8.2 — ${1:?usage: ...} for the required argument, ${seed:-unknown} for the optional one.
• 8.3 — ${results%.csv} to build the output name, ${results##*/} for the bare file name.
• 8.4 — $(date +%F' '%T) for the timestamp.
• 8.5 — integer counting, and pass * 100 / total in the correct order.
• 8.6 — IFS=, read -r ... || [ -n "$test" ], redirected rather than piped.

case and the grouping braces around the report belong to Module 16 and Module 12. Everything else you now know.

Practice

1 — Prompt for something.

read -r -p "test name: " name
echo "you entered: [$name]"

2 — Watch -r save you.

printf 'C:\\Users\\path\n' > back.txt
read    l < back.txt; echo "plain : [$l]"
read -r l < back.txt; echo "with -r: [$l]"

3 — Watch a line swallow the next one.

printf 'first\\\nsecond\nthird\n' > cont.txt
read    l < cont.txt; echo "plain : [$l]"
read -r l < cont.txt; echo "with -r: [$l]"

4 — Watch whitespace vanish.

printf '   padded   \n' > pad.txt
read -r l < pad.txt;      echo "[$l]"
IFS= read -r l < pad.txt; echo "[$l]"

5 — Split a line.

echo "alu pass 12 extra bits" | { read -r a b c; echo "a=[$a] b=[$b] c=[$c]"; }
echo "a:b:c" | { IFS=: read -r x y z; echo "x=[$x] y=[$y] z=[$z]"; }

6 — Reproduce the subshell loss.

printf 'a\nb\nc\n' > three.txt
n=0; cat three.txt | while read -r l; do n=$(( n + 1 )); done; echo "piped:    $n"
n=0; while read -r l; do n=$(( n + 1 )); done < three.txt;     echo "redirect: $n"

7 — Lose the last line, then recover it.

printf 'one\ntwo\nthree' > nonl.txt
while read -r l; do echo "  [$l]"; done < nonl.txt
echo "---"
while read -r l || [ -n "$l" ]; do echo "  [$l]"; done < nonl.txt

8 — Time out.

read -r -t 3 -p "say something in 3 seconds: " x || echo "(nothing)"

9 — Build the whole thing. Make a small CSV and run §7's script over it. Then remove the trailing newline from the file and confirm the last result is still counted.

Check Your Understanding

1. A script reads paths from a file and some contain backslashes, which disappear. What is missing, and why is there no error?

2. Why is IFS= read -r line the canonical form rather than plain read line?

3. read -r a b on the line one two three four — what ends up in b?

4. A loop counts lines with cat f | while read -r l; do n=$(( n+1 )); done and n is 0 afterwards. Where did the count go, and what is the fix?

5. A three-line file processes as two lines. What is different about that file?

6. Why does read -p send its prompt to error output instead of normal output?

8. Remember

• read takes one line of input into a variable; with no name given, it lands in REPLY.
• read -r is the command's real spelling. Without it, backslashes are removed and a trailing backslash swallows the next line.
• IFS= stops leading and trailing whitespace being stripped.
• IFS= read -r line is the canonical form for reading a line exactly as written.
• Several variable names split the line, and the last one takes everything remaining.
• Set IFS to split on something else — IFS=: or IFS=, — for structured data. It is not a CSV parser; a quoted comma will break it.
• read -a splits into an array when the field count varies.
• read returns non-zero at end of input, which is what ends a while loop.
• Redirect into a while loop, never pipe into it. A pipeline stage is a subshell and every variable set inside it is lost.
• < <(command) puts a command's output where a file is expected, without the subshell.
• A file with no trailing newline loses its last line. || [ -n "$line" ] recovers it.
• -p prompts to error output, -s hides typing, -t times out. Use -t in anything unattended.
• Prefer arguments to prompts, and prompt only when a value is missing.

That completes Module 8. The shell now computes: it holds values, supplies defaults for the ones that are missing, takes strings apart, turns a command's output into data, counts, and reads input from a person or a file.

Every one of those produces text that then has to survive reaching a command — and you have been told six times to put quotes around it without ever being told what quotes actually do. Module 9 is that chapter: the full set of expansions the shell performs on a line, the order it performs them in, and the exact difference between a single quote, a double quote and a backslash.