Reading a File Line by Line

Part I · Module 17 · Chapter 17.5 · Reading a File Line by Line

A testlist, a filelist, a results table, a log. Shell scripts spend most of their lives walking lines of text, and there is exactly one correct way to do it:

while IFS= read -r line; do
  ...
done < input.txt

Four tokens that most people copy without knowing what any of them do. Here is what each one prevents.

1. Why read Works as a Condition

Chapter 16.7 named this and deferred it. read reads one line into a variable and reports whether it got one:

$ printf 'a\nb\n' > two.txt
$ exec 3< two.txt
$ read -r x <&3; echo "x=$x rc=$?"
x=a rc=0
$ read -r x <&3; echo "x=$x rc=$?"
x=b rc=0
$ read -r x <&3; echo "x=$x rc=$?"
x= rc=1

Zero while there was a line, non-zero at end of input. So while read means keep going while there is more — Chapter 17.4's loop with Chapter 16.7's command-as-condition, and nothing new.

2. IFS= — Or Your Whitespace Is Gone

$ printf '   indented\ttabbed   \nplain\n' > sp.txt

$ while read -r l; do printf '[%s]\n' "$l"; done < sp.txt
[indented	tabbed]
[plain]

$ while IFS= read -r l; do printf '[%s]\n' "$l"; done < sp.txt
[   indented	tabbed   ]
[plain]

Three leading spaces and three trailing spaces, silently removed.

read splits the line into fields using IFS — Chapter 9.2's internal field separator, which by default is space, tab and newline — and assigns the fields to the variables you named. With one variable it assigns the whole remainder to it, but it still strips the leading and trailing IFS characters first.

Setting IFS= to the empty string means there are no field separators, so there is nothing to strip and the line arrives byte-exact.

The IFS= prefix is a one-command assignment (Chapter 10.1): it applies to read only, for that one invocation, and does not change IFS for the loop body. That matters, because the body usually wants normal splitting.

When does it bite? Any format where leading whitespace is meaningful — an indented config file, a Makefile, a Python snippet, a diff, a column-aligned report where you are reading the whole line to re-emit it. If you are discarding the line's edges on purpose, a bare read is fine; just know you are doing it.

3. -r — Or Your Backslashes Are Eaten, and Two Lines Become One

$ printf 'C:\\users\\path\na\\tb\nends with backslash \\\nnext\n' > bs.txt
$ cat bs.txt
C:\users\path
a\tb
ends with backslash \
next

Four lines. Read them without -r:

$ while IFS= read l; do printf '[%s]\n' "$l"; done < bs.txt
[C:userspath]
[atb]
[ends with backslash next]

Three results for four lines, and all three are damaged.

Without -r, read treats a backslash as an escape character. So C:\users\path lost both backslashes, a\tb lost one — note it became atb, not a tab, because this is read's escape processing and not printf's. And the third line ended with a backslash, which read interpreted as a line continuation and joined it to the fourth line.

That last one is the reason -r is not optional. A single trailing backslash in an input file does not corrupt one value — it silently merges two records, so every count downstream is wrong and nothing looks broken.

$ while IFS= read -r l; do printf '[%s]\n' "$l"; done < bs.txt
[C:\users\path]
[a\tb]
[ends with backslash \]
[next]

-r means raw: no escape processing, bytes as they are, four lines in and four out.

Always pass -r. There is no realistic case where you want read's escape handling; if you need to interpret escapes, do it deliberately with printf '%b' on a value you already have intact.

4. The Last Line, If Nobody Terminated It

The third defect, and the only one the four-token incantation does not fix.

$ printf 'one\ntwo\nthree' > nonl.txt
$ wc -c < nonl.txt
      13
$ tail -c1 nonl.txt | xxd -p
65

Thirteen bytes, and the last one is 0x65 — the letter e. There is no trailing newline, which is entirely legal and extremely common: a file a program wrote without a final \n, a value pasted into a heredoc, output from a tool that terminates records with separators rather than terminators.

$ while IFS= read -r l; do printf '[%s]\n' "$l"; done < nonl.txt
[one]
[two]

three is gone. No error, no warning, no clue. A testlist whose last line lacks a newline silently loses its last test, and the run looks completely normal.

Here is exactly what happened, with the reads done by hand:

$ exec 3< nonl.txt
$ read -r a <&3; echo "a=$a rc=$?"
a=one rc=0
$ read -r b <&3; echo "b=$b rc=$?"
b=two rc=0
$ read -r c <&3; echo "c=$c rc=$?"
c=three rc=1

Look at the last line. c is three and the status is 1. read got the data and reported failure, because what it reports is not "did I get characters" but "did I find a line terminator". The loop condition believed the status, so the body never ran for a value that was sitting right there in the variable.

Which gives the fix:

$ while IFS= read -r l || [ -n "$l" ]; do printf '[%s]\n' "$l"; done < nonl.txt
[one]
[two]
[three]

|| [ -n "$l" ] is Chapter 16.5's || and Chapter 16.3's -n: continue if read succeeded, or if it failed but left something in the variable. On a file that does end with a newline the final read sets l to empty, so [ -n "$l" ] is false and the loop ends normally — the guard costs nothing and changes nothing in the common case.

5. Splitting Into Fields

Name more than one variable and read splits the line for you, using IFS — which is why you drop the IFS= prefix when you want this:

$ printf 'alu_rand  PASS  42\nfifo_full FAIL  7\n' > res.txt
$ while read -r name verdict n; do printf '%-10s %-5s %s\n' "$name" "$verdict" "$n"; done < res.txt
alu_rand   PASS  42
fifo_full  FAIL  7

Three variables, three fields, and runs of whitespace collapse — which is what you want for a column-aligned report.

Extra fields all land in the last variable:

$ printf 'a b c d e\n' > f5.txt
$ while read -r one two rest; do echo "one=[$one] two=[$two] rest=[$rest]"; done < f5.txt
one=[a] two=[b] rest=[c d e]

That is a feature and the standard way to read "a key and then everything else" — a log line's timestamp and message, a config file's name and value.

Change IFS for a different delimiter:

$ printf 'root:0:/root\nnobody:99:/var/empty\n' > pw.txt
$ while IFS=: read -r u uid home; do printf '%-8s %-4s %s\n' "$u" "$uid" "$home"; done < pw.txt
root     0    /root
nobody   99   /var/empty

IFS=: for a colon-separated file, IFS=, for simple CSV, IFS=$'\t' for TSV. This is the right tool for a well-behaved delimited file and it costs no processes — Chapter 13.2's cut and Chapter 13.7's awk are for when the format needs more than splitting.

6. The Loop Body That Eats the Input

A failure that looks impossible until you see the mechanism:

$ printf 'line1\nline2\nline3\n' > in.txt
$ while IFS= read -r l; do echo "GOT $l"; cat > /dev/null; done < in.txt
GOT line1

One iteration. cat here stands in for anything that reads standard input — ssh, ffmpeg, mysql, vcs in some modes — and the reason is simple once stated: < in.txt attached the file to the loop's standard input, and every command inside the loop inherits it. cat read the whole rest of the file and threw it away, so the next read found end of input.

Two fixes, and you often want both.

Redirect the inner command away from stdin:

$ while IFS= read -r l; do echo "GOT $l"; cat < /dev/null > /dev/null; done < in.txt
GOT line1
GOT line2
GOT line3

ssh -n is the same idea with a dedicated flag, and it exists precisely because this bug is so common.

Or read on a different file descriptor:

$ while IFS= read -r -u 3 l; do echo "GOT $l"; cat < /dev/null > /dev/null; done 3< in.txt
GOT line1
GOT line2
GOT line3

3< in.txt attaches the file to descriptor 3 and read -u 3 reads from there, so the loop's input is somewhere the body cannot reach by accident.

And one measured warning about that second fix: read -u 3 alone is not enough. With the inner command still reading descriptor 0, it reads whatever the script's stdin is — and if that is a terminal, the script hangs waiting for you to type. Writing this chapter, exactly that happened:

$ while IFS= read -r -u 3 l; do echo "GOT $l"; cat > /dev/null; done 3< in.txt
GOT line1
^C

The loop's input was protected; the body still blocked forever on the terminal. The fd trick protects the loop, not the body. Redirect the body's input too, or use the tool's own flag.

7. Never Pipe Into the Loop If You Need the Variables

The preview of Chapter 17.7, because you will hit it in this chapter's shape:

$ n=0; printf 'a\nb\nc\n' | while IFS= read -r l; do n=$(( n + 1 )); done; echo "pipe: n=$n"
pipe: n=0

$ n=0; while IFS= read -r l; do n=$(( n + 1 )); done < <(printf 'a\nb\nc\n'); echo "proc-sub: n=$n"
proc-sub: n=3

Same loop, same input, same counter. Zero against three. Chapter 17.7 is the explanation; the rule to carry until then is: feed a while read loop by redirection (< file) or process substitution (< <(cmd)), never by a pipe, whenever anything the body sets needs to be visible afterwards.

8. A Complete Script

A testlist runner — the single most common shape of real shell automation:

#!/usr/bin/env bash
# run-testlist.sh — run every test named in a testlist file
# usage: run-testlist.sh <testlist> <outdir>
#
# testlist format: one test per line; blank lines and # comments ignored;
#                  "<name>" or "<name> <seed>"
#
# exit: 0 all passed · 1 at least one failed · 2 unusable arguments

set -o nounset

list=${1:-}
outdir=${2:-}

if [ -z "$list" ] || [ -z "$outdir" ]; then
  echo "usage: ${0##*/} <testlist> <outdir>" >&2
  exit 2
fi
if ! [ -f "$list" ] || ! [ -r "$list" ]; then
  echo "${0##*/}: $list is not a readable file" >&2
  exit 2
fi
mkdir -p "$outdir" || exit 2

passed=0
failed=0
lineno=0

# read on fd 3 so a tool in the body cannot steal the testlist
while IFS= read -r -u 3 line || [ -n "$line" ]; do
  lineno=$(( lineno + 1 ))

  # strip a trailing comment, then the surrounding whitespace
  line=${line%%#*}
  line=${line#"${line%%[![:space:]]*}"}
  line=${line%"${line##*[![:space:]]}"}

  [ -z "$line" ] && continue

  # split the surviving line into at most two fields
  read -r name seed <<< "$line"
  seed=${seed:-1}

  if ! [ "$seed" -ge 0 ] 2>/dev/null; then
    printf '%s:%d: seed must be a number, got: %s\n' "$list" "$lineno" "$seed" >&2
    failed=$(( failed + 1 ))
    continue
  fi

  log="$outdir/${name}_s${seed}.log"

  # stand-in for the simulator; reads nothing, so < /dev/null is belt-and-braces
  if [ "$name" = "known_bad" ]; then
    printf 'UVM_ERROR : mismatch\n' > "$log"
  else
    printf 'TEST PASSED\n' > "$log"
  fi

  if grep -q 'TEST PASSED' "$log"; then
    printf '  pass   %-12s seed %s\n' "$name" "$seed"
    passed=$(( passed + 1 ))
  else
    printf '  FAIL   %-12s seed %s\n' "$name" "$seed"
    failed=$(( failed + 1 ))
  fi
done 3< "$list"

printf -- '-- %d passed, %d failed\n' "$passed" "$failed"
[ "$failed" -eq 0 ]

A testlist with every awkward feature in it, including no final newline:

$ printf '# the smoke tests\n\n  smoke 7\nburst\nknown_bad 3\n\n# trailing comment test\nreset_seq 12   # the long one\nbad_seed xyz\nlast_one' > tests.txt
$ tail -c1 tests.txt | xxd -p
65

$ ./run-testlist.sh tests.txt results
  pass   smoke        seed 7
  pass   burst        seed 1
  FAIL   known_bad    seed 3
  pass   reset_seq    seed 12
tests.txt:9: seed must be a number, got: xyz
  pass   last_one     seed 1
-- 4 passed, 2 failed
$ echo $?
1

Now remove the || [ -n "$line" ] guard and run the identical list:

$ sed 's/ || \[ -n "$line" \]//' run-testlist.sh > no-guard.sh && chmod +x no-guard.sh
$ ./no-guard.sh tests.txt results2
  pass   smoke        seed 7
  pass   burst        seed 1
  FAIL   known_bad    seed 3
  pass   reset_seq    seed 12
tests.txt:9: seed must be a number, got: xyz
-- 3 passed, 2 failed

last_one is gone and the pass count dropped from 4 to 3. Nothing announced it.

Everything in this chapter is doing work there:

|| [ -n "$line" ] caught last_one, which has no trailing newline. Without it the last test in every testlist is silently skipped.

-u 3 with 3< "$list" means the loop's input lives on descriptor 3, so a simulator that reads stdin cannot consume the testlist. This is the bug that turns a four-hundred-test regression into a one-test regression, and the symptom — "it only ran the first test" — gives no hint of the cause.

IFS= on the outer read, and a bare read for the fields. The outer read must not trim, because the trimming is done deliberately afterwards; the inner read -r name seed <<< "$line" wants normal splitting, and gets it.

read -r name seed <<< "$line" splits one value into fields using a here-string (Chapter 12.4) rather than a pipe — which is Section 7's rule. A pipe would put the assignment in a subshell and $name would be empty.

The two ${...} lines strip whitespace using Chapter 8.3's pattern removal, with no sed and no subprocess. They are dense, and the comment above them is why they are allowed to be.

bad_seed xyz was reported with a file name and line number rather than crashing or silently defaulting. lineno is counted by the loop because read does not track it for you.

Practice

1. Create the Section 2 fixture and read it with and without IFS=, printing [%s]. Then pipe both through cat -A (or cat -e) to see the exact bytes you kept and lost.
2. Create the Section 3 fixture with a line ending in a single backslash. Count the output lines with and without -r. Which specific input line causes the count to change, and why is that worse than losing a character?
3. Build a file with printf 'a\nb\nc' — no final newline. Read it naively, then with the guard. Then verify wc -l on the file and explain why it disagrees with the number of lines you can see.
4. Reproduce the explicit-fd experiment: open the unterminated file on fd 3 and read three times, printing the variable and $? each time. Explain the third result in one sentence.
5. Add the || [ -n "$line" ] guard to a loop reading a file that does end in a newline. Show that it changes nothing, and explain why.
6. Read /etc/passwd with IFS=: into five variables. Now try it on a line with more than five fields — where do the extras go?
7. Put cat in a loop body with < in.txt on the loop and watch it stop after one iteration. Fix it two ways, then break fix B deliberately by leaving the body's stdin attached to your terminal.
8. Run the Section 7 pipe-versus-process-substitution pair. Then add a printf inside each loop showing $n per iteration, and say where the pipe version's increments went.
9. Give run-testlist.sh a testlist whose final line is last_one with no newline, then remove the guard and re-run. Which test disappears?
10. Write a testlist with a test named my test (a space in the name). What does run-testlist.sh do, and what would it take to support it? Name the module.

Check Your Understanding

1. What does read return at end of input, and why does that make while read work?
2. What does the IFS= prefix prevent, and why does it not affect the loop body?
3. Name the two things a missing -r does, and say which is worse.
4. A file's last line has no newline. What happens, and what is read's status on that line?
5. Why does || [ -n "$line" ] cost nothing on a well-formed file?
6. A loop over a testlist runs only the first test. What is the most likely cause?
7. Why < <(cmd) rather than cmd |?

9. Remember

• while IFS= read -r line — four tokens, four defences, and a fifth you often need.
• read reports whether it found a terminator, not whether it got data. That is why while read works and why the last line can be lost.
• IFS= stops both ends of the line being trimmed. It is a one-command assignment and does not change IFS for the body.
• -r stops escape processing. Without it a line ending in a backslash is joined to the next one, which corrupts your record count silently.
• An unterminated last line is dropped. Add || [ -n "$line" ] — read sets the variable and still returns 1.
• Several variables split the line on IFS, and extra fields all land in the last one. IFS=: and IFS=$'\t' handle delimited files.
• IFS splits on a character; it does not parse a format. Quoted commas need a real CSV parser.
• A command in the body can eat the loop's input. Use < /dev/null on it, or read -u 3 with 3< file — and note the fd trick protects the loop, not the body.
• Feed the loop with < file or < <(cmd), never a pipe, if anything it sets must survive.
• Count the line number yourself if you want to report errors against the input.

Chapter 17.6 is the two statements that let you leave a loop deliberately — and the measured reason a search loop needs a flag variable even though break seems like it should be enough.