Redirecting Errors, and Combining Streams

Part I · Module 12 · Chapter 12.3 · Redirecting Errors, and Combining Streams

Chapter 12.2 moved stream two into a file and left stream three on your screen. This chapter is about the other one.

1. Errors to Their Own File

$ ls /tmp /nope 2> errors.log
/tmp:
agent.sock  build  scratch

$ cat errors.log
ls: /nope: No such file or directory

2> redirects stream three, exactly as > redirects stream two. The 2 is the stream number from Chapter 12.1.

Everything from the previous chapter applies unchanged:

make 2> errors.log       # replace
make 2>> errors.log      # append

And the two can be sent to different places at once:

$ ls /tmp /nope > out.txt 2> err.txt
$ wc -l out.txt err.txt
      42 out.txt
       1 err.txt

2. Both Into One File

The common need is one complete log. The operator is 2>&1:

$ ls /tmp /nope > build.log 2>&1
$ cat build.log
ls: /nope: No such file or directory
/tmp:
agent.sock  build  scratch

Both streams in one file. Now the part that matters.

3. The Ordering Rule

Redirects are processed left to right. Combine that with "copies the destination right now" and the rule falls out.

$ ls /tmp /nope > a.log 2>&1
$ grep -c 'No such' a.log
1

Both in the file. Now reverse the two:

$ ls /tmp /nope 2>&1 > b.log
ls: /nope: No such file or directory

$ grep -c 'No such' b.log
0

The error appeared on your terminal and is not in the file.

Walk 2>&1 > b.log through the table:

Entry 2 copied the terminal, and then entry 1 moved away. Nothing told entry 2 to follow.

And > a.log 2>&1:

4. The Shorthand

Bash has a single operator for the common case:

$ ls /tmp /nope &> all.log

&> is exactly > file 2>&1, with no ordering to get wrong. It is a Bash extension and is not in POSIX sh.

The append form is where portability bites:

$ echo x &>> ap.log
bash: syntax error near unexpected token `>'

&>> needs Bash 4, and Bash 3.2 — still what macOS ships — rejects it as a syntax error. The portable append is the long form:

make >> build.log 2>&1

5. Sending Output to the Error Stream

The reverse direction, 1>&2 — usually written >&2 since 1 is the default:

echo "warning: no filelist found" >&2

Chapter 12.1 introduced this; here is why it matters, demonstrated:

$ cat both.sh
#!/bin/bash
echo "this is a result"
echo "this is a problem" >&2

$ ./both.sh > /dev/null
this is a problem

$ ./both.sh 2> /dev/null
this is a result

The two messages are genuinely separable, because the script put them on different streams. Had both used plain echo, redirecting the output would have silenced the problem too.

Every diagnostic in a script you hand to someone else belongs on stream three, with no exceptions worth making:

die() { echo "error: $*" >&2; exit 1; }
warn() { echo "warning: $*" >&2; }

[ -f "$config" ] || die "no config at $config"
[ -d "$outdir" ] || warn "output directory missing, creating it"

Those two one-line functions are worth having in every script of any size. (Functions are Chapter 18.1, and Chapter 18.4 is why the >&2 belongs inside the helper.)

6. Keeping Them Apart On Purpose

Merging is the common case. Separating is the better one when you intend to check the result:

./run_sim.sh > sim.log 2> sim.err

if [ -s sim.err ]; then
  echo "the simulation wrote to the error stream:" >&2
  cat sim.err >&2
fi

[ -s file ] tests whether a file is non-empty (Module 16), so this is "did anything go to the error stream at all?" — a cheap and surprisingly effective check for tools whose exit status is not trustworthy.

That last part is not hypothetical. Several EDA tools exit zero after a fatal error, which makes Chapter 11.1's status useless on its own. A separate error file gives you a second signal.

7. Putting It Together

#!/bin/bash
set -uo pipefail

run="${1:?usage: $0 <run-name>}"
logdir="$HOME/runs/$run"
mkdir -p "$logdir" || { echo "cannot create $logdir" >&2; exit 2; }

die()  { echo "error: $*"   >&2; exit 1; }
warn() { echo "warning: $*" >&2; }

command -v vcs >/dev/null 2>&1 || die "vcs not on PATH — source the tool setup"

# Separate files: we want to check whether anything reached the error stream.
vcs -full64 -sverilog -f filelist.f -o simv \
  > "$logdir/compile.log" 2> "$logdir/compile.err"
status=$?

if [ -s "$logdir/compile.err" ]; then
  warn "compiler wrote to the error stream:"
  head -20 "$logdir/compile.err" >&2
fi

[ "$status" -eq 0 ] || die "compile failed with status $status"

# One merged log for the run itself, because here the interleaving is useful.
./simv +UVM_TESTNAME="$run" > "$logdir/sim.log" 2>&1

Four decisions worth naming:

• >/dev/null 2>&1 on the command -v check — the output is irrelevant, only the status matters (Chapter 12.7).
• Separate files for the compile, because [ -s …err ] is a second signal beyond the exit status.
• A merged log for the simulation, because reading a run log is easier with errors in context.
• >&2 on every diagnostic, so redirecting the script's output never hides a problem.

Practice

mkdir -p /tmp/err && cd /tmp/err

1 — Separate the streams.

ls /tmp /nope > out.txt 2> err.txt
echo "out: $(wc -l < out.txt) lines   err: $(wc -l < err.txt) lines"

2 — Prove the ordering rule.

ls /tmp /nope > a.log 2>&1 ;   echo "a.log has the error? $(grep -c 'No such' a.log)"
ls /tmp /nope 2>&1 > b.log ;   echo "b.log has the error? $(grep -c 'No such' b.log)"

The second prints the error on your screen as it runs. That is the point.

3 — Walk the table yourself. For cmd 2>&1 > f, write down where entries 1 and 2 point after each step, then check against §3.

4 — Try both shorthands.

{ echo n; echo e >&2; } &> c.log;  echo "c.log = [$(tr '\n' ' ' < c.log)]"
{ echo n2; echo e2 >&2; } &>> c.log

The second may be a syntax error. Check with bash --version.

5 — Use the portable append.

{ echo n2; echo e2 >&2; } >> c.log 2>&1
echo "c.log = [$(tr '\n' ' ' < c.log)]"

6 — Meet the space trap.

echo 2 > f1.txt;   echo "f1 = [$(cat f1.txt)]"
echo hi 2> f2.txt; echo "f2 = [$(cat f2.txt)]  (empty — hi went to the screen)"

7 — Write a script that uses both streams properly.

cat > checked.sh <<'EOF'
#!/bin/bash
die()  { echo "error: $*"   >&2; exit 1; }
warn() { echo "warning: $*" >&2; }
echo "result: 42"
warn "this is only a warning"
[ -f /definitely/not/here ] || die "required file missing"
EOF
chmod +x checked.sh
./checked.sh;            echo "  -> $?"
./checked.sh 2>/dev/null; echo "  -> $? (diagnostics hidden, result visible)"
./checked.sh >/dev/null;  echo "  -> $? (result hidden, diagnostics visible)"

8 — Check for anything on the error stream.

./checked.sh > o.txt 2> e.txt
[ -s e.txt ] && { echo "the script complained:"; cat e.txt; }

9 — Clean up. cd /tmp && rm -rf /tmp/err

Check Your Understanding

1. What does 2>&1 mean, read literally?

2. Why does make 2>&1 > build.log leave errors on your screen?

3. What is the difference between echo 2 > f.txt and echo hi 2> f.txt?

4. You want both streams appended to a log, on any machine. Which form?

5. A build log has all the errors at the top and the compile lines below. Is the file corrupt?

6. Why should a script's warnings go to stream three?

8. Remember

• 2> redirects the error stream, and 2>> appends — everything from Chapter 12.2 applies.
• The number must touch the operator. echo 2 > f writes a 2; echo hi 2> f redirects.
• 2>&1 means "point entry 2 wherever entry 1 points right now". The & marks 1 as a stream, not a file.
• It copies a destination at that moment and does not link the streams.
• Redirects run left to right, so > file 2>&1 captures both and 2>&1 > file does not.
• The destination comes first, then the copy. 2>&1 must follow the redirect it is meant to track.
• &> is the Bash shorthand for > file 2>&1. &>> needs Bash 4 and fails on macOS's 3.2.
• In scripts write the long form. It is portable, explicit, and has no version trap.
• >&2 sends output to the error stream — where every diagnostic in a script belongs.
• die and warn one-liners are worth having in any script of size.
• Separate files let you check [ -s errors.log ] — a second signal for tools whose exit status lies.
• Merging interleaves unpredictably because of buffering. Errors appearing before their output is normal, not corruption.

Both output streams are now under control. The next chapter turns the arrow around: feeding a command from a file, or from text written directly in the script.