Redirecting Output to a File: > and >>

Part I · Module 12 · Chapter 12.2 · Redirecting Output to a File

Chapter 12.1 described a table of three entries, each pointing somewhere. This chapter changes entry one.

1. Two Operators

$ echo one > t.txt
$ echo two > t.txt
$ cat t.txt
two

> replaces the file's contents. The first line is gone.

$ echo one >  t.txt
$ echo two >> t.txt
$ cat t.txt
one
two

>> appends. Both lines survive.

Both create the file if it does not exist. The only difference is what happens when it does.

make > build.log                # a fresh log for this build
echo "finished at $(date)" >> build.log   # add a line to it

Where the operator goes

The redirect is not an argument to the command — it is an instruction to the shell, and it can sit anywhere on the line:

make > build.log                # conventional
make all > build.log            # also conventional
> build.log make all            # legal, and nobody writes this

Put it at the end. The first form is what every reader expects, and a redirect buried in the middle of a long command line is a reliable way to be misread.

2. The File Is Opened First

Here is the fact the rest of the chapter depends on.

The shell sets up every redirect before it starts the command. It parses the line, sees > file, opens that file — creating it or emptying it — and only then runs the program with its output already pointed there.

Two consequences, both measured, both expensive.

A command that never ran still destroyed the file

$ echo "important" > keep.txt
$ cat keep.txt
important

$ nosuchcommand > keep.txt
bash: nosuchcommand: command not found

$ wc -c < keep.txt
0

The file is empty. nosuchcommand does not exist — nothing ran, nothing produced output, the shell reported command not found. And keep.txt was emptied anyway, because the shell opened it before discovering there was no command to run.

The same happens when the command exists and fails:

$ make > build.log

If make immediately errors out, build.log is a zero-byte file. The previous build's log is gone, which is exactly when you wanted it.

A file cannot be its own input

$ printf 'c\nb\na\n' > s.txt
$ cat s.txt
c
b
a

$ sort s.txt > s.txt
$ wc -c < s.txt
0

Empty. Not sorted — gone.

Read the order: the shell opened s.txt for writing, which emptied it, and then started sort with s.txt as its argument. sort opened a file with nothing in it, read nothing, and wrote nothing.

A redirect with no command at all

$ echo data > z.txt
$ > z.txt
$ wc -c < z.txt
0

> file on its own is a complete command line: the shell performs the redirect and runs nothing. It is the shortest way to empty a file deliberately, and it is also what a half-typed line does if you press Enter early.

3. Making the Shell Refuse

Bash can be told to decline rather than overwrite:

$ echo one > nc.txt
$ set -o noclobber

$ echo two > nc.txt
bash: nc.txt: cannot overwrite existing file

$ cat nc.txt
one

noclobber makes > fail on a file that already exists. >> is unaffected, and creating a new file is unaffected.

When you do mean it:

$ echo three >| nc.txt
$ cat nc.txt
three

>| is the override — "yes, overwrite it".

4. Which Stream Moves

> redirects stream two only — ordinary output. Error messages are untouched:

$ ls /tmp /nope > out.txt
ls: /nope: No such file or directory

$ cat out.txt
/tmp:
agent.sock  build  scratch

The listing went to the file and the complaint stayed on your screen, exactly as Chapter 12.1 said it would.

This is usually what you want. A build whose output goes to a log and whose errors appear live is the right default: you see problems as they happen and you still have the full record afterwards.

When you want both in the file, that is 2>&1 — and getting it right is the whole of Chapter 12.3.

1> is the explicit spelling of >:

make 1> build.log      # identical to make > build.log

Nobody writes the 1, because there is nothing to disambiguate — but it is worth recognising, because it makes the symmetry with 2> obvious.

5. Appending, and When It Is Wrong

>> is right whenever you are adding to a running record:

echo "$(date +%F' '%T) run started" >> history.log

It is wrong as a default for a command's whole output, and the failure mode is quiet:

make >> build.log       # today's build appended to last week's

Now the log contains several runs with no clear boundary, and grep ERROR build.log finds an error from Tuesday. You will spend twenty minutes on it.

6. Redirecting More Than One Command

A redirect attaches to one command:

$ echo a > out.txt; echo b
a

...only echo a was redirected. echo b printed to the screen and out.txt holds just a.

To cover several commands, group them — Chapter 11.4:

$ { echo a; echo b; } > out.txt
$ cat out.txt
a
b

One redirect, one open of the file, both commands writing into it. This is the right way to build a report:

{
  echo "generated : $(date +%F' '%T)"
  echo "host      : $(hostname -s)"
  echo "passed    : $pass"
  echo "failed    : $fail"
} > "$report"

Note what the alternative would require: > on the first line and >> on the other three — which breaks the moment somebody reorders them, and opens the file four times instead of once.

And for an entire script, redirect at the point of invocation:

./regress.sh > run.log 2>&1

Which covers the script and everything it starts, by inheritance (Chapter 12.1 §7).

7. Practical Shapes

Four patterns that cover most real use.

A log per run, with a predictable name:

logdir="$HOME/runs/$(date +%Y%m%d)"
mkdir -p "$logdir"
./run_sim.sh > "$logdir/sim_$(date +%H%M%S).log" 2>&1

Writing a generated file safely:

tmp="$(mktemp)" || exit 1
generate_filelist > "$tmp" && mv "$tmp" filelist.f

mktemp creates a uniquely-named temporary file, and the && means a failed generation leaves the old filelist.f intact. Without this, a generator that dies halfway leaves a truncated filelist that the next tool will happily try to use.

Appending to an audit trail:

printf '%s  %s  %s\n' "$(date +%F' '%T)" "$USER" "$run_name" >> "$HOME/runs/history.tsv"

Deliberately emptying a file:

> "$logfile"

Practice

mkdir -p /tmp/redir && cd /tmp/redir

1 — Truncate and append.

echo one > t.txt; echo two > t.txt;  echo "after two >:  [$(cat t.txt)]"
echo one > t.txt; echo two >> t.txt; echo "after > then >>: [$(tr '\n' ' ' < t.txt)]"

2 — Destroy a file with a command that does not exist.

echo "important" > keep.txt
cat keep.txt
nosuchcommand > keep.txt
echo "keep.txt is now $(wc -c < keep.txt) bytes"

3 — Sort a file onto itself.

printf 'c\nb\na\n' > s.txt
cat s.txt
sort s.txt > s.txt
echo "s.txt is now $(wc -c < s.txt) bytes"

4 — Then do it properly.

printf 'c\nb\na\n' > s.txt
sort s.txt > s.sorted && mv s.sorted s.txt
cat s.txt

5 — Empty a file with no command.

echo data > z.txt; wc -c < z.txt
> z.txt;           wc -c < z.txt

6 — Try noclobber.

echo one > nc.txt
( set -o noclobber; echo two > nc.txt )
cat nc.txt
( set -o noclobber; echo three >| nc.txt ); cat nc.txt

7 — Confirm > moves only one stream.

ls /tmp /nope > out.txt
echo "--- out.txt has $(wc -l < out.txt) lines, and the error was on screen"

8 — Redirect a group.

echo a > out.txt; echo b
cat out.txt
{ echo a; echo b; } > out.txt
cat out.txt

9 — Meet ambiguous redirect.

unset d
( echo hello > $d/file.txt )
( d="my dir"; echo hello > $d/file.txt )

10 — Clean up. cd /tmp && rm -rf /tmp/redir

Check Your Understanding

1. What is the difference between > and >>?

2. You run mak > build.log with a typo and the previous log is gone. Why, when nothing ran?

3. sort data.txt > data.txt leaves an empty file. Explain the order of events, and give a safe alternative.

4. noclobber is on. Which of these fail: > new.txt, > existing.txt, >> existing.txt, >| existing.txt?

5. make > build.log and errors still appear on screen. Which stream was redirected?

6. Why is { echo a; echo b; } > out.txt better than two lines using > and >>?

8. Remember

• > replaces; >> appends. Both create the file if it is missing.
• The shell opens the file before running the command. Everything surprising follows from that.
• A command that does not exist still empties the file. So does one that fails immediately.
• A file cannot be its own input. sort f > f empties it. Use a temporary and && mv.
• > file with no command is a legal line that empties the file.
• noclobber makes > refuse an existing file, and >| overrides it. Good at a prompt, awkward in scripts.
• noclobber does not protect against sort f > f — that file is opened for writing either way.
• > moves only stream two. Errors keep going to your terminal, which is usually right.
• 1> is the explicit spelling and makes the symmetry with 2> visible.
• Prefer a new log per run to an ever-growing one. A dated filename sorts chronologically and cannot mix two runs.
• A redirect attaches to one command. Group with { …; } to cover several.
• Quote the destination. An unquoted one with a space or an unset variable gives ambiguous redirect.
• mktemp plus && mv is the safe way to generate a file something else depends on.

You can now capture output. The errors are still going somewhere else — and the operator that collects them has an ordering rule that catches almost everybody exactly once.