Linux & Bash · Part I · Module 12 · Streams, Redirection and Pipes

Three Streams: stdin, stdout and stderr

Output and errors look like one thing on your screen because they both arrive there. They are not one thing. Underneath, every process is born with three separate channels, and keeping results apart from complaints about results is the decision that makes the rest of this module possible.

Foundation17 min readLinuxPart IFundamentalsStreamsRedirection

Part I · Module 12 · Chapter 12.1 · Three Streams: stdin, stdout and stderr

Chapter 7.1 listed five things a process carries and mentioned open files in a sentence, promising a module. This is it.

1. Three Channels, Not One

Every process starts life with three streams already open. They have numbers and names:

Number	Name	Carries
`0`	standard input, `stdin`	What the process reads
`1`	standard output, `stdout`	Its ordinary results
`2`	standard error, `stderr`	Its error messages and diagnostics

The numbers matter because that is how you refer to them in the operators that fill the rest of this module. The names matter because that is how everyone talks about them.

Figure 1 — the three streams every process is born with. It reads from the first, writes its ordinary results to the second, and writes error messages and diagnostics to the third. All three are connected to your terminal by default, which is why results and errors appear mixed together on your screen even though they are entirely separate channels underneath.

By default all three are connected to your terminal. That is why they look like one thing: input comes from your keyboard, and both output streams arrive on the same screen, interleaved in the order they were written.

They are not one thing, and the next section proves it.

2. Proof That They Are Separate

ls given two arguments, one good and one bad, writes to both streams:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ls /tmp /nope
ls: /nope: No such file or directory
/tmp:
agent.sock  build  scratch

On screen that is a single lump of text. Send each stream to a different file and the lump comes apart:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ls /tmp /nope > out.txt 2> err.txt

$ cat out.txt
/tmp:
agent.sock  build  scratch

$ cat err.txt
ls: /nope: No such file or directory

The listing went one way and the complaint went the other. ls wrote to two different places without being asked to; it always does.

Discard either one and the other survives untouched:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ls /tmp /nope > /dev/null
ls: /nope: No such file or directory

$ ls /tmp /nope 2> /dev/null
/tmp:
agent.sock  build  scratch

The separation is the whole point

It would have been simpler to give a process one output channel. The design deliberately did not, and the reason is composability:

A pipe carries only stream two. So make | grep warning filters the build output and leaves the error messages on your screen where you can see them — rather than feeding them into grep and losing them.

A log file can hold only what you asked for. make > build.log keeps the output and leaves the errors visible, which is usually exactly what you want while a build runs.

Errors stay visible by default. Redirecting a command's output never silences its complaints — you have to ask for that separately, which means you cannot hide an error by accident.

Every operator in this module exists because there are two output streams rather than one. With a single channel, none of them would be useful.

3. Which Stream Does What

The convention, stated plainly:

Stream two — results. The thing the command was asked to produce. A file listing, a line count, matched lines, a compiled file name. Anything another program might reasonably want to consume.

Stream three — everything else. Error messages, warnings, progress bars, "3 of 47 files processed", prompts. Anything meant for a person rather than for the next program in a pipeline.

That second list surprises people. Progress output goes to the error stream, and it is not an error. The test is not "is this bad news?" but "would the next program in a pipeline want this?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ curl -s -o file.iso https://example.com/file.iso

curl's progress meter goes to stream three precisely so that curl url | tar xz works — the archive flows down the pipe and the percentages go to your screen.

Not every program follows the convention

It is a convention, not a rule, and the violations are a genuine nuisance:

Some tools print errors on stream two, so a pipeline silently ingests "ERROR: licence denied" as though it were data.
Some print ordinary output on stream three, so redirecting the output captures nothing and the log is empty.
Some check whether they are writing to a terminal and behave differently — colour and progress when you watch, plain text when you redirect. That is why a log file sometimes looks nothing like what was on screen.

The way to find out is to look, not to assume:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

mytool > out.txt 2> err.txt
wc -l out.txt err.txt

Thirty seconds, and you know which stream the thing you care about is on. Do this once for every EDA tool you rely on — several of them are surprising.

4. Writing to Each Stream Yourself

In a script, echo writes to stream two. To write to stream three — which is where your error messages belong — redirect it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

echo "starting run $run_name"                 # results
echo "warning: no filelist found" >&2         # diagnostics

>&2 is Chapter 12.3's notation and is worth adopting immediately. It matters for a reason that is easy to miss:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ./myscript.sh > run.log

Anything the script wrote with plain echo is now in the file and invisible. Anything it wrote with >&2 is still on your screen. If your error messages are not on stream three, the person running your script in a pipeline will never see them.

The rule for a script worth handing to someone else:

What you are printing	Where it goes
The answer	Stream two — plain `echo`
Errors, warnings, progress, prompts	Stream three — `>&2`

5. Input

Stream one is where a command reads from, and it has two modes that behave very differently.

Given a filename, most commands read the file:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ wc -l names.txt
       3 names.txt

Given nothing, they read stream one — which is your keyboard by default:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ wc -l
hello
there
^D
       2

Ctrl-D sends end-of-input, which is how you tell a command reading from the keyboard that you have finished. (Chapter 1.4 introduced it; this is why it exists.)

That second mode is what makes a command usable in a pipeline at all. A program that can only read a named file cannot be the second stage of anything.

A command that appears to hang is often reading stream one

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ grep ERROR

No file, so grep is reading from your keyboard and waiting. It is not stuck — it is doing exactly what it was asked.

Ctrl-D ends it politely and Ctrl-C ends it rudely (Chapter 7.6). This is the single most common "the terminal froze" that is not a frozen terminal.

In a script, a command that unexpectedly reads from the keyboard will hang an unattended job forever. < /dev/null is the guard — Chapter 12.7.

Filename versus stream

The same command can produce slightly different output depending on which mode you use, which matters more than it sounds:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ wc -l names.txt
       3 names.txt

$ wc -l < names.txt
       3

Given a filename, wc prints it. Reading from stream one there is no name to print — wc has no idea where the data came from. Chapter 8.4 hit this when capturing a line count and getting the filename along with it; Chapter 12.4 covers the < operator properly.

6. Where the Streams Actually Point

The three streams are entries in a small per-process table, and on Linux you can look at it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ls -l /proc/$$/fd
lrwx------ 1 engineer engineer 64 Oct  8 11:02 0 -> /dev/pts/3
lrwx------ 1 engineer engineer 64 Oct  8 11:02 1 -> /dev/pts/3
lrwx------ 1 engineer engineer 64 Oct  8 11:02 2 -> /dev/pts/3

All three pointing at /dev/pts/3 — your terminal, which is a device file like any other. That is the literal meaning of "connected to your terminal".

Redirect something and an entry changes:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'ls -l /proc/$$/fd' > /tmp/fds.txt
$ grep ' 1 ->' /tmp/fds.txt
l-wx------ 1 engineer engineer 64 Oct  8 11:02 1 -> /tmp/fds.txt

Entry 1 now points at the file. Entries 0 and 2 still point at the terminal.

7. Inheritance

Streams are inherited, exactly like everything else in Chapter 7.4. That chapter listed "open files" among the things a child receives, and this is what it meant:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ./outer.sh > everything.log

outer.sh, every command it runs, and every command those run all write into everything.log, because each inherited the redirected stream from its parent. One redirect covers an entire tree of processes.

This is why:

./regress.sh > run.log 2>&1 captures the whole run, including output from tools three levels deep that know nothing about your redirect.
A background job keeps writing to your terminal (Chapter 7.5) — it inherited the terminal as its output and backgrounding does not change that.
A detached job must be redirected (Chapter 7.8), because its inherited terminal is about to disappear.

Three behaviours from three different modules, one mechanism.

8. What the Rest of the Module Does

Everything that follows is this model plus an operator:

Chapter	What it changes
12.2	Point stream two at a file
12.3	Point stream three somewhere, or at the same place as two
12.4	Point stream one at a file or at inline text
12.5	Connect one process's stream two to the next one's stream one
12.6	Send stream two to two places at once
12.7	Point a stream at nothing
12.8	Find out whether any of it worked

Seven operators, one idea. If the table in §6 makes sense, the rest of the module is notation.

Practice

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

mkdir -p /tmp/streams && cd /tmp/streams
printf 'charlie\nalpha\nbravo\n' > names.txt

1 — Split the two output streams.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

ls /tmp /nope > out.txt 2> err.txt
echo "--- out ---"; head -3 out.txt
echo "--- err ---"; cat err.txt

2 — Discard each in turn.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

ls /tmp /nope > /dev/null
ls /tmp /nope 2> /dev/null | head -2

3 — Write to each stream from a script.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf '#!/bin/bash\necho "this is a result"\necho "this is a problem" >&2\n' > both.sh
chmod +x both.sh
./both.sh
echo "--- with output redirected: ---"
./both.sh > /dev/null
echo "--- with errors redirected: ---"
./both.sh 2> /dev/null

4 — Watch a command read from the keyboard.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

wc -l

Type two lines, then press Ctrl-D.

5 — See the filename difference.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

wc -l names.txt
wc -l < names.txt

6 — Find out which stream a tool uses.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

mytool() { echo "result line"; echo "progress: 50%" >&2; }
mytool > o.txt 2> e.txt
echo "stdout: [$(cat o.txt)]"
echo "stderr: [$(cat e.txt)]"

Then do the same with a real tool you use.

7 — Look at the table, if you are on Linux.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

ls -l /proc/$$/fd
bash -c 'ls -l /proc/$$/fd' > fds.txt; grep ' 1 ->' fds.txt

8 — Prove inheritance.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf '#!/bin/bash\necho "from the outer script"\n./both.sh\n' > outer.sh
chmod +x outer.sh
./outer.sh > everything.log 2>&1
cat everything.log

One redirect, output from two scripts.

9 — Clean up. cd /tmp && rm -rf /tmp/streams

Check Your Understanding

1. Why are there two output streams rather than one?

2. make > build.log and the errors still appear on your screen. Is something wrong?

3. A tool prints a progress percentage. Which stream should it use, and why is that not an error?

4. You run grep ERROR with no filename and nothing happens. What is it doing?

5. wc -l data.txt prints 42 data.txt but wc -l < data.txt prints 42. Why?

6. You redirect a script's output to a log and a tool it calls three levels down also ends up in the log. How?

Answers

1. For composability. Because a pipe carries only stream two, make | grep warning can filter the build output while leaving error messages visible on your screen. With a single channel, every redirect and every pipe would mix results with diagnostics, and neither could be used without losing something.

2. No — that is the design working. > redirects only stream two. Error messages travel on stream three, which still points at your terminal. Silencing them requires asking separately, which means you cannot hide an error by accident.

3. Stream three, the error stream. The test is not "is this bad news?" but "would the next program in a pipeline want this?" A progress meter is for a person, so it must not be fed into the next stage — which is exactly what putting it on stream three guarantees.

4. Reading from your keyboard. With no filename, grep reads stream one, which is the terminal by default, and waits for input. Press Ctrl-D to signal end of input or Ctrl-C to abandon it. This is the most common "frozen terminal" that is not frozen.

5. Given a filename, wc knows the name and prints it alongside the count. Reading from stream one it has no idea where the data came from, so there is nothing to print. That is why $( ) captures of line counts should use <.

6. Because streams are inherited. Each child process receives a copy of its parent's stream table, so the redirect applied to the top-level script is passed down through every level. One redirect captures an entire tree of processes.

9. Remember

Every process starts with three open streams: 0 input, 1 ordinary output, 2 error output.
All three point at your terminal by default, which is why output and errors look like one thing.
They are separate, and that is the point. A pipe carries only stream two, so diagnostics stay visible.
Stream two is for results — anything the next program might want. Stream three is for everything meant for a person: errors, warnings, progress, prompts.
Progress output belongs on stream three and is not an error. The test is "would a pipeline want this?"
Not every tool follows the convention. Run it with the two streams in two files and look.
In a script, write diagnostics with >&2 or they vanish when somebody redirects your output.
A command with no filename reads stream one — your keyboard — and appears to hang. Ctrl-D ends it.
Reading from a stream means there is no filename to print, which is why wc -l < f differs from wc -l f.
The streams are entries in a per-process table, which is why the operators use numbers — 2> means "point entry 2 somewhere else".
Streams are inherited, so one redirect covers a whole tree of processes.
The rest of the module is seven operators applied to this one model.
Which stream each of your own messages belongs on is Chapter 20.7 — and it decides whether your script can be used in a pipeline at all.

The first operator is the one everybody learns on day one and nobody is told the dangerous part of. The next chapter is > — including exactly when it destroys a file, and why it can do so even when the command never ran.

Where this fits

Part of the Linux & Bash curriculum.