Foreground, Background and Job Control

Part I · Module 7 · Chapter 7.5 · Foreground, Background and Job Control

The last three chapters were about looking. This one is about doing — specifically, about the fact that your shell, left to itself, will only do one thing at a time.

1. Why the Terminal Locks Up

Type a command and the shell starts it as a child process (Chapter 7.4) and then waits. It does not print a prompt, does not read what you type, does nothing at all until the child exits.

$ sleep 30

Thirty seconds of nothing. The terminal is not frozen and nothing is broken; your shell is simply doing what it always does — waiting for its child.

A process the shell is waiting for is said to be in the foreground. There is exactly one at a time, it has the terminal to itself, and anything you type goes to it rather than to the shell.

That is the right behaviour for ls and for grep. It is useless for a forty-minute simulation.

2. Three States

Job control gives a process three possible states, and the whole chapter is about moving between them.

The third one is the one people get wrong, so it is worth saying twice: a stopped job is not running. It is suspended — frozen mid-instruction, consuming no CPU, making no progress. §6 returns to this.

3. Starting in the Background: &

Put an ampersand at the end of a command line and the shell starts the process and does not wait:

$ sleep 120 &
[1] 11407
$

The prompt comes straight back. Two numbers come with it, and they are different things:

• [1] is the job number — the shell's own small counter, starting at 1, meaningful only inside this shell.
• 11407 is the PID — the system-wide process ID from Chapter 7.1.

Listing what you have

$ sleep 120 &
[1] 11407
$ sleep 200 &
[2] 11409

$ jobs
[1]-  Running                 sleep 120 &
[2]+  Running                 sleep 200 &

Four fields per row:

• [1] — the job number.
• + or - — the current and previous job. The + is what commands act on when you do not say which; the - is next in line. They move as jobs come and go.
• Running — the state, from §2.
• sleep 120 & — the command line, as you typed it.

Add -l for the PIDs:

$ jobs -l
[1]- 11407 Running                 sleep 120 &
[2]+ 11409 Running                 sleep 200 &

4. Suspending What Is Already Running: Ctrl-Z

The common case is not planning ahead. You start something, it turns out to take much longer than expected, and the & you should have typed is thirty seconds in the past.

Ctrl-Z suspends the foreground process:

$ sleep 200
^Z
[2]+  Stopped                 sleep 200
$

You have your prompt back. The job exists, it is listed, and it is Stopped — frozen exactly where it was, using no CPU, making no progress.

That gives you a choice:

$ bg %2
[2]+ sleep 200 &

bg resumes it in the background: running again, with your prompt still yours. Note that jobs now shows it with the trailing & it never had when you typed it.

$ fg %2
sleep 200

fg resumes it in the foreground: back to where it started, with the shell waiting again. fg prints the command line as it goes, so you can see what you just pulled back.

5. Naming a Job

Every job-control command takes a job specification — a % followed by something identifying.

Verified against a live shell with two jobs:

$ jobs %1
[1]-  Running                 sleep 100 &
$ jobs %+
[2]+  Running                 sleep 101 &
$ jobs %-
[1]-  Running                 sleep 100 &
$ jobs %?101
[2]+  Running                 sleep 101 &

In practice you will type fg with no argument more than anything else, because the job you want is almost always the one you just suspended — which is, by definition, the current job.

6. The First Surprise: Stopped Is Not Running

$ sleep 200 &
[2] 11409

$ kill -STOP %2

$ jobs
[1]-  Running                 sleep 120 &
[2]+  Stopped                 sleep 200

Job 2 is stopped. It is not progressing. Your terminal is free, there is no error, nothing warns you — and the job you believe is running is doing nothing whatsoever.

The usual way to arrive here by accident is Ctrl-Z. You suspend something intending to come back to it, get distracted, and an hour later discover an hour of nothing.

7. The Second Surprise: Background Output Still Arrives

Backgrounding a job detaches it from the shell's waiting. It does not detach it from your terminal.

$ (sleep 1; echo 'I AM BACKGROUND OUTPUT BARGING IN') &
[1] 14400
$ I AM BACKGROUND OUTPUT BARGING IN

Look at where that landed — after the prompt, in the middle of whatever you were typing. The background job inherited the terminal as its output (Chapter 7.4 listed it) and writes to it whenever it likes.

For a chatty job this makes the terminal unusable. The fix is to send the output somewhere else, which is redirection — Module 12's subject, previewed here because you will need it immediately:

./long_job.sh > run.log 2>&1 &

That sends normal output and error output to run.log and leaves your terminal clean. Then watch the file grow with the technique from Chapter 4.4:

tail -f run.log

Input is worse

Output barges in. Input stops the job dead.

$ bash ask.sh &
[1] 14389
$ name?
$ jobs
[1]+  Stopped                 bash ask.sh

The script tried to read from the terminal. A background job is not allowed to do that — the foreground has the terminal — so the kernel suspended it. The prompt it managed to print is sitting there looking like a question nobody will answer, and the job is frozen.

fg fixes it, because bringing the job to the foreground gives it the terminal:

$ fg %1
bash ask.sh
engineer
hello engineer

So: a background job that shows Stopped for no obvious reason is probably waiting for input. Bring it forward, answer it, and background it again if you still want to.

8. The PID of the Last Background Job: $!

Job specs work interactively. In a script there are no job numbers worth tracking, and you want the PID:

$ sleep 300 &
[1] 11407

$ echo $!
11407

$! is the PID of the most recently backgrounded process. It is a companion to $$ (this shell) and $PPID (its parent) from Chapter 7.1, and it is the standard way to keep hold of something you started:

./long_job.sh > run.log 2>&1 &
JOB_PID=$!
echo "started as $JOB_PID"

Capture it immediately. $! changes the moment anything else is backgrounded.

Checking whether it is still alive

With a PID in hand, the existence check is a signal that does nothing:

$ kill -0 11407 && echo "still running"
still running

$ kill 11407
$ kill -0 11407
bash: kill: (11407) - No such process

kill -0 sends no signal at all. It performs the permission and existence check and then stops, so it succeeds if the process exists and you could signal it, and fails otherwise. Chapter 7.6 explains why "signal zero" is spelled that way.

Waiting for it deliberately

wait makes the shell block until a background job finishes — the foreground behaviour, on demand:

$ sleep 5 &
[1] 11407
$ wait $!
$ echo "and now it has finished"
and now it has finished

With no argument, wait waits for all background jobs. That is the standard ending for a script that starts several things at once:

for seed in 1 2 3 4; do
  ./run_sim.sh "$seed" > "sim_$seed.log" 2>&1 &
done
wait
echo "all four finished"

Four simulations in parallel, then one line that blocks until the last of them is done. (The for loop is Chapter 17.1, and Chapter 17.4 is why you should use wait here rather than poll; wait also reports each job's exit status, which is Module 11.)

9. Jobs Belong to One Shell

A point that saves confusion later: the job table lives inside a single shell process.

• jobs in one terminal never shows another terminal's jobs.
• A job that was [1] in a shell you have closed is not [1] anywhere.
• Jobs cannot be moved between shells, and there is no command to adopt one.

The PID is the durable identifier; the job number is a convenience local to one shell's lifetime. Which is also a hint about what happens when that shell goes away — the subject of Chapter 7.8.

10. The Whole Set, in One Session

$ sleep 120 &
[1] 11407

$ sleep 200 &
[2] 11409

$ jobs
[1]-  Running                 sleep 120 &
[2]+  Running                 sleep 200 &

$ kill -STOP %1
[1]+  Stopped                 sleep 120

$ jobs
[1]+  Stopped                 sleep 120
[2]-  Running                 sleep 200 &

$ bg %1
[1]+ sleep 120 &

$ jobs
[1]-  Running                 sleep 120 &
[2]+  Running                 sleep 200 &

$ fg %2
sleep 200
^Z
[2]+  Stopped                 sleep 200

$ jobs
[1]-  Running                 sleep 120 &
[2]+  Stopped                 sleep 200

$ kill %1 %2

$ jobs
[1]-  Terminated: 15          sleep 120
[2]+  Terminated: 15          sleep 200

$ jobs
$

Worth noticing in that transcript:

• The + moved. Stopping job 1 made it the current job; resuming it handed + back to job 2.
• fg %2 printed sleep 200 before taking over the terminal.
• ^Z is echoed by the terminal when you press Ctrl-Z.
• The Terminated lines appear once and then the entry is gone. Bash reports each job's end exactly once; the second jobs is empty.

Practice

Do these at a real interactive prompt. Job control behaves differently inside a script, so a terminal is the only place these work.

1 — Feel the foreground. Run sleep 20 with no & and try to type. Then run sleep 20 & and notice the difference.

2 — The sequence that matters. Run sleep 300, wait a moment, press Ctrl-Z, then type bg, then jobs. Do it until it is muscle memory.

3 — Juggle three. Start three sleeps with different durations. Use jobs to watch the + and - markers move as you fg and Ctrl-Z different ones.

4 — Name a job by its command. Start sleep 500 & and sleep 600 &, then bring the second forward with fg %?600 rather than by number.

5 — Catch the output problem. Run this and try to type while it works:

(for i in 1 2 3 4 5; do echo "line $i"; sleep 1; done) &

Then run the same thing redirected to a file and watch it with tail -f instead.

6 — Catch the input problem. Put read -p 'name? ' n in a script, run it with &, and watch it stop. Fix it with fg.

7 — Capture a PID and poll it.

sleep 30 &
PID=$!
kill -0 "$PID" && echo "alive"
sleep 31
kill -0 "$PID" 2>/dev/null || echo "finished"

8 — Run four things at once. Use the for loop from §8 with four sleep commands of different lengths and a wait at the end. Time the whole thing; it should take as long as the longest, not the sum.

9 — Meet the exit warning. Suspend a job with Ctrl-Z, then type exit. Read what Bash says.

Check Your Understanding

1. You background a forty-minute job and jobs says Stopped. How much of the forty minutes is passing?

2. What is the difference between kill %1 and kill 1?

3. You start a job with & and your terminal fills with its output. What should you have done?

4. A background job shows Stopped and you did not press Ctrl-Z. What is the most likely explanation?

5. In a script you write ./job.sh & on one line and PID=$! three lines later, with another & command in between. What is in PID?

6. You close the terminal that has two background jobs in it. Does jobs in a new terminal show them?

This matters for more than convenience: Chapter 27.7 measures a background simulation that survived being killed, because the signal went to the subshell rather than to the tool.

Chapter 29.5 builds the timeout this machine does not have, and measures three run shapes rather than two — including the one that exits 0 in under a second with no verdict.

Chapter 30.1 measures what nohup actually guarantees: it protected sleep and a shell script and did not protect the simulator.

11. Remember

• The shell runs one foreground command and waits, which is why a long command appears to lock the terminal.
• A job has three states: foreground (shell waiting), background (running, prompt yours), stopped (not running).
• & starts in the background. The shell prints [job] PID — two different numbers.
• A job number is local to one shell; a PID is system-wide. Job specs need %. kill %1 and kill 1 are not the same command.
• Ctrl-Z suspends the foreground job; bg resumes it in the background; fg brings it forward. The Ctrl-Z-then-bg sequence is the most useful thing here.
• jobs lists them; jobs -l adds PIDs; + marks the current job and - the previous.
• %?text names a job by its command line, which beats counting.
• A stopped job makes no progress at all. Check the state column before walking away, and heed the "There are stopped jobs" warning.
• Background does not mean detached from the terminal. Output still barges in; redirect it to a file.
• A background job that reads from the terminal is suspended automatically. Unexplained Stopped usually means it wants input.
• $! is the PID of the last background job. Capture it on the very next line.
• kill -0 PID tests whether a process exists without signalling it.
• wait blocks until background jobs finish, which is how a script runs several things in parallel and still ends in the right order.
• Jobs belong to one shell and do not transfer between terminals.

You can now start, stop, resume and track work from a single terminal. Every one of those operations was really the same thing underneath — sending a signal — and the next chapter is about what a signal actually is, which turns out to explain why Ctrl-C sometimes does nothing at all.