Ending a Process Properly: kill, and Why -9 Comes Last

Part I · Module 7 · Chapter 7.7 · Ending a Process Properly: kill, and Why -9 Comes Last

Chapter 7.6 established what a signal is and proved that most of them are requests. This chapter is about the decision that follows: which one to send, when to send the next, and what the shortcut costs.

1. The Command Is Misnamed

kill 4821

kill does not kill anything. It sends a signal, and with no option it sends SIGTERM — the polite one. What happens next is the program's business.

The name is a historical accident and it has done real damage to how people think about the command, because it suggests a single violent act where the reality is a conversation. kill -STOP, kill -CONT and kill -0 from the last two chapters all use the same command to do things that are not killing at all.

kill 4821            # send SIGTERM — "please stop"
kill -TERM 4821      # identical, written explicitly
kill -KILL 4821      # send SIGKILL — "stop, no discussion"
kill -9 4821         # identical, by number
kill -STOP 4821      # suspend
kill -CONT 4821      # resume
kill %1              # the job spec from Chapter 7.5

2. The Escalation

The correct sequence has three steps, and the middle one is the one people skip.

Step 1 — ask.

$ kill 4821

Most programs exit promptly. Many do a few seconds of work first.

Step 2 — wait, and check.

$ sleep 5
$ kill -0 4821 2>/dev/null && echo "still running" || echo "gone"
gone

This is the step that gets skipped, and skipping it is what makes people believe SIGTERM does not work. Five seconds is usually plenty. A simulator writing a large waveform database may legitimately want thirty.

Step 3 — insist, only if you must.

$ kill -9 4821

Guaranteed, immediate, and unconditional — at a price.

Why the habit is so hard to break

kill -9 works immediately, every time. kill sometimes appears to do nothing for several seconds. The feedback loop teaches exactly the wrong lesson, and the costs above are invisible at the moment you pay them — you discover the truncated database tomorrow and the held licence when a colleague complains.

The fix is procedural rather than intellectual: always type kill first, and only reach for -9 when you have actually checked and it is still there.

3. Watching the Difference

Two scripts, identical except for one line.

#!/bin/bash
# tidy.sh
trap 'echo "[cleaning up]"; rm -f /tmp/work.lock; echo "[done]"; exit 0' TERM
touch /tmp/work.lock
echo "working as PID $$"
while true; do sleep 1; done

With SIGTERM:

$ ./tidy.sh &
[1] 13401
working as PID 13401

$ ls /tmp/work.lock
/tmp/work.lock

$ kill 13401
[cleaning up]
[done]
[1]+  Done                    ./tidy.sh

$ ls /tmp/work.lock
ls: /tmp/work.lock: No such file or directory

The handler ran. The lock file is gone.

With SIGKILL:

$ ./tidy.sh &
[1] 13455
working as PID 13455

$ kill -9 13455
[1]+  Killed: 9               ./tidy.sh

$ ls /tmp/work.lock
/tmp/work.lock

The lock file is still there, and the next run of the tool will refuse to start because of it. The program had cleanup code; -9 meant it never ran.

That lock file stands for everything in the warning above. It is the cheapest possible demonstration of a cost that is usually much larger.

4. Killing the Parent Does Not Kill the Children

Chapter 7.4 proved this; here is why it matters when you are trying to stop something.

$ ps -u $USER -o pid,ppid,etime,cmd | head -4
  PID  PPID     ELAPSED CMD
 9981     1 02-04:11:32 /bin/bash ./nightly_regress.sh
12811  9981    01:22:04 python3 regress.py --seed 4
12834  9981    01:21:58 python3 regress.py --seed 5

$ kill 9981

$ ps -o pid,ppid,etime,cmd -p 12811,12834
  PID  PPID     ELAPSED CMD
12811     1    01:24:30 python3 regress.py --seed 4
12834     1    01:23:58 python3 regress.py --seed 5

The supervisor is gone. Both simulations are still running, re-parented to PID 1 — and now nothing is collecting their results or cleaning up after them. You have made the situation worse than before you started.

Ending a whole tree

There are three ways to end a group of related processes, in increasing order of bluntness.

Find the children first and work downward.

pgrep -aP 9981           # look at them
pkill -P 9981            # end the children
kill 9981                # then the parent

Signal the process group. A kill target that is negative means a process group rather than a process:

kill -TERM -9981

That signals every process in group 9981. The - prefix is the whole syntax, and because it looks exactly like an option, use -- when the group ID could be misread:

kill -TERM -- -9981

A job started from your shell gets its own process group, which is why kill %1 from Chapter 7.5 ends a whole pipeline rather than one member of it — job specs signal the group, PIDs signal one process. That difference is quietly important and is the reason kill %1 sometimes appears to do more than kill <pid>.

Match on the command line. Which is pkill, and it needs its own section.

5. pkill, and the Habit That Saves You

pkill takes exactly the options pgrep does from Chapter 7.2, and sends a signal to everything that matches instead of printing it.

pkill -f run_sim          # SIGTERM to everything matching
pkill -9 -f run_sim       # SIGKILL to the same set
pkill -u engineer python3 # every python3 owned by engineer
pkill -P 9981             # every direct child of 9981

It is the right tool when you have forty processes to end and listing their PIDs by hand is absurd. It is also the single easiest way to end work that is not yours.

The anchoring problem

pkill patterns are regular expressions, and an unanchored one matches anywhere:

$ pgrep -af sim
12811 python3 simulate.py --seed 4
13044 vim simple_notes.txt
13102 /opt/tools/simvision -gui

Three matches, one of which is an editor with unsaved work in it. Anchor the pattern to the start, or use enough of the command line to be unambiguous:

pgrep -af '^python3 simulate\.py'

6. When It Still Does Not Die

A decision path, in order. Each step is a command you have already met.

1 — Does it still exist?

kill -0 "$PID" 2>/dev/null && echo "still there" || echo "already gone"

Half the time the answer is that it ended and you were looking at a stale ps listing.

2 — Is it yours?

$ kill 4821
bash: kill: (4821) - Operation not permitted

You can only signal processes you own, unless you are root. That message means the process belongs to somebody else — and the right response is to find out whose it is and talk to them:

ps -o pid,user,etime,cmd -p 4821

3 — Is it already dead?

$ ps -o pid,stat,comm -p 12901
  PID STAT COMM
12901 Z    defunct

Z is a zombie, and Chapter 7.4 explained it: already finished, waiting for a parent that is not collecting. Signalling it does nothing, because there is nothing running. Deal with the parent named in the PPID column.

4 — Is it stuck on I/O?

$ ps -o pid,stat,comm -p 4821
  PID STAT COMM
 4821 D    simulator

D is uninterruptible sleep. Not even -9 will act until the I/O completes (Chapter 7.6). This is not a program you can fix by signalling it harder; it is a storage or network-filesystem problem, and the signal you have already sent will be delivered when the wait ends.

5 — Did it respawn?

A PID that changes each time you look means something is restarting it — a service manager, a supervisor script, a wrapper. Find the parent and stop that:

ps -o pid,ppid,cmd -p "$(pgrep -f the_thing | head -1)"

6 — Only now, -9.

kill -9 "$PID"

Having established that it exists, is yours, is not a zombie, is not blocked on I/O and is not being restarted — this will work, and you know what it is costing you.

7. A Worked Shutdown

A regression needs to stop. Done properly:

$ pgrep -af regress
 9981 /bin/bash ./nightly_regress.sh
12811 python3 regress.py --seed 4
12834 python3 regress.py --seed 5

Three processes: a supervisor and two workers. Stop the workers first, so the supervisor can record what happened:

$ pkill -u "$USER" -f 'regress\.py'

$ sleep 5

$ pgrep -af regress
 9981 /bin/bash ./nightly_regress.sh

The workers are gone and the supervisor is still there, which is what you wanted — it had the chance to notice and write its summary. Now stop it:

$ kill 9981

$ sleep 3

$ pgrep -af regress
$

Everything gone, in order, with every program given its opportunity to finish properly. Four commands, about ten seconds, and no truncated databases or held licences.

Compare with pkill -9 -f regress, which takes one command, finishes instantly, and leaves you with two unreadable waveform files, two checked-out licences, a lock file that blocks tomorrow's run, and no record of how far the regression got.

Practice

1 — Prove the handler runs. Build tidy.sh from §3. Run it, kill it, and confirm the lock file is removed. Run it again, kill -9 it, and confirm the lock file survives.

2 — Measure how long a clean shutdown takes. Give the handler a sleep 5 before its exit, then time the gap between your kill and the job reporting Done. That interval is what you are throwing away with -9.

3 — Build the escalation as one command.

kill "$PID"
sleep 5
kill -0 "$PID" 2>/dev/null && { echo "did not stop; escalating"; kill -9 "$PID"; }

Try it against a script that handles TERM by exiting, and then against one that ignores it.

4 — Reproduce the orphan problem. Write a parent script that backgrounds two long children. Kill only the parent. Confirm with ps -o pid,ppid that the children survive with PPID 1.

5 — Then do it properly. Same setup, but pkill -P <parent> first, then kill the parent.

6 — See what an unanchored pattern catches. Start sleep 500 & and open a file called sleepy_notes.txt in an editor in another terminal. Run pgrep -af sleep and look at what a pkill -f sleep would have hit.

7 — Signal a process group. Start a pipeline in the background (sleep 500 | cat &), then compare kill <pid of sleep> with kill %1. The job spec ends both members.

8 — Try to kill something that is not yours. Pick any PID owned by another user from ps -ef and run kill on it. Read the error.

Check Your Understanding

1. Why does kill 4821 sometimes appear to do nothing for several seconds, and what is the correct response?

2. A simulation ends with kill -9. The next morning the waveform file will not open and a licence is still checked out. Explain both.

3. You kill the shell script supervising six simulations. What is the state of the machine afterwards?

4. What is the difference between kill 9981 and kill -TERM -9981?

5. You run pkill -f test and a colleague's editor closes. What two things should you have done?

6. kill -9 4821 returns no error and the process is still in ps a minute later. What is the explanation?

8. Remember

• kill sends a signal; it does not necessarily kill. With no option it sends SIGTERM.
• Escalate: kill, wait and check, then -9 only if you must. The middle step is the one people skip and the reason they believe SIGTERM does not work.
• kill -9 costs you everything the program had not finished — unflushed buffers, truncated databases, leftover temporary files, held licences, stale lock files.
• A program that pauses before exiting on SIGTERM is behaving well. Give it a few seconds.
• Killing a parent does not kill its children. They are orphaned and re-parented to PID 1, now unsupervised.
• Stop children first, then the parent, so the supervisor can record what happened.
• A negative target is a process group: kill -TERM -- -9981. A job spec (kill %1) signals the whole group too; a bare PID does not.
• pkill takes pgrep's options and acts instead of printing. Run pgrep -af first, read it, then change one letter.
• Add -u "$USER" so a wrong pattern cannot reach anyone else's work.
• Patterns are unanchored regular expressions. sim matches simvision, simple_notes.txt and an editor.
• Operation not permitted means it is not yours. Find the owner and talk to them.
• A zombie cannot be signalled; deal with its parent.
• A process in state D ignores even -9 until its I/O completes.
• Prefer pkill to killall, whose meaning differs between systems.

You can now end anything you started, in the right order, for the right reasons. The last chapter of the module turns the question around: how do you start something that keeps running — after the connection drops, after you log out, after the terminal is gone?