Seeing What Is Running: ps and pgrep

Part I · Module 7 · Chapter 7.2 · Seeing What Is Running: ps and pgrep

Chapter 7.1 said a process carries five things. This chapter is about reading them off a live machine, which turns out to be less about memorising a command and more about resisting the temptation to look at everything at once.

1. The Default Is Almost Useless

Run ps on its own and you get a disappointment:

$ ps
  PID TTY           TIME CMD
11274 ttys000    0:00.42 /bin/zsh
11286 ttys000    0:00.03 bash
12688 ttys000    0:00.01 ps

Three rows, on a machine running six hundred processes. That is not a bug. Bare ps shows only the processes attached to your terminal — your shell, anything it is running, and ps itself, caught in the act of listing.

It is occasionally what you want. It is almost never what you came for.

2. Everything, With Parentage

The form you will actually type is ps -ef:

$ ps -ef | head -6
  UID   PID  PPID   C STIME TTY          TIME CMD
    0     1     0   0 Mar28 ?        123:17.90 /sbin/init
    0   318     1   0 Mar28 ?          0:14.22 /usr/lib/systemd/systemd-journald
  501   639     1   0 Mar28 ?          8:02.19 /usr/libexec/session-manager
  501  2168   639   0 Mar28 ?         30:21.04 /usr/share/code/code --type=renderer
  501 11274 24759   0 13:11 ttys000    0:00.42 /bin/zsh

Two options, doing two separate jobs:

• -e — every process, not just yours, not just this terminal's.
• -f — full format: add the columns that say who, from whom, and since when.

The | head -6 is the pipe from Chapter 4.3, keeping the output to a screenful. Without it you get six hundred lines and the top of the list scrolls away.

Reading the columns

Two of those are read wrongly often enough to be worth stopping on.

The STIME column is the other one. It shows a time for processes started today and a date for anything older, which means a column of mixed formats and no way to sort it sensibly. Again, etime is the better question.

3. Only Mine

On a shared machine the UID column is the first filter you want, and there is an option for it:

$ ps -u $USER -f | head -5
  UID   PID  PPID   C STIME TTY          TIME CMD
  501   208     1   0 Sep28 ?          0:00.13 /usr/bin/some-service
  501   639     1   0 Mar28 ?          8:02.19 /usr/libexec/session-manager
  501 11274 24759   0 13:11 ttys000    0:00.42 /bin/zsh
  501 11286 11274   0 13:11 ttys000    0:00.03 bash

$USER is an environment variable holding your login name, so ps -u $USER reads as "processes owned by me" without you having to type your own name. It works unchanged on any machine and in any script.

This is the form to reach for when a shared machine is struggling and you want to know, honestly, whether the problem is yours.

4. Choosing Your Own Columns

-f is a preset. When it shows the wrong things — and it usually shows at least one wrong thing — -o lets you name the columns yourself, in order:

$ ps -u $USER -o pid,etime,cmd | head -4
  PID     ELAPSED CMD
  208 08-19:41:47 /usr/bin/some-service
11274       02:14 /bin/zsh
11286       02:14 bash

Three columns, exactly the three asked for, in the order asked for. And the middle one is the column -f does not give you:

etime is elapsed wall-clock time — how long the process has actually been alive. The format is [[DD-]HH:]MM:SS, which reads as:

Short runs show two fields; long ones grow a field at a time from the left. A process showing 08- has been running for over a week, which on a shared machine is usually either a service or somebody's forgotten job.

The column names worth knowing

The last two deserve a side-by-side, because the difference between them drives the rest of this chapter:

$ ps -o pid,comm,cmd -p 12560
  PID COMM CMD
12560 bash /bin/bash ./run_sim.sh

One process, two descriptions. comm is bash — the name of the program that is executing. cmd is /bin/bash ./run_sim.sh — the whole command line, including the script being run.

For a shell script, those are completely different answers, and only one of them tells you what the process is doing. Hold that thought until §7.

5. A Process Has a State

The stat column answers a question you will have before long: is this thing working, waiting, or stuck?

$ ps -eo pid,stat,comm | head -5
  PID STAT COMM
    1 Ss   /sbin/init
  208 SN   bash
  318 Ss   /usr/lib/systemd/systemd-journald
  311 R+   next-server

The first letter is the state. The rest are modifiers.

So R+ is "running, in the foreground of its terminal" and Ss is "sleeping, and a session leader".

Two of these are worth a sentence each, because they get misread:

• S is not a problem. The overwhelming majority of processes on any machine are sleeping, because they are waiting for input, a timer or a network reply. A machine where everything is R is a machine in trouble.
• D usually is. A process in uninterruptible sleep is blocked inside the kernel waiting on hardware, and it will not respond to anything — including attempts to stop it — until that I/O completes. A process stuck in D for minutes generally means a sick disk or a hung network filesystem, not a sick program. Chapter 7.7 returns to why this one process cannot be killed.

Z — the zombie — belongs to Chapter 7.4, where the parent-child relationship explains both what it is and why it is usually harmless.

6. The ? in the TTY Column

Several rows above showed ? where a terminal should be. That means the process has no controlling terminal: nobody can type at it and its output is not going to a screen.

That is normal and expected for:

• system services, started at boot by PID 1, long before any terminal existed;
• anything deliberately detached from a terminal — which is Chapter 7.8's entire subject.

A row showing ttys000 or pts/3 is attached to a specific terminal; a row showing ? is not attached to any. When a job you started yesterday still shows your terminal's name, it is still tied to that terminal — and that fact will matter a great deal by the end of the module.

7. Finding One Process: pgrep

Everything so far has been about listing. In practice you rarely want a list — you want one row, about one thing you already have in mind. Piping ps into grep works, and for years it was the idiom:

$ ps -ef | grep sleep
  501 12547 11286   0 13:22 ttys000  0:00.00 sleep 300
  501 12702 11286   0 13:23 ttys000  0:00.00 grep sleep

Note the second row. grep found itself, because by the time ps listed the processes, the grep was one of them. That is a genuine nuisance: every such search returns one spurious row, and scripts that count the results get the wrong answer.

pgrep is the purpose-built tool, and it excludes itself by construction:

$ pgrep -l sleep
12547 sleep
12554 sleep

-l adds the name alongside the PID. Without it you get bare numbers, which is exactly what you want when feeding the result to another command.

The -f that changes everything

Here is the demonstration that makes pgrep worth learning properly. A script called run_sim.sh is running:

$ ./run_sim.sh &
[1] 12560

$ pgrep run_sim
$

Nothing. The script is running — you just started it — and pgrep cannot find it.

Nothing is broken. By default pgrep matches against the process name, which for a shell script is the name of the interpreter running it. §4 already showed what that is:

$ ps -o pid,comm,cmd -p 12560
  PID COMM CMD
12560 bash /bin/bash ./run_sim.sh

The process is called bash. run_sim.sh is an argument. So searching for run_sim by name fails, and searching for bash returns every shell on the machine.

-f matches against the full command line instead:

$ pgrep -f run_sim.sh
12560

Combining the two

On Linux, -a prints the full command line alongside each PID, which is what you want when checking you are about to act on the right thing:

$ pgrep -af python
12811 python3 regress.py --seed 4
12834 python3 regress.py --seed 5

Two runs of the same script, distinguishable only by their arguments — which is precisely the situation -f and -a exist for.

Other filters worth knowing

-P is quietly one of the most useful. "What did my shell start?" is a single command:

$ pgrep -P $$
12547
12560

8. Putting It Together

A realistic sequence. A shared machine is slow, and you want to know whether it is your fault.

Is anything of mine running at all, and for how long?

$ ps -u $USER -o pid,etime,stat,cmd --sort=-etime | head -5
  PID     ELAPSED STAT CMD
 9981 02-04:11:32 S    /bin/bash ./nightly_regress.sh
12811    01:22:04 R    python3 regress.py --seed 4
12834    01:21:58 R    python3 regress.py --seed 5
11286       12:04 Ss   bash

--sort=-etime is a GNU option putting the oldest first; the leading - reverses the order. On a BSD ps, pipe through sort instead.

The top row is the find: a script that has been running for over two days. Whether that is a problem depends on what it was supposed to do, which is the next question.

What is it, exactly, and who started it?

$ ps -o pid,ppid,user,etime,cmd -p 9981
  PID  PPID USER     ELAPSED CMD
 9981     1 engineer 02-04:11:32 /bin/bash ./nightly_regress.sh

PPID 1. Its original parent is gone and PID 1 has adopted it — so whatever terminal launched it closed long ago. Chapter 7.4 explains how that happens and Chapter 7.8 explains how to make it happen on purpose.

What did it start?

$ pgrep -af -P 9981
12811 python3 regress.py --seed 4
12834 python3 regress.py --seed 5

Two children, still working. The two-day-old script is not stuck; it is supervising a regression. Which is a completely different conclusion from the one you would have drawn from the elapsed time alone — and you reached it with four commands and no guessing.

Practice

1 — Count what is running. Compare ps -e | wc -l with ps -u $USER | wc -l. The ratio tells you how much of a shared machine is anybody's business but yours.

2 — Build the listing you will actually use. Type this out and keep it:

ps -u "$USER" -o pid,ppid,etime,stat,cmd

Then remove one column at a time and decide whether you miss it.

3 — See TIME and etime disagree. Start something that sleeps and something that works:

sleep 120 &
yes > /dev/null &
sleep 20
ps -o pid,time,etime,cmd -p $(pgrep -f 'sleep 120') -p $(pgrep -f 'yes')

Both have been alive for twenty seconds. Only one has used twenty seconds of CPU. Kill the yes when you are done — it will happily burn a core indefinitely.

4 — Reproduce the pgrep failure. Write a one-line script that does nothing but sleep 300, run it in the background, then try pgrep <scriptname> and pgrep -f <scriptname>. Explain the difference to yourself out loud before reading §7 again.

5 — Find your own children. Run pgrep -P $$ with nothing in the background, then start two sleep 300 & jobs and run it again.

6 — Catch a D state if you can. Run ps -eo stat | sort | uniq -c | sort -rn to tally every state on the machine. You will see mostly S. If D appears and stays, the machine is waiting on hardware.

Check Your Understanding

1. ps -ef shows a simulation with TIME of 0:03 after six hours. Is it stuck?

2. Why does ps -ef | grep regress almost always return one more row than you expected?

3. You run pgrep nightly_regress and get nothing, but ps -ef | grep nightly_regress clearly shows it running. What is the explanation, and what is the fix?

4. A row shows PPID 1 and TTY ?. What two things does that tell you?

5. You want the PIDs of every python3 process you own, formatted so you can pass them straight to another command. Which form gives you bare numbers and nothing else?

Chapter 30.3 puts these fields on a live simulation and finds RSS falling by 29% inside a perfectly healthy run — so one sample is a number, not a signal.

9. Remember

• Bare ps shows only your terminal's processes. ps -ef shows everything, with parents and command lines.
• -e is "every"; -f is "full". They are two independent options that happen to be typed together.
• PPID is the most useful column -f adds. Who started it answers more questions than what it is called.
• TIME is CPU time, not age. For age, ask for etime.
• etime reads [[DD-]HH:]MM:SS and grows a field at a time from the left, so a leading 08- means eight days.
• -o names your own columns in your own order, and is how you escape the presets.
• comm is the program name; cmd is the whole command line. For a script those are completely different answers.
• stat's first letter is the state: R running, S sleeping (normal), D uninterruptible I/O (usually a sick disk), T stopped, Z zombie. + means foreground, s means session leader.
• TTY of ? means no controlling terminal — a service, or something deliberately detached.
• pgrep beats ps | grep because it never matches itself.
• pgrep -f matches the full command line, and is the only form that finds a shell script, a Python module or anything run through a wrapper. In engineering work it is the common case.
• pgrep -P <pid> lists a process's direct children, and pgrep -P $$ lists what your own shell started.
• Always pgrep before you pkill. Same options, very different consequences.
• ps has two incompatible option traditions. The short forms here work everywhere; GNU long options such as --forest and --sort are Linux-only.

You can now find any process on the machine and read what it is doing. The next chapter changes the question from what is running to how hard is this machine working — telling apart a busy machine, a machine out of memory, and one process that has gone wrong.