Live Load: Reading CPU and Memory

Part I · Module 7 · Chapter 7.3 · Live Load: Reading CPU and Memory

Chapter 7.2 gave you a snapshot: what exists, who owns it, how long it has been there. That answers what is running. It does not answer the question people actually ask, which is whether the machine is in trouble and whether it is your fault.

1. Three Illnesses, One Symptom

A slow machine is a symptom, not a diagnosis. There are three common causes, and they want opposite responses:

• Too much work. The processors are genuinely saturated. Nothing is broken; there is simply more to do than there is machine. The response is to run fewer things, or wait.
• Not enough memory. The machine has run out and is shuffling pages between memory and disk instead of computing. The response is to stop something — and critically, starting anything else makes it strictly worse.
• Something is stuck. A process is blocked waiting on a disk or a network filesystem. The processors are mostly idle. The response is usually to look at the storage, not the program.

The reason to be deliberate about this is that the instinctive response to a slow machine — "I'll just start my job anyway, it's only one more" — is correct for the first case, harmless for the third, and actively destructive for the second.

2. The Cheapest Question: uptime

One command, no options, instant:

$ uptime
 13:13  up 34 days, 15 mins, 1 user, load averages: 5.25 4.51 8.94

Four pieces of information:

• 13:13 — the current time.
• up 34 days, 15 mins — how long since the machine booted.
• 1 user — how many login sessions are open. Not how many people; one person with four terminals open may count as four.
• load averages: 5.25 4.51 8.94 — the three numbers that matter.

The uptime figure is worth a glance of its own. A shared machine showing up 12 mins when it was up yesterday means it rebooted, which explains an awful lot of "my job vanished" reports in one number.

3. What a Load Average Actually Measures

This is the number people quote most and understand least, so it is worth being precise.

The load average is the average number of processes that wanted to run but could not — plus, on Linux, those blocked waiting on disk. It is not a percentage. It has no maximum. A load of 5.25 does not mean 5.25% of anything.

The three numbers are averages over one minute, five minutes and fifteen minutes, in that order. So the reading above says:

The rule that makes the number mean something

A bare load figure is meaningless. Compare it to the number of processor cores.

$ nproc
8

(nproc is the Linux command. On macOS the equivalent is sysctl -n hw.ncpu. lscpu gives the full picture on Linux if you want it.)

Eight cores means the machine can genuinely run eight things at once. So:

The reading above — 5.25 on eight cores — is a busy but healthy machine. The same 5.25 on a two-core machine would be a machine in serious difficulty.

Reading the trend

The three figures together tell you a direction, which a single figure cannot:

The real reading above is 5.25 4.51 8.94 — the one-minute figure well below the fifteen-minute one. The machine was heavily loaded a quarter of an hour ago and is recovering. If you were about to investigate a crisis, you have just discovered it already ended.

4. The Live View: top

uptime is a glance. top is the dashboard — a full-screen display that refreshes every few seconds.

$ top

The header is the part worth learning. On Linux it looks like this:

top - 13:13:22 up 34 days,  0:15,  1 user,  load average: 5.22, 4.53, 8.84
Tasks: 658 total,   5 running, 650 sleeping,   0 stopped,   3 zombie
%Cpu(s): 20.3 us, 75.3 sy,  0.0 ni,  4.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  31890.0 total,   1124.0 free,   8402.0 used,  22364.0 buff/cache
MiB Swap:   8192.0 total,   7680.0 free,    512.0 used.  21455.0 avail Mem

Line 1 is uptime, repeated. Nothing new.

Line 2 is a tally of the process states from Chapter 7.2. The numbers to look at are running (how many are actually working) and zombie. A handful of zombies is normal; hundreds means a program is failing to clean up after its children — Chapter 7.4 explains how that happens.

Line 3 is where the processor time went. Five of the eight fields matter:

The other three (ni, hi, si) are rarely the answer to anything.

The three diagnostic patterns:

• us high, id near zero — genuine computation. The machine is doing what you asked.
• sy very high — the kernel is working hard. Usually heavy I/O, enormous numbers of small system calls, or memory pressure.
• wa high — the processors are waiting for storage. This is the third row of Figure 1 and no amount of CPU will fix it.

The reading above — 20% user, 75% system, 4% idle — is a machine where the kernel is doing most of the work, which is unusual enough to be worth investigating.

Living in top

M and u together are the pair you will use most on a shared machine: show me, sorted by memory, only what belongs to me.

Press 1 once and the single %Cpu(s) line becomes one line per core. That immediately distinguishes "one single-threaded process pinning one core while seven sit idle" from "all eight cores saturated", which the averaged line cannot.

5. The Per-Process Columns

Below the header, top lists processes. Two columns there are misread often enough to be worth stopping on.

%CPU can exceed 100

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  311 engineer  20   0  2841.0m 248.5m  32.1m R 116.7   3.0  36:33.21 simulator

116.7 is not an error. %CPU is measured against one core, so a process using one and a sixth cores shows 116.7, and a program making full use of eight cores shows close to 800.

This is the number that tells you whether a tool is actually multi-threaded. A simulator sitting at 99.8 on a sixteen-core machine is using one core, and buying a bigger machine will not help it.

RES is the memory number; VIRT is not

A process showing VIRT of 300 gigabytes on a 32-gigabyte machine is not a crisis; it has merely reserved address space. RES is the column that competes for real memory. (ps calls the same figure rss.)

6. Memory, and the Mistake Everyone Makes

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       8.2Gi       1.1Gi       1.2Gi        22Gi        21Gi
Swap:         8.0Gi       512Mi       7.5Gi

-h means human-readable units, the same flag Chapter 3.6 used for file sizes.

Look at free: 1.1 GiB out of 31. Three per cent. It is very tempting to conclude the machine is out of memory.

It is not. The number to read is available: 21 GiB, two thirds of the machine, free for the taking.

The reconciliation is the buff/cache column — 22 GiB. That is memory Linux is using to cache files it has read from disk. It is a pure optimisation: the data is still on disk, and the instant any program needs memory, the kernel discards cache and hands it over, immediately and without warning anyone.

The column that is a real alarm

The Swap line is the one to worry about.

Swap is disk space the kernel uses as overflow when physical memory runs out. Pages that have not been touched recently get written out to disk; when something needs them again, they are read back. Disk is thousands of times slower than memory, so a machine doing this constantly spends its time shuffling pages instead of computing. The old name for it is thrashing, and it is the second row of Figure 1.

The distinction is between swap used and swap active. A steady 512 MiB is a historical fact. A figure that climbs while you watch is the machine actively struggling, and that is when starting another job is genuinely harmful.

7. The Whole Diagnosis in Four Commands

Put it together. A colleague says the machine is slow.

$ uptime
 13:13  up 34 days, 15 mins, 1 user, load averages: 22.14 21.80 19.02

$ nproc
8

Load 22 on 8 cores — nearly three times oversubscribed, and rising. Something is genuinely going on.

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       8.2Gi       1.1Gi       1.2Gi        22Gi        21Gi
Swap:         8.0Gi          0B       8.0Gi

21 GiB available, no swap in use. Memory is fine — this is not the second row of Figure 1.

Now, is it CPU or I/O? The %Cpu(s) line in top answers it. If id and wa are both near zero, the processors are genuinely busy; if wa is high, they are waiting on storage.

Assume the former, and find out whose:

$ ps -eo pid,user,%cpu,etime,cmd --sort=-%cpu | head -5
  PID USER       %CPU     ELAPSED CMD
12811 colleague  99.4    01:22:04 python3 regress.py --seed 4
12834 colleague  99.1    01:21:58 python3 regress.py --seed 5
12840 colleague  98.8    01:21:51 python3 regress.py --seed 6
  311 engineer  116.7       36:33 simulator top_tb

Row one of Figure 1, with a name on it. Someone is running many single-core jobs at once. Nothing is broken, nobody did anything wrong, and the correct response is a conversation rather than a kill.

And note the last row: 116.7 against the others' 99.4. That one process is multi-threaded; the rest are not.

Practice

1 — Establish your units. Run nproc (or sysctl -n hw.ncpu) and write the number down. Then run uptime and divide. That ratio, not the raw load, is the state of the machine.

2 — Make the load rise and watch the trend. Start one busy process per core and watch the one-minute figure climb while the fifteen-minute figure lags:

for i in $(seq 1 $(nproc)); do yes > /dev/null & done
uptime     # wait a minute, run it again, and again

Then stop them and watch it fall:

pkill yes

Note how slowly the fifteen-minute figure responds. That lag is the whole point of having three numbers.

3 — Prove cache is not used memory. Run free -h and note free and available. Read a large file (cat somebigfile > /dev/null), run free -h again. buff/cache will have grown and free shrunk — and available will be almost unchanged.

4 — Separate the cores. Open top, press 1. Start a single yes > /dev/null &. Watch exactly one core go to 100% while the rest stay idle, and compare that with what the averaged line showed.

5 — Sort by memory. In top, press M, then u and your username. That is the view to reach for when a machine is short of memory and you want to know your own contribution.

6 — Find the single-threaded tool. Run ps -eo pid,%cpu,cmd --sort=-%cpu | head. Anything pinned near 100.0 and never above it is using one core.

Check Your Understanding

1. A machine reports a load average of 12.0. Is that bad?

2. uptime shows load averages: 2.1 8.4 14.9. What happened, and should you investigate?

3. free -h shows 400 MiB free out of 64 GiB. A colleague wants to kill jobs to free memory. What do you check first?

4. Load is 38, the processors are 95% idle, and several processes show state D. What is wrong?

5. top shows your simulation at %CPU 99.7 on a 32-core machine. Would a 64-core machine make it finish sooner?

6. A long regression disappears overnight with nothing in its own log. Where do you look?

8. Remember

• "Slow" is a symptom with three common causes — too much work, not enough memory, and blocked I/O — and they call for opposite responses.
• A load average is not a percentage. It is the average number of processes wanting to run, and it has no maximum.
• Divide load by core count. nproc first, uptime second. Around 1.0 per core is saturation.
• The three figures are 1, 5 and 15 minutes, and comparing them gives you a direction. One-minute below fifteen-minute means the worst is over.
• On Linux, load includes processes blocked on disk, so a high load with idle processors is an I/O problem.
• top's %Cpu(s) line splits the time: us is program code, sy is kernel, id is idle, wa is waiting for storage. High wa is storage, not computation.
• In top: q quits, M sorts by memory, u filters by user, 1 splits the cores. Do not use k.
• %CPU above 100 is normal — it is measured per core. A process stuck at 99 on a big machine is single-threaded.
• RES is real memory; VIRT is address space and means almost nothing.
• In free -h, read available, never free. Low free is healthy — Linux spends idle memory on file cache and reclaims it the moment anything asks.
• Swap steadily climbing is the real memory alarm. Swap merely used is history; swap growing is the machine thrashing.
• The OOM killer ends processes silently. When a job vanishes with no error of its own, check dmesg.
• On a machine other people are using, the load number is a question about them — Chapter 22.5 divides it by the core count and turns it into a decision.
• Starting another job is fine on a busy machine and harmful on one out of memory. Knowing which you are looking at is the whole point of this chapter.

You can now read the machine as a whole. The next chapter goes back to individual processes and makes the parent-child relationship precise — what a child inherits, what it cannot change, and what happens to it when its parent dies.