Finding What Is Filling the Disk

Part II · Module 28 · Chapter 28.3 · Finding What Is Filling the Disk

Chapter 28.2 inventoried one run. The operational question is never about one run — it is "the filesystem is at 100% and three people's regressions are failing." That question has two tools, and the first thing to understand is that they are not two ways of measuring the same quantity.

1. The Two Commands, and What Each One Asks

$ df -h .
$ du -sh */ | sort -h | tail

df asks the filesystem how much space it has. It is authoritative about capacity and knows nothing about which directory is responsible.

du walks the directory tree and adds up what it finds. It is authoritative about your files and knows nothing about anything it cannot reach by name.

That difference sounds academic until it costs you an afternoon, which §4 measures. Start with the simpler surprise.

2. du Counts Blocks, Not Bytes

Two directories, deliberately constructed: 200 files of two bytes each, and one file of 400 KB.

$ for d in disk/small disk/one; do
>   bytes=$(find "$d" -type f -exec stat -f '%z' {} \; | awk '{s+=$1} END{print s}')
>   blocks=$(du -sk "$d" | awk '{print $1}')
>   n=$(find "$d" -type f | wc -l | tr -d ' ')
>   printf '  %-12s %4s files  %8s bytes of content  %6s KB on disk\n' "$d" "$n" "$bytes" "$blocks"
> done
  disk/small     200 files       400 bytes of content     800 KB on disk
  disk/one         1 files    409600 bytes of content     400 KB on disk

400 bytes of content occupying 800 KB — a factor of 2048. The cause is per-file allocation:

$ stat -f '%z bytes of content, %b blocks' disk/small/f1.txt
2 bytes of content, 8 blocks
$ echo $((8 * 512))
4096

Every file gets at least 4096 bytes of disk whatever is in it. Two hundred of them is 800 KB regardless of content.

Now apply the registry's own recommended command and watch what it reports:

$ cd disk && du -sh */ | sort -h
400K	one/
800K	small/

du ranks small/ as twice the size of one/, while one/ holds a thousand times more data. Both statements are true about different questions: small/ really is consuming 800 KB of the filesystem, and one/ really does contain all the data. For the question "what do I delete to free space", du is the right answer. For "where is my data", it is misleading by three orders of magnitude.

One piece of good news: sort -h works here, so the registry's pipeline is sound.

$ printf '1K\n2G\n500M\n' | sort -h
1K
500M
2G

3. The Working Pipeline

# the biggest consumers under here, largest last
du -sh */ | sort -h | tail

# capacity, and whether the problem is yours at all
df -h .

Two refinements worth having, both of which matter on a shared engineering filesystem.

Stay on one filesystem. A project area often has other mounts beneath it, and du will happily walk into them and attribute somebody else's storage to your directory:

du -shx */ | sort -h | tail        # -x: do not cross mount points

Count the files as well as the bytes. §2 is the reason: on a filesystem with a fixed inode table — ext4, or most NFS exports — a directory can exhaust the inodes while using very little space, and du -sh will report the area as nearly empty right up to the point where nothing can create a file. A regression writing four files per run across two hundred thousand runs is an inode problem, not a byte problem.

On this machine df already carries the inode columns, and APFS does not have a fixed table:

$ df -h .
Filesystem      Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s5   460Gi   342Gi    84Gi    81%    1.9M  876M    0%   /System/Volumes/Data

876M free inodes at 0% used, so there is nothing to exhaust here and the measurement below is from the byte side only. On a project filesystem that does have a fixed table, that %iused column is the one to watch, and it can read 100% while Capacity reads 40%.

4. When du and df Disagree by 400 MB

This is the case to recognise, because the symptom is "the disk is full and I cannot find anything", and every minute spent re-running du is wasted.

A 400 MB file is created, held open by a process, and then unlinked — its name removed while the descriptor is still open:

$ avail() { df -k . | awk 'NR==2 {print $4}'; }
$ avail
90115548
$ dd if=/dev/zero of=disk/held.bin bs=1m count=400
$ avail
89705948
$ ( exec 9< disk/held.bin; sleep 20 ) &   # hold it open
$ rm -f disk/held.bin                      # remove the NAME
$ avail ; du -sk disk
89705912
1204	disk

df says 409,636 KB are gone. du says the directory holds 1,204 KB. Neither is wrong:

• du walks names. The name is gone, so there is nothing for du to count.
• df asks the filesystem. The blocks are still allocated, because a file is only freed when its last link and descriptor are both gone.

The space comes back the instant the holder exits:

$ wait %1
$ avail
90115548          # exactly the original figure; 409,636 KB reclaimed

Finding the culprit requires a tool that looks at open descriptors rather than at names:

$ lsof -p "$holder"
COMMAND   PID   USER   FD   TYPE DEVICE  SIZE/OFF      NODE NAME
bash    57137   ...     9r   REG   1,14 314572800  83751164 /…/m28/disk/held.bin

The SIZE/OFF column still reads 314,572,800 and the path is printed even though nothing is there any more. On Linux the same entry is usually marked (deleted); the technique is identical.

5. A Report That Answers the Right Question

Putting it together: capacity first, then the biggest consumers, then the open-but-deleted check — in that order, because each one is only worth doing if the previous answer was unsatisfying.

#!/bin/bash
# diskreport [DIR] — capacity, the biggest consumers, and the invisible case.
set -u
dir=${1:-.}
cd "$dir" 2>/dev/null || { echo "diskreport: cannot enter $dir" >&2; exit 2; }

printf '== capacity ==\n'
df -h . | awk 'NR<=2'

printf '\n== biggest directories here (one filesystem only) ==\n'
du -shx */ 2>/dev/null | sort -h | tail -8

printf '\n== content bytes vs disk blocks, for the top one ==\n'
top=$(du -skx */ 2>/dev/null | sort -n | tail -1 | awk '{print $2}')
if [ -n "${top:-}" ]; then
  c=$(find "$top" -type f -exec stat -f '%z' {} \; 2>/dev/null | awk '{s+=$1} END{print s+0}')
  k=$(du -skx "$top" | awk '{print $1}')
  n=$(find "$top" -type f 2>/dev/null | wc -l | tr -d ' ')
  printf '%s %s files, %s bytes of content, %s KB on disk\n' "$top" "$n" "$c" "$k"
  if [ "$c" -gt 0 ] && [ "$((k * 1024 / c))" -ge 4 ]; then
    printf 'NOTE block overhead is %sx -- a many-small-files problem, not a data problem\n' "$((k * 1024 / c))"
  fi
else
  printf '(no subdirectories here)\n'
fi

printf '\n== deleted-but-open files, LARGEST FIRST ==\n'
# lsof +L1 lists every file whose link count is 0. On a live machine that is
# mostly harmless cache churn, so rank by size and show only what could
# plausibly explain a df/du gap.
if command -v lsof >/dev/null 2>&1; then
  out=$(lsof +L1 2>/dev/null \
        | awk 'NR>1 && $7+0 > 1048576 {printf "%12d  %-10s %-7s %s\n", $7, $1, $2, $NF}' \
        | sort -rn | head -5)
  if [ -n "$out" ]; then
    printf '%12s  %-10s %-7s %s\n' BYTES COMMAND PID PATH
    printf '%s\n' "$out"
  else
    printf 'none over 1 MB (or lsof lacks the privilege to see them)\n'
  fi
else
  printf 'lsof not available -- if df and du disagree, this is the gap\n'
fi

Run while a 200 MB file is held open with its name removed:

$ diskreport disk
== capacity ==
Filesystem      Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s5   460Gi   342Gi    84Gi    81%    1.9M  876M    0%   /System/Volumes/Data

== biggest directories here (one filesystem only) ==
4.0K	tiny/
400K	one/
800K	small/

== content bytes vs disk blocks, for the top one ==
small/ 200 files, 400 bytes of content, 800 KB on disk
NOTE block overhead is 2048x -- a many-small-files problem, not a data problem

== deleted-but-open files, LARGEST FIRST ==
       BYTES  COMMAND    PID     PATH
   209715200  sleep      63185   /…/m28/disk/held3.bin
    11616256  Spotlight  716     /…/com.apple.LaunchServices-20971544-v2(c84908a3~).csstore
    11550720  com.apple  37053   /…/com.apple.LaunchServices-20971544-v2(7828996e~).csstore
    11550720  Finder     643     /…/com.apple.LaunchServices-20971544-v2(7828996e~).csstore
    11517952  siriactio  60751   /…/com.apple.LaunchServices-20971544-v2(5c024c12~).csstore

The 200 MB file is at the top, named, with the process holding it. After the holder exits it is gone from the list and only the system's own cache churn remains.

Three deliberate choices in that script.

du -shx, with -x. Without it the report attributes other mounts to your directory, which on a shared engineering filesystem is how you end up investigating storage you do not own.

The overhead ratio is computed and reported as a NOTE. The number distinguishes the two problems that look identical in a du listing — too much data, and too many files — and they have completely different fixes. A NOTE rather than a warning, because 2048× is a fact about the directory, not a fault.

The lsof stage ranks, filters, and degrades honestly. lsof +L1 is exactly the deleted-but-open set; the size ordering and the 1 MB floor are what make it readable, per the note above. And when lsof is absent or unprivileged the script says it could not check rather than implying there is nothing to find — the same rule Chapter 27.7 applied to the unreadable FST.

6. Reading the Answer

The pipeline gives a number; the decision needs one more step. Three cases, distinguished by what the report said:

A few huge files. The overhead ratio is near 1 and a handful of names dominate. This is the normal case in a simulation flow, the files are waveform dumps, and the answer is Chapter 27.6's windowing plus Chapter 28.4's age rules.

Enormous numbers of small files. The overhead ratio is large and no single name stands out. The space is in allocation, not data, and deleting the largest files will barely move the number. The fix is fewer runs retained, or archiving them into one file per run — Chapter 28.5 — which converts thousands of 4 KB allocations into one.

df is full and du finds nothing. §4. Stop running du, find the process, and either let it finish or stop it properly.

And one case that is not yours: if df shows the filesystem full and your own area is small, the correct action is to say so to whoever owns the area that is large, rather than deleting your own evidence to make room. Chapter 22.5 is the etiquette; the technical point here is that df -h . tells you immediately whether the problem is yours at all, which is why it comes first in the report.

Practice

1. Create 200 two-byte files and one 400 KB file in separate directories. Compare du -sk against the summed stat -f '%z' for each.
2. Compute the ratio for both directories and explain which question each number answers.
3. Find the per-file allocation floor on your filesystem with stat -f '%b'.
4. Run du -sh */ | sort -h on those two directories and note which one it ranks larger.
5. Try du --apparent-size, then du -A -k -s, then the stat sum, on a one-byte file. Record all three.
6. Check the %iused column on your project area and work out how many runs it would take to exhaust a fixed inode table.
7. Create a large file, hold it open, and unlink it. Record df available and du before, during and after.
8. Find it with lsof -p and identify the column that still shows its size.
9. Confirm the space returns when the holder exits, and that the figure matches exactly.
10. Write diskreport and run it. Check that -x prevents it from crossing a mount point.
11. Make the overhead NOTE fire by pointing it at a directory of small files, then at one large file.
12. Run lsof +L1 | head -6 and then the size-ranked version, with a large deleted-but-open file present. Check whether the first one finds it.
13. Remove lsof from your PATH and confirm the report says it could not check rather than reporting nothing found.

Check Your Understanding

1. What does df ask, and what does du ask?
2. 400 bytes of content occupied 800 KB. Why?
3. du -sh */ | sort -h ranked a 400-byte directory above a 400 KB one. Is that wrong?
4. What is du -A actually reporting, and why is it not the content size?
5. What is available on this machine for an exact byte count?
6. Why does sort -h matter in that pipeline, and does it work here?
7. df said 409,636 KB gone and du said 1,204 KB. Which was right?
8. What has to be true for a file's blocks to be freed?
9. Which tool finds a deleted-but-open file, what option lists them, and why is the unsorted output unusable?
10. How does this situation arise naturally in a simulation flow?
11. Why does diskreport use -x?
12. What are the three cases the report distinguishes, and which one is not your problem?

7. Remember

• A run that fills the disk while it is running is a different problem — Chapter 34.5 measures what survives it, and finds the log that remains was written by the shell rather than the tool.
• df asks the filesystem; du walks names. They answer different questions and can both be right.
• du counts allocated blocks. 400 bytes of content occupied 800 KB — 2048× — because every file gets at least 4096 bytes.
• So du can rank a 400-byte directory above a 400 KB one, correctly, for the question it is answering.
• du -A is not content size — it quantises to 512 bytes — and --apparent-size does not exist here. For bytes, use stat.
• sort -h works on this sort, so du -sh */ | sort -h | tail is sound.
• Use -x so the report does not cross mount points, and watch %iused because a fixed inode table exhausts independently of bytes (not on APFS, which has none).
• df said 400 MB gone and du said 1.2 MB — a file whose name had been removed while a process still held it open. Blocks are freed when the last link and the last descriptor are gone.
• lsof +L1 finds them — ranked by size, with a floor. Unsorted, the system's own 11 MB cache churn hid a 200 MB file from a head -6. When df and du disagree, stop running du.
• This happens in real flows whenever a run directory is deleted under a job that is still writing.
• If the filesystem is full and your area is small, it is not your problem — df -h . tells you that first.

8. What Is Next

The space has been found and most of it is old runs. Chapter 28.4 deletes them — which is the one operation in this track that cannot be undone, so it is also where the guards matter most. Including the one that makes rm -rf "$RUNS"/* expand to the entire root.