Project Areas, Scratch and Quota

Part II · Module 22 · Chapter 22.4 · Project Areas, Scratch and Quota

Chapter 22.3 established that the filesystem is shared. This chapter is about the consequence: every byte you write is in someone else's space, and the directories you have access to are not interchangeable.

1. Four Kinds of Directory

They are distinguished by what happens to the data, not by where they sit in the tree:

The names differ at every company. The four behaviours do not, and the single most useful question when you are new is not "where is the code" but "which of these is which, here?"

Three failures follow from getting it wrong, and each is common enough to have a name on any team:

A regression writing waveforms into the project area. It fills, every engineer's compile starts failing at once, and the cause is one job from six hours ago.

A result you needed in scratch. It was fast, the run was fine, and the directory was deleted overnight by a policy you did not know existed.

And source in your home directory. It is backed up, so nothing is lost — but no compute job can read it with the permissions home directories usually have, so the regression that worked on your workstation fails everywhere else.

2. Asking the Filesystem Which Is Which

The policy is documentation, but three facts are measurable, and together they identify a directory's kind with reasonable confidence.

Is it local or network? From Chapter 22.3 §3: the word local in mount, or its absence.

How full is it, and how big? A 500 GB filesystem at 81% is a different proposition from a 50 TB one at 81%:

$ df -h .
Filesystem      Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s5   460Gi   342Gi    85Gi    81%    2.0M  887M    0%   /System/Volumes/Data

Avail is the number that matters, not Capacity. 85 GB free will not survive a hundred-job regression writing a gigabyte each, whatever the percentage says.

And is there a quota? The command is quota, and this is where Chapter 21.6 arrives again, because the useful flag is not portable:

$ quota -s
quota: illegal option -- s
Usage: quota [-guqv]
	quota [-qv] -u username ...
	quota [-qv] -g groupname ...

-s ("human-readable") is a GNU extension; the BSD quota here offers -guqv only. So the portable invocation is bare quota, or quota -v for per-filesystem detail, and a script that wants human units must do the arithmetic itself.

3. du and ls Disagree About the Same File

Before measuring anything, this has to be settled, because the numbers differ by three orders of magnitude in realistic cases.

ls -l reports the file's size in bytes. du reports how much disk it occupies. Those are not the same, because disk is allocated in blocks:

$ ls -l run
-rw-r--r--  1 engineer  staff    10 Oct  8 22:30 tiny.log
-rw-r--r--  1 engineer  staff  3000 Oct  8 22:30 wave.vcd
$ du -h run/*
4.0K	run/tiny.log
4.0K	run/wave.vcd
$ stat -f "%N size=%z blocks=%b blksize=%k" run/tiny.log run/wave.vcd
run/tiny.log size=10 blocks=8 blksize=4096
run/wave.vcd size=3000 blocks=8 blksize=4096

A ten-byte file costs four kilobytes. It cannot cost less; the block size is the minimum unit. Which gives the first practical rule of this chapter:

Ten thousand tiny files cost 40 MB of disk to hold 100 KB of data — and they cost far more than that in metadata operations on a network filesystem. A flow that writes a separate status file per test is choosing this. Module 28 is where workspace hygiene gets its own module; the arithmetic belongs here.

The reverse case is just as large. A sparse file — one with holes that were never written — reports its full logical size to ls and only its allocated blocks to du:

$ ls -l run/sparse.dat
-rw-r--r--  1 engineer  staff  52428800 Oct  8 22:30 run/sparse.dat
$ du -h run/sparse.dat
 16K	run/sparse.dat

Fifty megabytes by one measure, sixteen kilobytes by the other, both correct. Some waveform and coverage databases are sparse, so a directory listing can wildly overstate what a run is costing you, and du is the one to trust for "am I filling the disk".

4. Finding What Is Using the Space

The idiom, and it is worth memorising because you will type it on every new machine:

$ du -sh -- "$PROJECT_ROOT"/* | sort -h
 12K	proj/docs
 80K	proj/rtl
 80K	proj/tb
2.3M	proj/runs

Four parts, each doing something:

-s summarises each argument rather than listing every file beneath it. Without it you get thousands of lines.

-h gives human units. This is what makes sort -h necessary rather than sort -n, which would put 900K above 2.3M.

-- before the glob, because a directory name could begin with a dash — Chapter 1.5.

And /* rather than the directory itself, because du -sh "$PROJECT_ROOT" is one number and tells you nothing about where it went.

sort -h exists here, which is worth checking before relying on it:

$ printf '2K\n1M\n3K\n' | sort -h
2K
3K
1M

It is a GNU extension that BSD sort also has; where it is missing, the portable form keeps the raw numbers and sorts them numerically:

du -sk -- "$PROJECT_ROOT"/* | sort -rn | head -20

-sk is kilobytes everywhere, sort -rn is numeric and descending, and head -20 is what makes this usable on a project area with four hundred subdirectories. Drilling down is then the same command one level deeper, which is the whole technique:

du -sk -- "$PROJECT_ROOT"/runs/* | sort -rn | head -5

5. The Purge Rule, and the Off-by-One In It

Every scratch area has a policy of the form "deleted after N days", and every team eventually writes their own version of it. The rule is almost always expressed with find -mtime, and -mtime +7 does not mean "older than seven days" in the way people read it.

Measured, with files aged exactly 0, 1, 2, 7 and 8 days:

$ find run -name 'age_*' -mtime +7
run/age_8d
$ find run -name 'age_*' -mtime +6
run/age_7d
run/age_8d
$ find run -name 'age_*' -mtime 7
run/age_7d

find divides the file's age by 24 hours and discards the remainder. So the 7-day-old file has an age of exactly 7 periods, and +7 means strictly greater than 7, which excludes it. To catch everything a week old or more you write +6. Chapter 14.6 §4 established this arithmetic; what is new here is that it is now attached to a rule that deletes things.

Three consequences, and the first is the one that wastes an afternoon:

"The purge did not run." It ran. It just did not match the file you were looking at, because that file was 7.4 days old and +7 wanted 8 or more.

-mtime 7 with no sign is an exact match on the seventh 24-hour period — a one-day window, almost never what anyone wants.

And -mmin is the same arithmetic in minutes, which is the right tool when the window is hours rather than days: -mmin +120 is strictly more than two hours.

6. Where to Put Things: the Rules That Follow

Given sections 1 to 5, the decisions are mechanical:

A run's working directory goes in scratch, local if possible, and the run copies out only what must survive. That is run-local from Chapter 22.3 §6.

What survives is small and textual: the log, a one-line result, the arguments used. Waveforms survive only for failures, and only while someone is debugging them.

Source, filelists and released results go in the project area, which is backed up. Nothing generated goes there, because a generated file in a backed-up area is paying twice — once in disk and once in everyone's confusion about whether it is authoritative.

Your home directory holds your own configuration and nothing a compute job needs to read. Chapter 10.3's startup files live here, and that is close to the whole list.

And anything you want in six months goes to archive deliberately, because the alternative is discovering that it was in scratch.

7. A Report Worth Running Weekly

Small enough to read, and it answers "am I about to cause a problem":

#!/usr/bin/env bash
# space-report — where my data is, and how close the filesystem is to full
set -o nounset

for d in "${PROJECT_ROOT:-}" "${SCRATCH_ROOT:-}" "$HOME"; do
  [ -n "$d" ] && [ -d "$d" ] || continue
  printf '== %s\n' "$d"
  df -h -- "$d" | awk 'NR==2 { printf "   filesystem %s on %s — %s used, %s free\n", $1, $NF, $5, $4 }'
  du -sk -- "$d"/* 2>/dev/null | sort -rn | head -5 |
    awk '{ printf "   %8.1f MB  %s\n", $1/1024, $2 }'
done

Three details that make it work rather than nearly work:

2>/dev/null on the du, because a project area always contains something you cannot read, and one Permission denied per directory would bury the output. This is a case where discarding stderr is correct rather than lazy — Chapter 12.3 §5's distinction.

awk doing the unit conversion, so the sort is on raw kilobytes and the display is in megabytes. Sorting on a formatted column is the mistake this avoids.

And [ -n "$d" ] && [ -d "$d" ] || continue, so an unset variable skips the section instead of reporting on the current directory — which is what an empty "$d" would have done to df.

Practice

1. On a machine you use, identify a home, a project, a scratch and an archive area. Which ones are guesses?
2. For each, run df -h . from inside it and record Size, Avail and whether mount says local.
3. Run quota and quota -v. If quota -s fails, note the exact error and what the usage line offers instead.
4. Create a 10-byte file and compare ls -l, du -h, du -Ah and stat. Explain all four numbers.
5. Make a sparse file (dd if=/dev/zero of=f bs=1 count=1 seek=50M, or Python's seek) and compare ls -l with du -h.
6. Count the files in a run directory you have. Multiply by 4 KB and compare with du -sh. How much of the cost is block rounding?
7. Create files aged 0, 7 and 8 days with touch -t and run find -mtime +7, +6 and 7. Confirm the measured result in §5.
8. Write the one-line purge rule your team would use for "two weeks", then check it against the off-by-one.
9. Write purge-scratch and run it with DRY_RUN=1 against a directory of fake run output. Then point it at a path not under scratch and confirm it refuses with status 2.
10. Then set DRY_RUN=0 and confirm the list it deletes is identical to the list it previewed.
11. Write space-report and run it with PROJECT_ROOT unset. Confirm it skips that section rather than reporting on the current directory.
12. Find the largest single file under a project area in one pipeline, and say which of du or ls you used and why.

Check Your Understanding

1. What are the four kinds of directory, and which is not backed up?
2. Which df column matters more than Capacity, and why?
3. Why can a write fail with "quota exceeded" when your own usage has not changed?
4. What is the third quota limit people forget, and which df columns show it?
5. Why does a 10-byte file cost 4 KB, and what does that mean for a flow writing one status file per test?
6. For a sparse file, which of ls -l and du should you believe about disk usage?
7. Why does find -mtime +7 not match a file that is 7.4 days old?
8. In purge-scratch, why is the victim list collected once?

The largest single consumer of a scratch area is usually waveform dumping, and Chapter 27.6 measures the two ways of reducing it — one of which is nine hundred times more effective than the other.

Chapter 30.6 measures a resource cap enforced against a real job — and the unit that made a declared 50 KB limit twice as generous.

8. Remember

• Four kinds of directory: home, project, scratch, archive — distinguished by backup and purge policy, not by location.
• Scratch is not backed up and is deleted on a schedule. Anything you want tomorrow has to be copied out today.
• Avail beats Capacity. Percentages hide absolute numbers.
• Quotas are per user and per group, and soft limits turn hard when the grace period expires — so a working regression can start failing with no change.
• There is an inode quota too. iused / ifree; a flow writing many small files hits it on an empty-looking filesystem.
• ls reports bytes, du reports blocks. A 10-byte file costs 4 KB here.
• Ten thousand tiny files cost 40 MB for 100 KB of data, plus a network round trip each.
• A sparse file reads 50 MB to ls and 16 K to du. Trust du for disk usage, and never mix the two in one report.
• du -sh -- dir/* | sort -h, with du -sk | sort -rn | head as the portable form.
• -mtime +7 means strictly more than 7 whole days, so it skips the 7-day-old file. Write +6 for "a week or more".
• A purge needs a positive path rule, a resolved path, one victim list, and a dry run that is the default.
• :? for what you cannot default, :- for what you can, and nothing silently assumed.

Chapter 28.3 returns to this area with the two tools that measure it, and the case where they disagree by 400 MB and are both right.

9. What Is Next

Chapter 22.5 moves from the filesystem to the machine: how to tell whether a shared server has room for your work before you start it, how to run heavy jobs without degrading everyone else's session, and how to avoid being the engineer whose stray process is still running on Friday.