Linux & Bash · Part II · Module 34 · Defensive Automation Around EDA Tools

Disk, Quota and Running Out Mid-Run

Running out of space halfway through is the failure that leaves the most convincing wreckage: a log, a waveform and a plausible exit status, all of them partial. This chapter measures exactly what survives, finds the surviving log was written by the shell rather than the tool, and builds the check that distinguishes ran-out-of-space from everything else before and after the run.

Advanced24 min readLinuxPart IIEDADiskQuota

Part II · Module 34 · Chapter 34.5 · Disk, Quota and Running Out Mid-Run

Chapter 34.4 bounded a run that would not finish. This chapter is the run that cannot finish, and it is the most deceptive failure in the module because it leaves a complete-looking set of artifacts.

A file-size limit is the local, reproducible form of a full filesystem or an exceeded quota — Chapter 33.8 used it for exactly that, and it needs no cooperation from an administrator.

1. What Survives

The same simulation with a waveform, twice:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  uncapped: rc=0   log=229B dump=18678B
  capped  : rc=153 log=130B dump=4096B

Three artifacts, all present, all wrong. And the status is informative for once: 153 is 128 + 25, SIGXFSZ — the signal for exceeding a file-size limit, which Chapter 11.1 established and Chapter 30.6 measured arriving from a scheduler's own cap.

Now read the log that survived:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

bin/probe_m344.sh: line 40: 32379 Filesize limit exceeded: 25   vvp ../../logs/sim.vvp +TESTNAME=random +SEED=1 +CYCLES=200 +DUMP

That log was written by the shell, about a program that was already dead

One hundred and thirty bytes, and not one of them came from the simulator.

What happened: the tool was killed by SIGXFSZ while writing its waveform. The shell, which was waiting for it, noticed a child had died on a signal and printed its standard job-control notification — the script name, the line number, the pid, the signal's description, and the command. That notification went to the shell's standard error, which the wrapper had redirected into run.log.

So the artifact that every downstream stage reads as "the tool's output" contains:

the name of a script, not the tool
a line number in that script
a pid that no longer exists
and the command line, which is genuinely useful and is the only part that is

Everything the simulator actually printed before it died — the header line, any progress — is gone, because it was still in a buffer.

Three consequences worth separating.

Chapter 34.2's checker handles it correctly, and for the right reason: the log has no $finish called marker and states no verdict, so it returns 2, "present but incomplete". It does not need to understand SIGXFSZ to be right.

But a log-scraping check looking for tool errors finds nothing it recognises. There is no ERROR, no FATAL, no tool prefix. A classifier keyed on the tool's vocabulary — Chapter 33.7's signature grouping — puts this run in the (no diagnostic line) bucket alongside a testbench that simply forgot to report, which are entirely different problems.

And the message's own text is misleading about blame. bin/probe_m344.sh: line 40: reads as a problem in that script. It is not: the script's line 40 is where the tool was launched, and the tool is what hit the limit. Chapter 33.4 measured the same shape when set -u's diagnostic landed in a run's log and looked like a tool error.

The structural point: a wrapper that redirects 2>&1 into an artifact is also redirecting the shell's own voice into it. That is usually what you want — a tool's stderr belongs with its stdout — and it means the artifact is not purely the tool's.

2. The Units, Because They Are Not Bytes

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ( ulimit -f 1; dd if=/dev/zero of=dk/one.bin bs=1 count=4096 ) 2>/dev/null
$ wc -c < dk/one.bin
    1024

ulimit -f counts 1024-byte blocks, so ulimit -f 4 is 4,096 bytes and not 4. That is exactly the waveform size the capped run produced, which is how the measurement confirms itself.

This matters because the obvious reading is wrong in the dangerous direction: somebody intending a 50 MB cap writes ulimit -f 50000000 and gets 50 GB, which never fires. Chapter 30.6 measured me making the inverse mistake — assuming 512-byte blocks and asking for twice the cap I intended.

The adjacent units are worth having in one place, because they differ:

limit	unit
`ulimit -f`	1024-byte blocks
`ulimit -v`, `-m`, `-d`, `-s`	kilobytes
`ulimit -c`	1024-byte blocks
`du`	1024-byte blocks by default, `-k` to be sure
`df -k`	1024-byte blocks
`wc -c`	bytes

du and ls disagreeing is Chapter 28.3's subject — du reports blocks allocated and ls reports the byte length, which differ by the whole factor Chapter 32.4 measured at 72× for a directory of tiny files.

3. Checking Before, and Checking After

Two different checks, and both are needed.

Before: is there room?

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /bin/df -k .
Filesystem   1024-blocks      Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk3s5   482797652 358559180  89300952    81% 1919085 893009520    0%   /System/Volumes/Data

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# headroom: refuse to start a regression that cannot possibly fit
need_mb=${NEED_MB:-2048}
avail_mb=$(( $(/bin/df -k "$RUNS" | awk 'NR==2 {print $4}') / 1024 ))
[ "$avail_mb" -ge "$need_mb" ] || {
  printf 'regress: %d MB free under %s, need %d MB -- refusing to start\n' \
    "$avail_mb" "$RUNS" "$need_mb" >&2
  exit 3; }

Three things about that, and the third is the one people miss.

df on the target directory, not on /. A flow writing to /proj/scratch learns nothing from the root filesystem's free space, and the two are usually different filesystems.

awk 'NR==2 {print $4}' rather than a fixed column count, because df's first line is a header whose width varies with the device name, and a long device name wraps the output on some systems. Pinning /bin/df matters too — GNU df and BSD df differ in their default block size, which is why -k is explicit.

And the estimate has to come from measurement. Chapter 28.2 inventoried what one run leaves behind; the headroom figure is that times the matrix size, with the waveform dominating. A guessed NEED_MB is a check that refuses good regressions or admits doomed ones.

Written out as a tool, with every branch exercised:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# spacecheck DIR NEED_MB -- refuse to start if the target cannot hold the work.
#   0 enough room | 2 usage | 3 not enough room | 4 the path is not usable
set -u
d=${1:-}; need=${2:-}
[ -n "$d" ] && [ -n "$need" ] || { echo "spacecheck: need DIR and NEED_MB" >&2; exit 2; }
case $need in ''|*[!0-9]*) echo "spacecheck: NEED_MB '$need' is not a number" >&2; exit 2;; esac
mkdir -p "$d" 2>/dev/null
[ -d "$d" ] && [ -w "$d" ] || { echo "spacecheck: '$d' is not a writable directory" >&2; exit 4; }
# df on the TARGET, not on /, and -k so the block size is not the system default
avail_kb=$(/bin/df -k "$d" | awk 'NR==2 {print $4}')
case $avail_kb in ''|*[!0-9]*) echo "spacecheck: could not read df output for '$d'" >&2; exit 4;; esac
avail_mb=$(( avail_kb / 1024 ))
printf 'spacecheck: %s MB free under %s, need %s MB\n' "$avail_mb" "$d" "$need" >&2
[ "$avail_mb" -ge "$need" ] || { echo "spacecheck: not enough room -- refusing to start" >&2; exit 3; }

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  CASE                               rc   MESSAGE
  enough room                        0    86215 MB free under dk/t1, need 10 MB
  absurd requirement                 3    not enough room -- refusing to start
  no arguments                       2    need DIR and NEED_MB
  NEED_MB not a number               2    NEED_MB 'lots' is not a number
  an unwritable path                 4    '/System/x' is not a writable directory

The NEED_MB not a number row is there because of Chapter 33.5: [ "$avail_mb" -ge "$need" ] with a non-numeric need is a syntax error rather than a false, so the shape is validated before the comparison. And the mkdir -p before the writability test is deliberate — a flow's output directory usually does not exist yet, and df on a non-existent path reports the wrong filesystem.

After: did it run out?

A pre-flight check cannot prevent this — the space can disappear while the regression is running, taken by somebody else's job — so the per-run check matters more:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# in the run wrapper, after the tool
rc=$(cat "$d/tool.rc")
case $rc in
  153) printf 'run: killed by SIGXFSZ -- a file-size limit was hit\n' >&2;;
esac
/usr/bin/grep -qE 'No space left on device|Disk quota exceeded|File too large' "$d/run.log" \
  && printf 'run: the log reports running out of space\n' >&2

And as a tool, so the two signals are checked the same way every time:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# spaceclass RUNDIR -- did this run fail for want of space? Two signals,
# because a per-process LIMIT and a full FILESYSTEM look different.
set -u
d=${1:?need a run directory}
rc=$(cat "$d/tool.rc" 2>/dev/null || echo 0)
case $rc in
  153) echo "SIGXFSZ: a file-size limit was hit (128+25)"; exit 1;;
esac
if [ -r "$d/run.log" ] && /usr/bin/grep -qE 'No space left on device|Disk quota exceeded|File too large' "$d/run.log"; then
  echo "the log reports running out of space"; exit 1
fi
exit 0

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  capped  : rc=1  SIGXFSZ: a file-size limit was hit (128+25)
  uncapped: rc=0

Both, because they catch different causes. 153 is a per-process limit; No space left on device is the filesystem. A quota produces Disk quota exceeded and no signal at all — the write simply fails and the tool decides what to do, which may be to carry on with a corrupt artifact.

That last case is why Chapter 34.2's artifact checks are the real defence: a tool that handles a failed write gracefully produces a complete-looking run with a truncated waveform and a clean exit status. No signal, no error in the log, nothing but a short file.

Why retrying this is usually wrong, and when it is not

Chapter 34.3 put No space left on device on the retryable list with a qualification, and this is it.

Retry is right when the space was transient — another job filled the filesystem and has since finished, or a scratch area is cleaned on a schedule. Here a retry genuinely succeeds, and the failure is as transient as a licence shortage.

Retry is wrong when the regression is the cause. Two hundred runs each writing a 50 MB waveform need 10 GB; if that does not fit, every retry fails the same way and the backoff turns a disk problem into a four-hour disk problem. Chapter 31.5 listed disk as one of the four bounds on the slot count, and this is the case where it binds.

And retry is actively harmful when the artifacts are not cleaned first. A retried run that writes into the same directory as the failed attempt may find the truncated waveform already there, and depending on the tool either append to it or leave it — producing an artifact that is a mixture of two runs. Chapter 31.6's rule applies with extra force: the retry gets a clean directory, or it is not a retry.

The discriminating question is whether your own regression can fit. If the headroom check in Section 3 passed and the space then vanished, retry. If it would not have passed, the fix is fewer runs in flight or smaller artifacts, and a retry is a way of not noticing that.

4. Not Writing So Much

The cheapest defence is to produce less, and the measurements for it are already in this track.

Waveforms dominate. The capped run's dump was 18,678 bytes against a 229-byte log — eighty times the log on a toy fixture, and on a real block the ratio is thousands. Chapter 27.6 measured the knobs: dump a window rather than the whole run, limit the depth, and dump only on failure.

Dumping only on failure needs two passes, which is the honest cost: run without a dump, and re-run the failures with one. Chapter 32.6 is that second pass, and it is cheaper than it sounds precisely because the failures are a small subset — eight representatives rather than thirty-six runs, in Chapter 33.7's measurement.

And the passing runs' artifacts are the ones to drop. Chapter 32.8 measured that deleting the passing runs' logs saves only 6% because they are the smallest — but a waveform does not compress and a passing run's waveform is the one nobody will open. The asymmetry is the opposite way round for binary artifacts than for text.

So the policy that follows from the numbers:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# dump nothing by default; the rerun adds it for the runs that need it
vvp "$image" "+TESTNAME=$t" "+SEED=$s" ${DUMP:+ +DUMP +DUMP_FILE="$d/dump.vcd"}

${DUMP:+ …} expands to nothing when DUMP is unset, which is Chapter 8.2's alternate-value form — and note it is deliberately unquoted so that it expands to several words when set, one of the two legitimate cases Chapter 33.5 identified.

5. The Whole Defence

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# 1. before the regression: can it possibly fit?
check_headroom "$RUNS" "$NEED_MB" || exit 3

# 2. per run, bound the damage one run can do
( ulimit -f "${MAX_RUN_BLOCKS:-5242880}"      # 5 GB in 1024-byte blocks
  bounded "$LIMIT_S" "$d" vvp "$image" "+TESTNAME=$t" )

# 3. after each run: was it a space failure?
classify_space "$d"

# 4. and validate the artifacts rather than their presence
logcheck "$d/run.log"; printf '%s\n' "$?" > "$d/logcheck.rc"
[ -s "$d/dump.vcd" ] || printf 'no waveform\n' >> "$d/notes"

Step 2 is the one worth arguing for. A per-run ulimit -f seems like adding a failure mode, and it is — a deliberate one, in exchange for a guarantee: one runaway run cannot fill the filesystem and take the other 199 with it. Without it, a single test that dumps unboundedly turns one failure into a regression with no results at all, which is Chapter 31.7's NOSTATUS across the board.

The cap has to be generous enough that no legitimate run hits it — the same order-of-magnitude rule as Chapter 34.4's time bound — and recorded, so that a run which hit it says so rather than looking like a mysterious 153.

Practice

Run a simulation with a waveform under a file-size limit, and record the status, the log size and the dump size.
Decode the status and name the signal.
Read the surviving log and say which program wrote it.
List what that message contains, and which part is useful.
Say where the tool's own output went.
Run Chapter 34.2's checker on it and explain why it is right without knowing about the signal.
Run a tool-vocabulary classifier on it and say which bucket it lands in.
Explain why the message's text misattributes blame.
Find the units of ulimit -f by allowing exactly one block.
Compute what ulimit -f 50000000 actually permits.
Tabulate the units of ulimit -f, -v, du, df -k and wc -c.
Write a pre-flight headroom check against the target directory.
Run it against / instead and say what it tells you.
Parse df with awk 'NR==2' and say why not a fixed column.
Derive NEED_MB from a measurement rather than a guess.
Add an after-the-fact check for both the signal and the log messages.
Explain why a quota produces no signal.
Construct the case where a tool handles a failed write gracefully, and say what the artifacts look like.
Decide when a space failure is retryable and when it is not.
Say what a retry into an uncleaned directory can produce.
Compare a dump's size against a log's in your own flow.
Switch to dumping only on failure and cost the second pass.
Add a per-run ulimit -f and justify the deliberate failure mode.
Record the cap and say what a run that hit it would otherwise look like.

Check Your Understanding

Why is running out of space the most deceptive failure in this module?
Why is a file-size limit a good local stand-in?
Give the measured status, log size and dump size for both runs.
What is 153?
Who wrote the surviving log, and what is in it?
Where did the simulator's own output go?
Why is Chapter 34.2's checker right about it anyway?
What does a tool-vocabulary classifier do with it, and why is that wrong?
Why does the message misattribute blame?
State the structural point about 2>&1 into an artifact.
What are ulimit -f's units, and how was that established?
What does ulimit -f 50000000 actually permit?
Why is df on / not useful?
Why awk 'NR==2 {print $4}' and why /bin/df -k?
Where must NEED_MB come from?
Why can a pre-flight check not prevent this?
Why check both the signal and the log text?
What does a quota produce instead of a signal?
What do the artifacts look like when a tool handles a failed write gracefully?
When is retrying a space failure right, and when wrong?
What is the harm in retrying into an uncleaned directory?
What is the discriminating question for whether to retry?
What dominates a run's disk usage, and by roughly what ratio here?
What does dumping only on failure cost, and why is it cheaper than it sounds?
Why is the pruning asymmetry opposite for binary artifacts and text?
Why is ${DUMP:+ …} deliberately unquoted?
What does a per-run ulimit -f buy, and what does it cost?
Why must the cap be recorded?

Answers

Because it leaves a complete-looking set of artifacts — a log, a waveform and a plausible status, all partial.
Because it is the local, reproducible form of a full filesystem or an exceeded quota, and needs no administrator.
Uncapped: rc 0, log 229 B, dump 18,678 B. Capped: rc 153, log 130 B, dump 4,096 B.
128 + 25, SIGXFSZ — the signal for exceeding a file-size limit.
The shell. It contains a script name, a line number in that script, a dead pid, the signal's description and the command line — of which only the command line is useful.
Nowhere — it was still in a buffer when the process was killed.
Because the log has no end-of-run marker and states no verdict, so it returns "present but incomplete" without needing to know about the signal.
Puts it in the (no diagnostic line) bucket alongside a testbench that forgot to report — entirely different problems.
Because bin/probe…: line 40: reads as a fault in that script, when line 40 is merely where the tool was launched.
A wrapper that redirects 2>&1 into an artifact is redirecting the shell's own voice into it too, so the artifact is not purely the tool's.
1024-byte blocks — ulimit -f 1 allowed exactly 1,024 bytes, and ulimit -f 4 produced the 4,096-byte dump.
About 50 GB, so a cap intended as 50 MB never fires.
Because the flow writes to a different filesystem, and the two have different free space.
Because df's header line and device-name width vary, so a fixed column is fragile — and GNU and BSD df differ in default block size, so -k and the absolute path are both explicit.
From a measurement of what one run leaves behind, times the matrix size. A guess refuses good regressions or admits doomed ones.
Because the space can disappear while the regression is running, taken by another job.
Because 153 is a per-process limit and No space left on device is the filesystem — different causes.
Disk quota exceeded and no signal at all: the write simply fails and the tool decides what to do.
A complete-looking run with a truncated artifact and a clean exit status — no signal, no error, just a short file.
Right when the space was transient — another job filled it and has finished. Wrong when your own regression is the cause, where every retry fails identically and the backoff multiplies the delay.
The retry may find the failed attempt's truncated artifact and append to or keep it, producing a mixture of two runs.
Whether your own regression can fit. If the headroom check passed and space then vanished, retry; if it would not have passed, reduce the runs in flight or the artifacts.
The waveform — 18,678 bytes against a 229-byte log here, about eighty to one, and thousands to one on a real block.
A second pass over the failures. It is cheap because the failures are a small subset — eight representatives rather than thirty-six runs.
Because a passing run's log is the smallest and saves almost nothing, while a passing run's waveform is large, incompressible and will never be opened.
So it expands to several words when set — one of the two legitimate cases for leaving an expansion unquoted.
It guarantees one runaway run cannot fill the filesystem and destroy the other 199. The cost is a deliberate extra failure mode.
So that a run which hit it says so, instead of looking like a mysterious 153.

6. Remember

Running out of space leaves convincing wreckage: rc 153, a 130-byte log and a 4,096-byte waveform, all present.
The surviving log was written by the shell, not the tool — a script name, a line number, a dead pid and the command line. The tool's own output died in a buffer.
2>&1 into an artifact redirects the shell's voice into it too.
Chapter 34.2's checker is right about it without knowing the cause; a tool-vocabulary classifier is not.
ulimit -f counts 1024-byte blocks. ulimit -f 50000000 is 50 GB, not 50 MB.
Check headroom before, on the target filesystem, with /bin/df -k and awk 'NR==2' — and derive the figure from a measurement.
Check after, for both 153 and the log's own No space left on device / Disk quota exceeded — a quota produces no signal at all.
A tool that handles a failed write gracefully gives a clean status and a truncated artifact. Validating artifacts is the only defence.
Retry only if the space was somebody else's, into a clean directory — and never if your own regression cannot fit.
The waveform dominates. Dump on failure only, and prune passing runs' binary artifacts, not their logs.
Cap each run generously and record the cap, so one runaway run cannot take the other 199 with it.

7. What Is Next

Every chapter so far has assumed that two runs of the same thing agree. Chapter 34.6 checks that assumption on this fixture — finding the simulation itself byte-identical across five runs, and the variation somewhere nobody looks.

Where this fits

Part of the Linux & Bash curriculum.