Workstation, Compute Server and Shared Filesystem

Part II · Module 22 · Chapter 22.3 · Workstation, Compute Server and Shared Filesystem

"It works for me" is the most expensive sentence in engineering, and in this environment it has one dominant cause: the path you used does not mean the same thing on the machine where the job ran.

This chapter is about why that is structurally likely, and how to find out in one command.

1. Three Roles, Not Three Computers

The workstation is where you log in, edit, browse logs and launch things. It is sized for interactive work and is not where a hundred-hour regression belongs.

The compute servers are where jobs run. You usually do not choose one; a scheduler does, and you may never learn which machine ran a given test. That single fact drives most of this chapter: the machine that ran your job is not the machine you are standing on.

The file server holds the data, and both of the others reach it over the network.

In small teams two of these roles live on one box. In a large one there are thousands of compute servers. The number does not matter; the roles do, because the failures are role failures.

2. Finding Out Where You Are

Four questions, four commands, and they are the first things to run when something behaves differently than expected:

$ hostname
mbp-engineer.local
$ df -h .
Filesystem      Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s5   460Gi   342Gi    85Gi    81%    2.0M  887M    0%   /System/Volumes/Data
$ pwd
/Users/engineer/work/vlsi-mentor-tutorials
$ id -un
engineer

df -h . is the one worth dwelling on, because of what it reveals: the current directory is not on the device mounted at /. It is on /dev/disk3s5, mounted at /System/Volumes/Data. The root filesystem is a different device entirely:

$ df -h -t apfs
Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s1s1   460Gi    12Gi    85Gi    13%    459k  887M    0%   /
/dev/disk3s6     460Gi   9.0Gi    85Gi    10%       9  887M    0%   /System/Volumes/VM

So a path beginning /Users/... crosses a mount boundary on the way, and the "disk" your files are on is not the one / is on. On a compute farm the same observation is what tells you whether you are writing to the network or to the local machine, which is the difference between "visible to every job" and "gone when the job ends".

3. Reading mount

mount with no arguments lists what is mounted where, in a consistent shape: device, the word on, the mount point, then the type and flags in parentheses.

$ mount | head -6
/dev/disk3s1s1 on / (apfs, sealed, local, read-only, journaled)
devfs on /dev (devfs, local, nobrowse)
/dev/disk3s6 on /System/Volumes/VM (apfs, local, noexec, journaled, noatime, nobrowse)
/dev/disk3s2 on /System/Volumes/Preboot (apfs, local, journaled, nobrowse)
/dev/disk3s4 on /System/Volumes/Update (apfs, local, journaled, nobrowse)
/dev/disk1s2 on /System/Volumes/xarts (apfs, local, noexec, journaled, noatime, nobrowse)

Three flags there are worth recognising because their analogues appear on every engineering filesystem:

local — this filesystem is on hardware in this machine. On a compute server, the project area will not say local; it will name a remote server. That word is the fastest way to tell network storage from local storage.

read-only — writes fail, and the error names permissions rather than the mount, which sends people to chmod for an hour.

noexec — a file on this filesystem cannot be executed even with the execute bit set. Scratch areas are frequently mounted noexec, and the symptom is "Permission denied" on a script whose mode is 0755. Chapter 5.5 is where the execute bit was taught; this is the case where the bit is set and the answer is still no.

On a real engineering machine the corresponding line would name an NFS server and export, and the type would be nfs. The usual filter is the one in this chapter's own curriculum entry:

$ mount | grep -i nfs
$ echo $?
1

Nothing, status 1. There is no NFS on this machine — it is a laptop, not a compute node. So rather than invent a line, here is what to do with it when there is one: the mount point is the field after on, and the server is the part before the colon in the device.

# which of my mounts are on the network, and from where
mount | awk '/\(nfs|type nfs/ { print $3, "<-", $1 }'

$3 is the mount point and $1 the server:/export — Chapter 13.7's field model applied to a line format you now know. And the reason to care is §4.

4. The Failure: One File, Two Paths

Here is the chapter's real content, and it is runnable anywhere.

A directory, a file in it, and a symbolic link pointing at the directory:

$ mkdir -p proj && echo "module top; endmodule" > proj/top.v
$ ln -s "$PWD/proj" alias
$ stat -f "%d %i  %N" proj/top.v alias/top.v
16777230 83530510  proj/top.v
16777230 83530510  alias/top.v

Same device, same inode: one file, two paths. That part is Chapter 6.3. Now step into the link and ask the shell where you are:

$ cd alias
$ pwd
/var/folders/7g/8jhn.../tmp.NTLIZ8kj4T/alias
$ pwd -P
/private/var/folders/7g/8jhn.../tmp.NTLIZ8kj4T/proj

Two answers to "where am I", and they differ in both directions: the last component (alias versus proj) and the prefix (/var versus /private/var, because /var is itself a symlink here).

This is the whole problem in miniature. pwd is the shell's logical idea of the current directory, remembered from how you got there and exported as $PWD. pwd -P is the physical path, resolved through every link. Chapter 6.4 established the distinction; what Part II adds is the consequence:

A tool that records $PWD records a path that may not exist on the machine that reads it later.

That is precisely what happens on a farm. Your workstation reaches the project through /proj/chip_a, which is an automounted symlink into /net/fileserver3/vol12/chip_a. A log, a makefile dependency, or a coverage database records one of those, a job on another machine is handed the other, and the error you get is "No such file or directory" naming a path that visibly exists.

5. Why $PWD Exists at All, Given That

Because the logical path is the one you meant, and it is often the stable one. /proj/chip_a is a name the organisation controls; /net/fileserver3/vol12/chip_a names a server that will be replaced next year. Printing the logical path in a message to a human is kinder and more durable.

So the two are for different audiences:

That last row is a habit worth forming. Two lines in an error message cost nothing:

printf 'run: cwd %s (physical %s)\n' "$PWD" "$(pwd -P)" >&2

6. Local Disk Versus Network Disk

The second consequence of the topology is performance, and it is large enough to change how flows are written.

A network filesystem is fast for streaming a few big files and slow for thousands of small operations, because each one is a round trip. Compiling a design that opens two thousand headers is exactly the bad case, and simulation that writes a waveform continuously is the other one.

Hence the pattern used by nearly every production flow, stated here and built properly in Module 27:

Read the source from the network, write the scratch output to local disk, and copy back only what must survive.

#!/usr/bin/env bash
# run-local — compile and simulate on local disk, keep only the results
# exit: 0 pass · 1 fail · 2 setup or tool failure
set -o nounset

src=${SRC_ROOT:?set SRC_ROOT to the shared source tree}
keep=${RESULTS:?set RESULTS to the shared results area}

[ -d "$src" ]  || { printf 'run-local: no such source tree: %s\n' "$src" >&2; exit 2; }
mkdir -p -- "$keep" || exit 2

work=$(mktemp -d "${TMPDIR:-/tmp}/run.XXXXXX") || exit 2
trap 'rm -rf -- "$work"' EXIT

cd -- "$work" || exit 2
printf 'run-local: working in %s\n' "$(pwd -P)" >&2

iverilog -o sim.vvp -f "$src/files.f" > compile.log 2>&1 || {
  cp -- compile.log "$keep/"; exit 2
}
vvp sim.vvp < /dev/null > sim.log 2>&1
rc=$?

cp -- sim.log "$keep/"
[ "$rc" -eq 0 ] || exit 2
grep -q 'TEST PASSED' sim.log

Six decisions, each a Part I chapter arriving where it is needed:

${SRC_ROOT:?message} rather than a default. A missing source root is not something to guess at, and Chapter 20.3 §4 measured that :? stops the command being built at all.

mktemp -d under $TMPDIR, which on a compute node is local disk — Chapter 20.6.

A trap … EXIT that removes the scratch directory, so a killed job does not leave a hundred gigabytes behind. Chapter 20.5, and Chapter 22.4 is about why that matters more here than anywhere.

The logs are copied out before the status is checked. A log you cannot read is worthless, and the failing path is the one you need it on. This is the ordering mistake most first drafts make.

< /dev/null on the simulator, for Chapter 22.2 §6's measured reason.

And the script ends on a bare grep -q, so the verdict in the log becomes the script's exit status with no exit at all — Chapter 16.7's bare-test ending.

7. The Five-Line Environment Report

Everything above, as the thing to run — and to paste into a bug report — when a path behaves differently in two places:

#!/usr/bin/env bash
# where-am-i — the facts that explain most "works for me" reports
set -o nounset

printf '%-14s %s\n' \
  "host"      "$(hostname)" \
  "user"      "$(id -un)" \
  "cwd"       "$PWD" \
  "cwd (phys)" "$(pwd -P)" \
  "filesystem" "$(df -h . | awk 'NR==2 {print $1, "on", $NF}')" \
  "writable"  "$([ -w . ] && echo yes || echo NO)"

[ "$PWD" = "$(pwd -P)" ] || printf '\nNOTE: this path goes through a symbolic link.\n'

$ ./where-am-i
host           mbp-engineer.local
user           engineer
cwd            /Users/engineer/work/vlsi-mentor-tutorials
cwd (phys)     /Users/engineer/work/vlsi-mentor-tutorials
filesystem     /dev/disk3s5 on /System/Volumes/Data
writable       yes

awk 'NR==2 {print $1, "on", $NF}' takes the device from the first field and the mount point from the last, rather than counting columns — because df's middle columns vary between systems while the mount point is reliably last. $NF is Chapter 13.7's last-field idiom, and using it here is the difference between a script that works on two systems and one that works on one.

Practice

1. Run hostname, df -h ., pwd and id -un on every machine you have access to. Write down where the differences are.
2. Run df -h . and df -h /. If they report different filesystems, explain what that means for a file you create in the current directory.
3. Run mount and classify every line as local or network. If none is network, say what that tells you about where your work can run.
4. Reproduce §4: a directory, a file, a symlink to the directory, then stat both paths and compare pwd with pwd -P.
5. Write a script that prints $PWD to a file, then read that file from a shell that reached the same directory by its physical path. Explain whether the recorded path is usable.
6. Add the [ "$(pwd)" = "$(pwd -P)" ] test to a script of your own, and make it print a warning rather than fail.
7. Find a filesystem mounted noexec (or mount one), put a 0755 script on it, and run it. Note the exact error, and why chmod cannot fix it.
8. Write where-am-i and run it in three directories, one of which is reached through a symlink.
9. Change the awk in where-am-i to use $6 instead of $NF and find a system where that is wrong.
10. Write run-local and run it against a small design. Then make the compile fail on purpose and confirm compile.log still reaches the results area.
11. Remove the trap from run-local, interrupt it mid-run, and find what it left behind. Then put the trap back.

Check Your Understanding

1. What are the three machine roles, and which one runs your jobs?
2. What must all three agree about, and what happens when they do not?
3. What question does df -h . answer that df -h / does not?
4. What do local, read-only and noexec tell you in mount output?
5. What is the difference between pwd and pwd -P, and which belongs in a file another machine will read?
6. Why is device:inode the only way to test whether two paths are one file?
7. Why does a production flow write scratch output to local disk?
8. In run-local, why are the logs copied out before the exit status is checked?

8. Remember

• Three roles: workstation (you type), compute servers (jobs run), file server (data lives). A scheduler picks the compute server, so it is not the machine you are on.
• The agreement that makes it work is one mount point: the same path on every machine.
• df -h . answers "where am I writing" — and here that is a different device from /.
• In mount output, no local means network storage. noexec defeats a correct execute bit.
• pwd is logical, pwd -P is physical, and they differed in both the prefix and the last component when a symlink was involved.
• Record physical paths; print logical ones. A tool that writes $PWD writes a path another machine may not have.
• [ "$(pwd)" = "$(pwd -P)" ] is the one-line test for this entire class of bug.
• Two paths are the same file only if device:inode matches. Names and [ -f ] cannot tell you.
• Read from the network, scratch on local disk, copy back only results.
• Copy the logs out before checking the status, or you lose them exactly when you need them.
• Put -- before path operands and resolve with cd … && pwd -P.
• NFS caching means ls can lag, and a write can fail at close — so trust a tool's status over the existence of its output file.

9. What Is Next

Chapter 22.4 goes one level down: given that the filesystem is shared, which directory should a hundred gigabytes of waveform go in — and which of them is backed up, which is fast, which is purged tonight, and which has a quota that will stop your regression at two in the morning.