File Tests: Existence, Type, Readability and Age

Part I · Module 16 · Chapter 16.4 · File Tests: Existence, Type, Readability and Age

A script that reads a file it was given has two choices. It can try, and let whatever happens happen. Or it can ask first, and say something useful.

$ ./report.sh results.csv
./report.sh: line 14: results.csv: No such file or directory
./report.sh: line 15: results.csv: No such file or directory
awk: can't open file results.csv
Total: 0

Four lines of noise and then a confident, wrong answer. The alternative costs three lines at the top of the script and is the subject of this chapter.

1. The Matrix

Every file test is a single operator taking one filename. Here is the complete picture, measured against six real fixtures — a file with content, a directory, an empty file, a symlink, a symlink pointing at nothing, and a file with its permissions removed:

$ echo data > f.txt
$ mkdir d
$ : > empty.txt
$ ln -s f.txt f-link
$ ln -s nowhere dangling
$ echo x > noperm.txt && chmod 000 noperm.txt

That table is the chapter. Four rows in it are worth stopping on.

2. Existence Is Not Permission

Look at the noperm.txt row: -e says yes, -f says yes, -s says yes — and -r says no.

$ [ -e noperm.txt ] && echo "it exists"
it exists
$ [ -r noperm.txt ] && echo "readable" || echo "NOT readable"
NOT readable
$ cat noperm.txt
cat: noperm.txt: Permission denied

The file is right there, the right size, the right type. It cannot be opened. Chapter 5.2's permission bits are doing exactly their job, and -e has nothing to say about them.

This is the most common wrong guard in real scripts:

if [ -e "$log" ]; then
  wc -l < "$log"          # can still fail with Permission denied
fi

-r is what you want almost every time. It answers the question you actually care about — can I read this? — and a file you can read is necessarily a file that exists, so -r subsumes -e for any script that intends to open something. The same goes for -w before writing and -x before running.

3. A Broken Symlink Both Exists and Does Not

The dangling row is the only one where -e and -L disagree, and the disagreement is total:

$ ls -l dangling
lrwxr-xr-x  1 engineer  engineer  7 Oct  8 14:09 dangling -> nowhere

$ [ -e dangling ] && echo "exists" || echo "does not exist"
does not exist
$ [ -L dangling ] && echo "is a link" || echo "not a link"
is a link

The symlink is listed by ls. Deleting it works. And -e says it does not exist.

The resolution is that every file test except -L follows the symlink and reports on the target. -e dangling is not asking about the link; it is asking about nowhere, which genuinely is not there. -L is the one operator that asks about the link itself.

Which is why the working symlink's row is -f yes and -L yes: -f followed it to a regular file, -L saw the link. Both are true, about two different objects.

$ rm -f dangling
$ [ -e dangling ] || echo "still does not exist — and now really does not"
still does not exist — and now really does not

4. A Directory Is Not a File, and Has a Size

Two rows' worth of wrong guesses in the d column.

-f means regular file, which excludes directories:

$ [ -f d ] && echo "regular file" || echo "not a regular file"
not a regular file
$ [ -d d ] && echo "directory"
directory
$ [ -e d ] && echo "exists"
exists

So -f is a type test as well as an existence test, and that is usually what you want: a script expecting a log file should reject a directory named run.log rather than hand it to grep.

-s says yes for a directory, which surprises people:

$ [ -s d ] && echo "non-zero size"
non-zero size
$ ls -ld d
drwxr-xr-x  2 engineer  engineer  64 Oct  8 14:09 d

Sixty-four bytes. A directory is a real object on disk with a real size, so -s — which asks only is this larger than zero bytes — says yes. -s alone is not an "is there content here" test; it needs -f beside it.

-x on a directory means traversable, not runnable:

$ [ -x d ] && echo "x is set"
x is set
$ [ -x f.txt ] && echo "runnable" || echo "not runnable"
not runnable

Chapter 5.3 made this point about chmod: the execute bit on a directory grants the right to enter it and resolve names inside it, which is why nearly every directory has it set. [ -x "$dir" ] is a meaningful test — it is how you check that you can cd there — but it is not asking whether anything is executable.

5. The One Reason -s Exists

The empty.txt row: -f yes, -s no.

$ [ -f empty.txt ] && echo "the file is there"
the file is there
$ [ -s empty.txt ] && echo "and has content" || echo "but it is empty"
but it is empty

This distinction is worth more in practice than any other operator in the table, because an empty log file is the signature of a specific failure. A simulator that was killed by the queue, ran out of licences at startup, or died before its first write leaves behind a log that exists, is the right name, has the right permissions and contains nothing. A script that checks -f and then greps it finds no errors and reports a pass.

if [ ! -f "$log" ]; then
  echo "$log: never created — the tool did not start" >&2
  exit 2
elif [ ! -s "$log" ]; then
  echo "$log: created but empty — the tool died before writing" >&2
  exit 2
fi

Two different diagnoses, two different things to go and look at, and both of them are invisible to a script that only tests -f.

6. The Full Operator List

The table above covers the ones you will use daily. The rest, for completeness:

A few of those measured, because the guesses go wrong:

$ [ -c /dev/null ] && echo "character device"
character device
$ [ -p /dev/null ] && echo "pipe" || echo "not a pipe"
not a pipe
$ [ -b /dev/null ] && echo "block" || echo "not a block device"
not a block device

/dev/null is a character device, not a pipe — Module 12 called it a device and this is the operator that confirms it.

-t is the one that does not take a filename:

$ [ -t 0 ] && echo "stdin is a terminal" || echo "stdin is not a terminal"
stdin is a terminal

$ echo | bash -c '[ -t 0 ] && echo terminal || echo "not a terminal"'
not a terminal

-t 0 asks whether file descriptor 0 is attached to a terminal, which is how a program knows whether a human is typing at it. It is the mechanism behind every tool that behaves differently in a pipe than on a screen — ls dropping its columns, grep dropping its colour — and you can use it the same way:

if [ -t 1 ]; then
  echo "progress: 40%"          # a human is watching
fi

7. Comparing Files to Each Other

Three operators take two filenames.

$ touch -t 202001010000 old.txt
$ touch new.txt
$ [ new.txt -nt old.txt ] && echo "new is newer"
new is newer
$ [ old.txt -ot new.txt ] && echo "old is older"
old is older

-nt and -ot compare modification times, and they are how a hand-rolled rebuild check works:

if [ "$src" -nt "$obj" ]; then
  echo "recompiling $src"
fi

That is the core of what make does, in one line. make does it better — dependency graphs, parallelism, transitive rebuilds — but when you need to know whether a filelist has changed since a compile, this is the test.

With one trap:

$ [ missing.txt -nt old.txt ] && echo "true" || echo "false"
false

A missing operand gives false, not an error. Which is reasonable for -nt and dangerous for the inverse: [ "$obj" -ot "$src" ] is false when $obj does not exist, so a rebuild check written that way skips the very first build. Test for existence separately:

if [ ! -f "$obj" ] || [ "$src" -nt "$obj" ]; then
  echo "building $obj"
fi

And -ef asks whether two names are the same file — same device, same inode:

$ [ f.txt -ef f-link ] && echo "same file"
same file

It follows symlinks, so a link and its target are "the same file" by this test. It is the right way to check whether $dest and $src are secretly the same path before copying one onto the other and destroying it.

8. A Complete Script

Every guard in this chapter, in the shape they actually take at the top of a real script:

#!/usr/bin/env bash
# summarise.sh — summarise a simulation log, refusing anything unusable
# usage: summarise.sh <logfile>
#
# exit: 0 summarised · 2 the argument is unusable · 3 the log is empty

log=${1:-}

if [ -z "$log" ]; then
  echo "usage: ${0##*/} <logfile>" >&2
  exit 2
fi

if [ -d "$log" ]; then
  echo "${0##*/}: $log is a directory, not a log" >&2
  exit 2
fi

if [ -L "$log" ] && [ ! -e "$log" ]; then
  echo "${0##*/}: $log is a broken symlink -> $(readlink "$log")" >&2
  exit 2
fi

if [ ! -f "$log" ]; then
  echo "${0##*/}: no such log: $log" >&2
  exit 2
fi

if [ ! -r "$log" ]; then
  echo "${0##*/}: $log exists but is not readable by $(id -un)" >&2
  exit 2
fi

if [ ! -s "$log" ]; then
  echo "${0##*/}: $log is empty — the tool produced no output at all" >&2
  exit 3
fi

printf 'log      %s\n'   "$log"
printf 'lines    %s\n'   "$(( $(wc -l < "$log") ))"
printf 'errors   %s\n'   "$(grep -c 'UVM_ERROR' "$log")"
printf 'warnings %s\n'   "$(grep -c 'UVM_WARNING' "$log")"

The $(( ... )) around wc -l is not decoration. wc right-pads its count on BSD systems, so "$(wc -l < "$log")" is the string 3 and the line prints with a gaping hole in it. Wrapping it in Chapter 8.5's arithmetic expansion evaluates the string as a number and prints it without the padding — the shortest way to normalise a numeric field that came out of another tool.

Six guards, six distinct messages, and every one of them names a real failure mode:

$ ./summarise.sh
usage: summarise.sh <logfile>

$ ./summarise.sh /tmp
summarise.sh: /tmp is a directory, not a log

$ ln -s gone.log broken.log
$ ./summarise.sh broken.log
summarise.sh: broken.log is a broken symlink -> gone.log

$ ./summarise.sh nope.log
summarise.sh: no such log: nope.log

$ echo x > locked.log && chmod 000 locked.log
$ ./summarise.sh locked.log
summarise.sh: locked.log exists but is not readable by engineer

$ : > blank.log
$ ./summarise.sh blank.log
summarise.sh: blank.log is empty — the tool produced no output at all

$ printf 'UVM_ERROR : x\nUVM_WARNING : y\nUVM_ERROR : z\n' > run.log
$ ./summarise.sh run.log
log      run.log
lines    3
errors   2
warnings 1

Note the order. The broken-symlink check comes before -f, because -f is false for a broken link and would have reported "no such log" — a true statement that sends the reader looking in the wrong place. The specific diagnosis has to be tested before the general one, which is the same ordering argument Chapter 16.1 made about elif.

Practice

1. Rebuild the fixture set from Section 1 and reproduce the whole matrix yourself with a nested loop. Compare it against the table and account for any difference.
2. Run [ -e dangling ] and [ -L dangling ] on a broken symlink and explain the opposite answers in one sentence.
3. Write the [ -e "$f" ] && rm "$f" cleanup from the Section 3 callout, run it on a directory containing a broken symlink, and confirm the link survives. Then fix it.
4. chmod 000 a file you own and run -e, -f, -s, -r against it. Now run the same four tests under sudo and explain which changed.
5. Create a file, a directory and an empty file, then find a single [ ] test that is true for only the first of the three.
6. Run [ -t 0 ] at your prompt, then echo | bash -c '[ -t 0 ]; echo $?'. Use the difference to make a script print a progress line only when a human is watching.
7. Build the rebuild check from Section 7 for a source and an object file. Delete the object and show that the naive -nt version skips the build, then that the fixed version does not.
8. Add a seventh guard to summarise.sh that rejects a log older than seven days. You do not have the operator — construct it from find -mtime (Chapter 14.6) and an exit status.

Check Your Understanding

1. Which single operator would you use before reading a file, and why not -e?
2. -e is false for a broken symlink and -L is true. Why?
3. [ -s somedir ] is true. What does that tell you about -s?
4. What is the difference between -e and -f?
5. A log exists, is readable and contains no errors, so your script reports a pass. What did it fail to check?
6. [ "$obj" -ot "$src" ] is false when $obj does not exist. Why is that a problem for a rebuild check?
7. [ -x "$dir" ] is true for almost every directory. What is it actually telling you?

9. Remember

• -r before reading, -w before writing, -x before running. A file can exist and be unopenable; -e has nothing to say about permissions.
• -r asks about you. The same file gives different answers for different users, and root bypasses the check entirely.
• Every file test except -L follows symlinks. A broken link is false for -e and true for -L.
• A cleanup guarded by -e skips every broken symlink — exactly the files it was written to remove.
• -f means regular file and is false for a directory. -e is true for anything that exists.
• -s is true for a directory, because a directory has a size. Pair it with -f.
• -x on a directory means traversable, not executable.
• [ -f "$log" ] && ! [ -s "$log" ] is a killed tool. An empty log looks exactly like a clean one to grep.
• -nt and -ot return false for a missing operand, not an error. Test existence separately.
• -t 1 tells you a human is watching — the same mechanism that makes ls drop its columns in a pipe.
• Test the specific diagnosis before the general one. A broken-symlink check has to come before -f, or it will never run.
• A guard is a check, not a lock. For temporary files use mktemp rather than test-then-create.
• Validating a filelist before the tool sees it is four file tests — Chapter 26.5 turns a four-hundred-line compile error into one line naming the missing file.

Chapter 16.5 joins conditions together. You have already seen [ -L "$f" ] && [ ! -e "$f" ] and [ ! -f "$obj" ] || [ "$src" -nt "$obj" ] used without explanation — the next chapter is why they are written as two separate tests rather than one, and why the operators that look like they belong inside the brackets should stay outside them.