Wildcards: How the Shell Expands Names Before a Command Runs

Part I · Module 3 · Chapter 3.5 · Wildcards

1. The Problem Patterns Solve

A directory with a few dozen files in it:

$ pwd
/home/engineer/project/sim

$ ls
alu.sv  counter.sv  notes.txt  run_1.log  run_10.log  run_2.log

To list only the logs you could name all three. With thirty of them you would not. A wildcard — also called a glob — lets you describe the set instead:

$ ls *.log
run_1.log  run_10.log  run_2.log

Three names matched and were listed. That is the convenience. Now the mechanism, which is the part that matters.

2. The Command Never Sees the Pattern

Before anything else, settle what actually happened in that command.

Chapter 1.2 established step 2 of the shell loop: Bash rewrites the line before the program starts. A wildcard is handled there. Bash reads the names in the directory, finds the ones that match, and replaces the pattern with those names. Only then does ls start.

So ls was not given *.log. It was given three separate operands — run_1.log, run_10.log, run_2.log — exactly as if you had typed them yourself.

Prove it with echo, which simply prints the words it is given:

$ echo *.log
run_1.log run_10.log run_2.log

echo knows nothing about wildcards. It printed filenames because filenames are what it received.

3. * — Any Characters, Including None

The most used pattern. * stands for any run of characters, including an empty one.

$ ls
alu.sv  counter.sv  notes.txt  run_1.log  run_10.log  run_2.log

$ echo *.log
run_1.log run_10.log run_2.log

$ echo *.sv
alu.sv counter.sv

$ echo run_*
run_1.log run_10.log run_2.log

$ echo *
alu.sv counter.sv notes.txt run_1.log run_10.log run_2.log

Reading those: *.log is "anything, then .log". run_* is "run_, then anything". A bare * is "anything at all", which matches every name in the directory.

Two properties of * that are easy to assume wrongly.

It matches an empty run. run* matches a file called exactly run, because * is allowed to stand for nothing.

It does not cross a /. A pattern matches within one directory level:

$ echo *.log
run_1.log run_10.log run_2.log

$ echo */*.log
sub/deep.log

The first did not reach into sub. To look one level down you write the slash yourself. For searching a tree of unknown depth, the right tool is find, which is Module 14.

The sort order is lexical, not numeric

Look carefully at the order:

$ echo run_*.log
run_1.log run_10.log run_2.log

run_10.log comes before run_2.log. Bash sorts the matches as text, and as text 1 sorts before 2, so anything beginning run_1 precedes run_2.

This surprises people the first time, and it matters whenever order is meaningful — processing results in sequence, say. The usual fix is in the naming rather than the pattern: run_001, run_002, run_010 sort correctly as text because every name is the same length.

4. ? — Exactly One Character

$ echo run_?.log
run_1.log run_2.log

? stands for exactly one character — not zero, not two. run_10.log has two characters where the ? is, so it does not match.

That is the whole difference from *, and it makes ? the precise tool when you know the shape of the name:

$ echo run_?.log      # run_1.log, run_2.log — single digit
$ echo run_??.log     # run_10.log — exactly two

5. [...] — One Character From a Set

Square brackets match one character, chosen from the set inside:

$ echo run_[12].log
run_1.log run_2.log

The forms:

A range is a range of characters, not of numbers — [1-9] is one character, so it never matches 10.

$ echo run_[0-9].log
run_1.log run_2.log

$ echo *.[sc]*
alu.sv counter.sv

Brackets are the most precise of the three, and the one to reach for when * would sweep up too much.

6. When Nothing Matches

The behaviour nobody predicts correctly, and a direct consequence of §2.

$ echo *.nosuch
*.nosuch

Bash found no matching names, so it did not replace the pattern. It passed the pattern through as literal text, and echo printed it.

Which produces this:

$ ls *.nosuch
ls: cannot access '*.nosuch': No such file or directory

Read the message with Chapter 1.3's rule. The prefix is ls:, so ls ran — and it is complaining about a file whose name contains an asterisk, because that is literally the operand it was handed.

This is the tell. An error message containing your pattern means the pattern matched nothing, not that something went wrong with the matching. Once you recognise that, the message tells you what happened instead of looking like a glitch.

The consequences for a command that acts rather than looks can be sharper. rm *.bak in a directory with no .bak files tries to remove a file named *.bak, fails, and reports it — harmless. But a command that creates from the same pattern would produce a file with an asterisk in its name, which Chapter 3.4 explained is legal and awkward.

Bash has a setting that makes a non-matching pattern vanish instead, and another that makes it an error. Both are shell options, which are Module 10's subject.

7. * Does Not Match a Leading Dot

Chapter 2.6 showed ls declining to mention names that begin with a dot. Patterns have a matching rule, and it is a separate mechanism with the same effect:

$ ls -a
.  ..  .hidden.log  alu.sv  counter.sv  notes.txt  run_1.log  run_10.log  run_2.log

$ echo *
alu.sv counter.sv notes.txt run_1.log run_10.log run_2.log

.hidden.log is not in the expansion, even though it ends in .log and * supposedly matches anything. A leading dot must be written explicitly to be matched.

This is deliberate, and Chapter 2.3 shows why it must be. If * matched leading dots it would match . and .. — so rm -r * would try to delete the current directory and its parent. The rule exists to make that impossible.

The practical consequence was flagged in Chapter 3.2 and can now be explained properly:

$ cp -r project/* backup/

Every dotfile in project is silently left behind, because * never matched them. The copy looks complete and the configuration is missing. Copy the directory itself — cp -r project backup_project — and nothing is skipped.

8. Patterns Work With Every Command

Because expansion happens in the shell, no command implements wildcards and every command gets them for free:

$ ls *.log            # list them
$ cp *.log archive/   # copy them
$ rm *.log            # remove them
$ wc -l *.log         # count lines in them

One mechanism, four commands, identical behaviour. None of those programs contains any pattern-matching code for file names; each simply receives a list of operands.

This is also the uncomfortable part. ls *.log and rm *.log are the same selection. The only difference is which program is handed the result — and one of those programs has no undo.

9. Make Any Pattern Safe: Put echo in Front

Here is the technique, and it is the most valuable thing in the chapter.

Because the shell expands the pattern before the command runs, you can replace the command with a harmless one and see the exact list the real command would get:

$ echo rm -- *.log
rm -- run_1.log run_10.log run_2.log

That is not an approximation. It is the real command line, fully expanded, printed instead of executed. If that list is what you intended, run it for real — Chapter 1.8's Up arrow recalls the line, and you delete the word echo from the front.

This completes Chapter 3.3's routine. That chapter said to swap rm for ls and read the output; ls works when the targets are directories, and echo works for anything at all because it shows the command line itself rather than the contents of what it names.

$ pwd
/home/engineer/project/sim

$ echo rm -rf run_*
rm -rf run_1 run_2 run_archive

run_archive is in the list. You had not pictured it. The check cost one command and caught Chapter 3.3's Shape 2 before it could do anything.

10. Patterns With a Path

A pattern can appear in any part of a path, and each part is matched separately:

$ ls sim/*.log              # logs directly inside sim
$ ls sim/*/out.log          # out.log one level inside any directory in sim
$ ls */logs/*.log           # one level down, then logs, then any log

The middle one is worth reading carefully: sim/*/out.log matches sim/run_1/out.log and sim/run_2/out.log, but not sim/out.log (no middle level) and not sim/a/b/out.log (too many). Each * is exactly one level, as §3 said.

When the depth is unknown or varies, patterns are the wrong tool and find is the right one — Module 14.

11. Why This Matters in Engineering Work

Patterns are how anyone works with a directory of generated output, and generated output is most of what engineering work produces. Three habits carry over directly.

Select by pattern, verify by echo. Picking out the logs from one batch of runs is natural with a pattern and genuinely dangerous without the check, because the directory also contains things you did not create and have not looked at.

Name files so patterns can select them. run_001.log through run_200.log is selectable, sorts correctly, and does not collide with anything else. Names that vary unpredictably force you to widen the pattern, and a wide pattern is Chapter 3.3's Shape 2 waiting to happen. Naming is a safety decision, not a tidiness one.

Remember that * skips hidden entries. Archiving or copying "everything" with a pattern quietly omits configuration. Name the directory instead.

Part II works with real result directories throughout, and every one of these habits is assumed there.

12. Practice

The directory:

alu.sv  counter.sv  notes.txt  run_1.log  run_2.log  run_10.log  run_archive  .hidden.log

(run_archive is a directory; everything else is a file.)

Question 1. List exactly what each of these expands to: *.log, run_?.log, run_[12].log, *, run_*.

Question 2. Why does *.log not include .hidden.log? Give the rule, and the reason the rule has to exist (use Chapter 2.3).

Question 3. echo *.bak prints *.bak. What does that tell you, and what would rm *.bak do?

Question 4. Put run_1.log, run_2.log and run_10.log in the order Bash produces them, and explain the order.

Question 5. You intend to delete the two numbered run logs only. Write the echo check you would run first, state what you expect it to print, and then the real command.

Question 6. Someone runs rm -rf run_* expecting two directories to go. What else is destroyed, and which earlier chapter named this shape of accident?

Question 7. Explain why ls *.log and rm *.log always operate on exactly the same set of files.

Question 8. What is the difference between [12] and {1,2} in a directory that contains no matching files at all?

13. Remember

• Bash expands the pattern; the command receives names. No command implements wildcards, and none ever sees the pattern.
• * is any run of characters including none; ? is exactly one; [...] is one from a set. None of them crosses a /.
• Matches are sorted as text, so run_10 comes before run_2. Pad numbers — run_010 — when order matters.
• A pattern that matches nothing is passed through literally, which is why an error message containing your pattern means "no matches", not "something broke".
• * never matches a leading dot. That rule is what stops rm -r * from reaching . and .. — and is why a pattern-based copy silently loses dotfiles.
• ls *.log and rm *.log select identically. The expansion does not know or care which command follows.
• Put echo in front to see the real, fully expanded command line before running it. This is the complete version of Chapter 3.3's routine and works with any command.
• rm * acts on whatever directory you are actually in. pwd, then echo, then run it.
• Braces are not wildcards. {1,2} generates text regardless of what exists; [12] only matches real names. Module 9.

One chapter of Module 3 remains. Having created, copied, deleted and selected files, the last step is bundling a set of them into a single file you can keep or hand to somebody else — and taking one apart without scattering its contents across your working directory.