Linux & Bash · Part I · Module 9 · Quoting and Shell Expansion
The Expansion Pipeline: What Bash Does Before Running Anything
You have been told six times to put quotes around an expansion and never told what quotes actually do. The answer is that Bash rewrites your line in a fixed sequence of stages before any program starts, and quoting is how you switch individual stages off.
Intermediate19 min readLinuxPart IFundamentalsQuotingExpansion
Part I · Module 9 · Chapter 9.1 · The Expansion Pipeline
Most people learn quoting as a list of rules that mostly work. This module teaches it as one mechanism, because there is only one, and once you can see it the rules stop needing to be memorised.
1. The Command Never Sees What You Typed
Chapter 8.1 made this point about variables. It is true far more generally.
When you press Enter, Bash takes the text of your line and rewrites it, in a fixed sequence of stages. Only when every stage has run does it start a program — and what that program receives is the finished text, as a list of separate arguments.
So there is no point asking "what does ls do with a star?" ls never sees a star. By the time ls exists, the star is gone and has been replaced by file names.
2. Watching One Line Go Through
Take a line with several things happening at once:
echo ~/project/run_{1..2}/$name/*.logStage by stage, with name=alu:
| After stage | The line is |
|---|---|
| 1 — braces | ~/project/run_1/$name/*.log ~/project/run_2/$name/*.log |
| 2 — tilde | /home/engineer/project/run_1/$name/*.log …run_2… |
| 3 — variables | /home/engineer/project/run_1/alu/*.log …run_2… |
| 4 — splitting | two words (no spaces appeared, so nothing further split) |
| 5 — file names | each *.log replaced by the logs that exist |
| 6 — quote removal | nothing to remove here |
Two words went in and some number of file names came out. echo received those names and nothing else.
Nothing in that sequence is negotiable, and the next three sections prove the three places where the order is doing visible work.
3. Proof One: Braces Really Are First
$ echo {1..3}
1 2 3
$ n=5
$ echo {1..$n}
{1..5}The second one did not expand. Not because the syntax is wrong, but because brace expansion happens at stage 1 and $n is not replaced until stage 3 — so when the brace stage looked at {1..$n}, the end of the range was the literal text $n, which is not a number, so the whole thing was left alone.
And by stage 3 it is too late: $n becomes 5, giving the text {1..5}, but the brace stage is finished and nothing goes back.
4. Proof Two: Matching Happens After Splitting
This is the most valuable fact in the module, and it explains a failure that otherwise looks like the shell being inconsistent.
Set up a directory with an awkwardly named file:
$ ls
a.txt b.txt two words.txtNow reach that file two ways. First with a pattern:
$ for x in *.txt; do echo " [$x]"; done
[a.txt]
[b.txt]
[two words.txt]Three items, and the spaced name arrived whole. Now with a variable holding exactly the same text:
$ v="two words.txt"
$ for x in $v; do echo " [$x]"; done
[two]
[words.txt]The same characters, split in one case and not the other.
There is nothing inconsistent about it. Look at the order:
- The variable was substituted at stage 3, so its result went through stage 4 and was split at the space.
- The pattern was matched at stage 5 — after splitting had already finished. Its results are never offered to the splitting stage at all.
Here is the same point as the mistake people actually make:
$ for f in $(ls *.txt); do echo " [$f]"; done
[a.txt]
[b.txt]
[two]
[words.txt]
$ for f in *.txt; do echo " [$f]"; done
[a.txt]
[b.txt]
[two words.txt]Four items versus three. The first loop will try to process two files that do not exist.
The other half of the same stage
A value that arrives from a variable is also offered to stage 5:
$ f="*.txt"
$ echo $f
a.txt b.txt two words.txt
$ echo "$f"
*.txtUnquoted, the variable was replaced at stage 3 by the text *.txt, which then reached stage 5 and was matched against real files. Quoted, stage 5 is switched off for that word, so the pattern survives as text.
A pattern stored in a variable still matches when the expansion is unquoted. That surprises people in both directions — sometimes it is exactly what you wanted, and sometimes it silently turns a configuration string into a list of files.
5. Proof Three: Quote Removal Is Last
$ printf '[%s]\n' "hello" 'world'
[hello]
[world]No quote characters in the output. They were consumed by the shell at stage 6 and never reached printf.
This matters more than it sounds, because it means quotes are not part of your data. If you want a literal quote character in a value, you have to arrange it deliberately, which is Chapter 9.4.
It also explains why this does not do what it looks like:
$ q='"'
$ echo $q hello
" helloThe quote arrived as a character at stage 3, long after the stage that would have treated it as syntax. Characters that appear via an expansion are never re-read as quoting. That is a safety property — it is the reason a file name containing a quote cannot rewrite your command — and it is also why you cannot build a quoted command line by assembling it in a string.
6. Literal Text Is Not Split
One more consequence of the order, easy to miss:
$ printf '[%s]\n' two words.txt
[two]
[words.txt]
$ printf '[%s]\n' "two words.txt"
[two words.txt]That first case looks like word splitting, and it is not. The line was typed as two words, separated by a space you wrote. Stage 4 never came into it.
The distinction matters because it is what quoting is for in the first place. The shell has two different reasons to break a line into words:
- Tokenising what you typed — which happens before any of the six stages, and which quotes control directly.
- Word splitting at stage 4 — which applies only to the results of stage 3.
Double quotes switch off the second. Both kinds of quote switch off the first. Keeping them apart is what makes Chapter 9.3 make sense.
7. Reading a Bug Off the Diagram
The point of the order is diagnostic. Three real failures, each identified by naming the stage.
"My script deleted the wrong thing."
$ unset BUILDDIR
$ echo "would run: rm -rf ${BUILDDIR}/*"
would run: rm -rf /*Stage 3 replaced the unset variable with nothing, leaving /*, which stage 5 then matched against everything in the root directory. Two stages, working exactly as specified. (Chapter 9.6 and set -u from Chapter 8.1.)
"It works until a path has a space in it."
Stage 4. The expansion was unquoted.
"My {1..$n} prints braces."
Stage 1 ran before stage 3.
"echo "*" prints a star but echo * lists my files."
Stage 5, switched off by the quotes in the first case.
Every one of those is a stage, named. That is the whole value of knowing the order: a quoting bug stops being mysterious and becomes a question with a specific answer.
Practice
Make a scratch directory with an awkward name in it, because every interesting case needs one:
mkdir -p /tmp/exp && cd /tmp/exp
touch a.txt b.txt "two words.txt"1 — Prove braces are first.
echo {1..3}
n=5; echo {1..$n}
echo $(seq 1 "$n")2 — Prove matching is after splitting. Run both loops and count the items:
for x in *.txt; do echo " [$x]"; done
v="two words.txt"; for x in $v; do echo " [$x]"; done3 — Break the classic idiom.
for f in $(ls *.txt); do echo " [$f]"; done
for f in *.txt; do echo " [$f]"; done4 — Watch a pattern survive in a variable.
f="*.txt"
echo $f
echo "$f"5 — Prove quote removal happens.
printf '[%s]\n' "hello" 'world'
q='"'; echo $q hello6 — Separate tokenising from splitting.
printf '[%s]\n' two words.txt
printf '[%s]\n' "two words.txt"7 — Trace a whole line. Turn on the shell's own trace and watch the finished arguments:
name=alu
set -x
echo ~/project/run_{1..2}/$name
set +xset -x prints each command after expansion, which is a direct view of what stage 6 handed over. It is the single best debugging tool for everything in this module.
8 — Clean up. cd /tmp && rm -rf /tmp/exp
Check Your Understanding
1. Why does n=3; echo {1..$n} print {1..3} rather than 1 2 3?
2. A directory contains a file called my notes.txt. Why does for f in * handle it correctly while for f in $(ls) does not?
3. f="*.txt". What is the difference between echo $f and echo "$f", and which stage accounts for it?
4. Does echo "hello" pass the quote characters to echo?
5. printf '[%s]\n' two words.txt prints two lines. Is that word splitting?
6. You have q='"' and run echo $q hello. Why is the quote treated as an ordinary character?
8. Remember
- Bash rewrites your line before any program starts. The command receives the finished result, as a list of arguments.
- Six stages, always in this order: braces, tilde, variables and command output and arithmetic, word splitting, file-name matching, quote removal.
- Each stage sees only what the one before it produced. That is what makes the order matter.
- Braces are first, so a variable can never supply a brace range. Use
seqor a C-style loop. - File-name matching is after word splitting, so a pattern's results are never split — which is why
for f in *.txtis safe andfor f in $(ls)is not. - Word splitting applies to what an expansion produced, never to what a pattern matched or to literal text you typed.
- A pattern stored in a variable still matches when the expansion is unquoted.
- Quote removal is last, so quotes are syntax and never data — and a quote arriving from an expansion is just a character.
- Quoting does not have rules of its own; it switches stages off. Which stages depends on the quote.
set -xshows you the finished line, and is the best debugging tool in this module — Chapter 20.8 is it in full, including the fact that it quotes any word containing a space.
You now have the order. The next chapter takes stage four on its own — the one stage that turns a working command into a broken one — and shows exactly what it does and how to stop it.
Where this fits
Part of the Linux & Bash curriculum.