Bash Quoting and Variable Expansion

Part III · Module 8 · Chapter 8.2 · Bash Quoting and Variable Expansion

1. Two Languages, One Line

   sed 's/[[:space:]]*$//' file
       └────────┬───────┘
       the shell reads this first and decides what sed receives

The shell is not passing your text through. It is parsing it, possibly changing it, and then calling sed with the result. sed never sees what you typed; it sees what survived.

That one fact explains a whole class of baffling errors. When sed complains about a character you are certain you did not type, the shell put it there.

2. This Is Not sed's Escaping

Two escaping systems meet on that line, and conflating them is the usual confusion.

   's/\.sv$/.svh/'
      └┬┘  └┬┘
       │    └── sed's anchor
       └─────── sed's escape — a literal dot

Inside single quotes, the shell changes nothing at all. Every backslash, dollar and asterisk reaches sed exactly as written, so every escape you see is sed's, and Chapter 3.3's rules apply unmodified.

Inside double quotes, the shell gets a say first — and then sed sees the result. Now you are reasoning about two layers, and a backslash might be consumed before sed ever looks at it.

The rule: single-quote every sed script. The rest of this chapter is what that buys you and the one case where you cannot.

3. What Actually Differs

It is worth being precise here, because the usual advice is a long list of scary characters and most of them do not bite.

My first attempt at a demonstration used the registered example:

sed 's/[[:space:]]*$//' filelists/top.f | cat -et

+incdir+/proj/old/rtl$
/proj/old/rtl/alu.sv$

sed "s/[[:space:]]*$//" filelists/top.f | cat -et

+incdir+/proj/old/rtl$
/proj/old/rtl/alu.sv$

Identical. Both stripped the trailing whitespace. The $ survived in double quotes because $/ is not a valid parameter expansion — bash leaves a dollar alone unless a name or a brace follows it. So this example does not distinguish the two quotings at all.

The honest discriminator is a $ followed by a name:

cat money.txt

cost $HOME total

sed 's/$HOME/X/' money.txt

cost X total

Single quotes: the shell passed $HOME through untouched, sed matched the literal text, and replaced it.

sed "s/$HOME/X/" money.txt

sed: 1: "s//Users/azharuddinsaif ...": bad flag in substitute command: 'a'

Exit status 1. The shell expanded $HOME to a path, that path is full of slashes, and the slashes collided with the / delimiter — Chapter 3.3's delimiter error, caused by text sed was handed rather than text anyone wrote.

That single example shows both halves: the expansion happened, and it broke the command.

4. Two Warnings That Did Not Reproduce

Being specific cuts both ways. Two things people routinely warn about behaved identically in both quotings here.

Backslashes:

sed 's/\(a\)/[\1]/' one.txt

[a]

sed "s/\(a\)/[\1]/" one.txt

[a]

Same. Bash leaves a backslash alone in double quotes unless it precedes $, `, ", \ or a newline, and \( is none of those.

A glob character:

sed 's/a*/X/' one.txt

X

sed "s/a*/X/" one.txt

X

Same again. The * would only expand if the shell found matching filenames, and inside quotes of either kind it does not try.

5. When a Variable Must Expand — the Loud Failure

Sometimes the value genuinely comes from outside. Then you need double quotes, and the first hazard is immediate.

The registered example, with well-behaved values:

OLD=/proj/old; NEW=/proj/new; sed "s|$OLD|$NEW|g" filelists/top.f

+incdir+/proj/new/rtl
/proj/new/rtl/alu.sv

Correct — and note it uses | as the delimiter precisely because the values are paths full of slashes.

Now give one of those values a | in it:

OLD="/proj/old|x"; NEW=/proj/new; sed "s|$OLD|$NEW|g" filelists/top.f

sed: 1: "s|/proj/old|x|/proj/new|g
": bad flag in substitute command: '/'

Exit status 1. The value contained the delimiter, so sed saw five separators where it expected three. The same thing with a path value and a / delimiter:

OLD=/proj/old; sed "s/$OLD/X/" filelists/top.f

sed: 1: "s//proj/old/X/
": bad flag in substitute command: 'o'

There is no delimiter that is safe against arbitrary values. You can pick one unlikely to appear — |, #, % — and that is a bet, not a guarantee. The honest mitigations, in order:

1. Control the values. If $OLD comes from a config file you own, the bet is a good one and a comment saying which delimiter is assumed is enough.
2. Validate before substituting. case "$OLD" in *"|"*) echo "value contains the delimiter" >&2; exit 1;; esac turns a confusing sed error into a clear one.
3. Escape the value, which is correct and ugly — and still leaves the replacement side's & and \ to handle, per Chapter 3.3.
4. Do not build the script from untrusted input at all. If the value comes from a user, a filename or a network, section 7 applies.

6. The Silent One — an Unset Variable

This is the failure worth remembering, because nothing reports it.

The registered example updates a test name in a run script:

cat sim/run.sh

#!/bin/sh
vcs -sverilog +UVM_TESTNAME=smoke top.sv

TEST=stress; sed "s/+UVM_TESTNAME=[a-z_]*/+UVM_TESTNAME=$TEST/" sim/run.sh

#!/bin/sh
vcs -sverilog +UVM_TESTNAME=stress top.sv

Correct. Now run exactly the same command with TEST never set — a typo in the variable name, a export that did not happen, a CI job that forgot an environment:

unset TEST; sed "s/+UVM_TESTNAME=[a-z_]*/+UVM_TESTNAME=$TEST/" sim/run.sh

#!/bin/sh
vcs -sverilog +UVM_TESTNAME= top.sv

Exit status 0. An unset variable expands to an empty string, so the substitution ran perfectly and produced +UVM_TESTNAME= with nothing after it. The file is still a valid shell script. It will be committed, it will run, and the simulator will fail later with a message about a missing test name that points nowhere near this command.

One more related trap: an unquoted variable is also split on whitespace:

T="a b"; sed s/x/$T/ /dev/null

sed: 1: "s/x/a
": unescaped newline inside substitute pattern

The shell split s/x/a b/ into two arguments, s/x/a and b/, so sed got an incomplete script and the second word became a filename. Quote the expansion — "s/x/$T/" — even when you are already inside a larger quoted string.

7. The Pattern That Avoids All of This

If the script is more than a line or two, or takes values from anywhere you do not control, stop building it with string interpolation. Put it in a file.

Chapter 6.4 made the case: a -f script file is read by sed directly, so the shell never sees it at all. No quoting, no expansion, no delimiter collisions from values you did not anticipate, and the script is reviewable in a diff.

Where a value genuinely must vary, the clean shapes are:

• Generate the script file from a template in a language with real string handling, then run sed -f. Chapter 8.4 is about when that is the right call.
• Keep the script single-quoted and pass the variable as data, not as code. sed has no -v equivalent — awk does, which is one more reason parameterised text processing often belongs there.

8. Think It Through

1. Why does sed "s/$HOME/X/" file fail while sed 's/$HOME/X/' file works?

In double quotes the shell expands $HOME to a path containing slashes, which collide with the / delimiter — section 3 measured bad flag in substitute command: 'a', exit 1. In single quotes $HOME is four literal characters that sed matches as text. The two commands are not variants of one thing; they are different commands.

2. Someone says "always single-quote because of backslashes and globs". Is that right?

The conclusion is right, the reasoning is partly folklore. Section 4 measured both of those behaving identically in single and double quotes here. The real reasons are a $ followed by a name, an unset variable expanding to nothing, and not having to know which of the two languages ate your character. Single-quote because it removes the question, not because every listed character is dangerous.

3. Your CI pipeline rewrites a test name and the simulation fails with "no test specified". The sed exited 0. What happened?

The variable was unset, so it expanded to an empty string and sed faithfully produced +UVM_TESTNAME=. Section 6 measured exactly this. Add ${TEST:?TEST must be set} or set -u, and the job fails at the right place with a message naming the right variable.

4. Write a guard that rejects a path value containing the delimiter before sed sees it.

case "$OLD" in *"|"*) echo "OLD contains the | delimiter" >&2; exit 1;; esac. Three lines that turn bad flag in substitute command into a message naming the variable and the character. Worth it in any script where the value comes from a file or an argument rather than from the line above.

5. When should a sed script stop being an argument and become a file?

When it needs a variable it cannot safely interpolate, when it has a label — Chapter 6.3 showed branching is only reliably writable in a file — or when it is long enough that someone will have to review it. A -f file is never seen by the shell, which removes this entire chapter's problems at once.

6. sed "s/a/$REPL/" file works for months and then corrupts a file. What is the likely value of REPL?

Something containing /, & or \. A / ends the command early (section 5); an unescaped & expands to the whole match, which Chapter 3.3 measured silently producing wrong text; a backslash starts an escape. The first fails loudly and the other two do not, which is why validating the value beats hoping about it.

9. Summary

• Two languages share the command line. The shell parses your text first; sed reports on what survived, which is why errors name characters you never typed.
• Single-quote every sed script. Inside single quotes the shell changes nothing, so every escape is sed's and Chapter 3.3's rules apply directly.
• The real discriminator is $ followed by a name. 's/$HOME/X/' matched literal text; "s/$HOME/X/" expanded to a path and died with bad flag in substitute command: 'a', exit 1.
• Two common warnings did not reproduce here: \( and a* behaved identically in both quotings. Single-quote to avoid the question, not because every listed character bites.
• When a variable must expand, a value containing the delimiter fails loudly — exit 1, no file touched. There is no delimiter that is safe against arbitrary values; validate instead.
• An unset variable fails silently. +UVM_TESTNAME=$TEST with TEST unset produced +UVM_TESTNAME= and exit 0 — a run script that is broken and looks fine.
• Guard it: ${VAR:?message} stops before sed runs; set -u does it for a whole script, alongside set -o pipefail.
• An unquoted expansion is also word-split — s/x/$T/ with T="a b" became two arguments and an incomplete script.
• A -f script file is never seen by the shell, which removes every problem in this chapter at once.

Quoting gets one command safely to sed. The next question is what that command is applied to — because a sweep across a tree is only as good as the list of files it was handed, and that list is where the most expensive mistakes in this track live. Chapter 8.3 — Batch Editing and Safe File Selection is about building it deliberately, including the measured case where an ordinary filename with a space in it turned one file into three that do not exist.