Delimiters, Escaping, and Special Characters

Part II · Module 3 · Chapter 3.3 · Delimiters, Escaping, and Special Characters

1. Characters That Do Not Mean Themselves

So far every pattern has been plain text, and plain text is easy — DEBUG means the letters D, E, B, U, G.

The moment your text contains punctuation, that stops being true. A substitution reserves a few characters for its own purposes, and if your data happens to contain them, one of three things happens:

• sed refuses to run and tells you why. This is the good outcome.
• sed runs and corrupts the output, reporting success.
• sed runs, matches more than you meant, and reports success.

This chapter covers one of the first kind and three of the other two. Keep that ranking in mind as you read: the dangerous characters are not the complicated ones, they are the quiet ones.

2. The Delimiter Problem

The delimiter is whatever character follows the s. The convention is /:

s / pattern / replacement /

Which works beautifully until your pattern is a path. Here is the filelist again:

cat -n filelists/top.f

     1	+define+DEBUG
     2	+incdir+/proj/old/rtl
     3	/proj/old/rtl/alu.sv
     4	/proj/old/rtl/fifo.sv
     5	/proj/old/tb/env.sv

Suppose you want to rewrite that +incdir+ path. Written with slashes as the delimiter, and the slashes in the data left alone, it falls apart:

sed 's/+incdir+/proj/old/rtl/+incdir+/proj/new/rtl/' filelists/top.f

sed: 1: "s/+incdir+/proj/old/rtl ...": bad flag in substitute command: 'o'

Exit status 1. Work through what sed saw. The first / opened the pattern. The second / — the one in /proj — closed it, so the pattern was just +incdir+. The third / closed the replacement, making the replacement proj. Everything after that was supposed to be flags, and the next character was o, which is not a flag. Hence the message.

The error names a character from the middle of your path as a flag, which is confusing until you realise sed stopped reading your command three characters in.

3. Two Ways Out, One of Them Better

You can escape the slashes. A backslash before a character tells sed to treat it as data:

sed 's/+incdir+\/proj\/old\/rtl/+incdir+\/proj\/new\/rtl/' filelists/top.f

+define+DEBUG
+incdir+/proj/new/rtl
/proj/old/rtl/alu.sv
/proj/old/rtl/fifo.sv
/proj/old/tb/env.sv

Correct. Also close to unreadable — six backslashes, and if you miscount one the command either errors or silently matches something else. This is the style sometimes called leaning toothpick syndrome, and in a code review it is impossible to check by eye.

Or you can change the delimiter. The character after s is simply whatever you choose, and sed uses it consistently throughout the command:

sed 's|+incdir+/proj/old/rtl|+incdir+/proj/new/rtl|' filelists/top.f

+define+DEBUG
+incdir+/proj/new/rtl
/proj/old/rtl/alu.sv
/proj/old/rtl/fifo.sv
/proj/old/tb/env.sv

Identical result, and now the path is written exactly as it appears in the file. You can read this command against the data and confirm them character by character, which is the whole point.

Rule of thumb: when the text contains slashes, use a different delimiter. | is the most common choice, # and , and % are all fine. The only requirements are that the character does not appear in your pattern or replacement, and that the shell will leave it alone inside single quotes — which | and # both are.

4. The Ampersand Inserts the Whole Match

Now the first character that fails quietly. On the replacement side, & does not mean the ampersand character. It expands to everything the pattern matched.

That is genuinely useful. Here is a simulator run script:

cat -n sim/run.sh

     1	vcs -full64 -timescale 1ns/1ps -o simv
     2	./simv +UVM_TESTNAME=smoke

You need to add -sverilog immediately after the timescale option, without retyping the option itself:

sed 's|-timescale 1ns/1ps|& -sverilog|' sim/run.sh

vcs -full64 -timescale 1ns/1ps -sverilog -o simv
./simv +UVM_TESTNAME=smoke

& stood in for -timescale 1ns/1ps, so the replacement became that text followed by -sverilog. This is the cleanest way to append to or wrap something you matched — you never restate the matched text, so it cannot drift out of sync with the pattern.

You can use it more than once:

printf 'clk\n' | sed 's/clk/&_i, &_o/'

clk_i, clk_o

And here is the trap. If you actually want a literal ampersand in your output, you must escape it. Unescaped, it silently expands:

printf 'clk\n' | sed 's/clk/A & B/'

A clk B

You asked for A & B and got A clk B. Escaped, it behaves:

printf 'clk\n' | sed 's/clk/A \& B/'

A & B

Both ran with exit status 0. There is no diagnostic for the first one, because nothing was wrong from sed's point of view — you used a documented feature.

The same mistake on the run script is worse, because the output still looks like a plausible command line:

sed 's|-timescale 1ns/1ps|\& -sverilog|' sim/run.sh

vcs -full64 & -sverilog -o simv
./simv +UVM_TESTNAME=smoke

Here the escape was what the author wanted — a literal & — and the result is a shell line containing a background-job operator in the middle of a VCS invocation. The command did exactly what was asked and the file is now wrong. Compare it with the correct version at the top of this section: the difference is one backslash.

5. The Dot Matches Any Character

The second quiet one lives on the pattern side. A . in a pattern means "any single character", not "a full stop".

Here is a package filelist:

cat -n filelists/pkg.f

     1	/proj/rtl/alu_pkg.sv
     2	/proj/rtl/fifo_pkg.sv
     3	/proj/rtl/top.sv

The job is to change the extension from .sv to .svh. Escaping the dot — \. means a literal dot — gives:

sed 's/\.sv$/.svh/' filelists/pkg.f

/proj/rtl/alu_pkg.svh
/proj/rtl/fifo_pkg.svh
/proj/rtl/top.svh

Now the same command with the dot left unescaped:

sed 's/.sv$/.svh/' filelists/pkg.f

/proj/rtl/alu_pkg.svh
/proj/rtl/fifo_pkg.svh
/proj/rtl/top.svh

Identical. Every line here ends in a real .sv, so "any character followed by sv" and "a dot followed by sv" happen to pick out the same text. This fixture cannot tell the two commands apart, and if you tested here you would conclude the escape is optional.

Add one line that discriminates them — a file whose name uses an underscore where you expected a dot:

printf 'alu_sv\nalu.sv\n' | sed 's/.sv$/.svh/'

alu.svh
alu.svh

printf 'alu_sv\nalu.sv\n' | sed 's/\.sv$/.svh/'

alu_sv
alu.svh

There it is. With the dot unescaped, alu_sv matched — the . matched the underscore — and the file was renamed to alu.svh, inventing an extension that was never there. With the dot escaped, alu_sv was correctly left alone.

Note also that the dot in the replacement needs no escaping. The replacement is not a pattern; .svh there is three ordinary characters. Only the pattern side has metacharacters.

6. Escapes That Work, and Two That Only Look Like They Do

A few replacement-side escapes are worth knowing, and they were checked byte by byte here rather than taken on trust.

\n in the replacement does produce a newline, splitting one line into two:

printf 'a:b\n' | sed 's/:/\n/' | od -c | head -2

0000000    a  \n   b  \n                                                
0000004

The bytes confirm it: a, newline, b, newline. \t gives a tab the same way. This is worth stating plainly because it is often claimed that the BSD sed on macOS cannot do this — on the sed these examples were run against, it can.

A literal backslash in the output needs doubling:

printf 'a\n' | sed 's/a/x\\y/'

x\y

Now the two that do not work here. GNU sed has case-conversion escapes: \U upper-cases what follows, \L lower-cases it. They are genuinely convenient and they are a GNU extension. On BSD sed they are not an error — the backslash is simply dropped and the letter is taken literally:

printf 'axi_x\n' | sed -E 's/(axi)/\U\1/' | od -c | head -2

0000000    U   a   x   i   _   x  \n                                    
0000007

Exit status 0. The output is Uaxi_x — a literal U followed by the unchanged capture. \L behaves the same way, emitting a literal L.

This is the worst failure shape in the chapter. The command runs, reports success, and produces output that is nearly right — a stray capital letter glued to the front of an identifier. In a sweep across a hundred RTL files that is a hundred broken signal names, with nothing in the exit status to suggest a problem.

If you need case conversion portably, do it outside sed — tr for a whole stream, or awk's toupper, which Chapter 8.1 covers.

7. Ranked by Whether sed Warns You

That ordering is the thing to remember from this chapter, more than any individual character.

Only the first row protects you. The other three are indistinguishable from success unless you look at the output — which is why the diff habit from Chapter 3.1 is not a nicety. Three of the four mistakes in this chapter are invisible to an exit-status check and obvious in a diff.

Module 4 covers the rest of the metacharacters — *, [, ^, $, +, ? — and the dialect question that decides which of them need escaping. The pattern established here holds for all of them: a metacharacter you forgot to escape widens your match silently.

8. Think It Through

1. Why does sed 's/a/b/c/d/' file fail, and what is the error likely to say?

Five delimiters where the command expects three. The pattern is a, the replacement is b, and then c/d/ is read as flags — c is not a valid flag, so sed reports a bad flag and exits 1. This is the section 2 failure with different letters, and it is the reason a path full of slashes cannot use a slash delimiter.

2. You want to turn report_timing into report_timing -nosplit across an STA script. Write it two ways.

Either restate the text, s/report_timing/report_timing -nosplit/, or use the ampersand, s/report_timing/& -nosplit/. The second is better: there is one copy of the text, so a later edit to the pattern cannot leave the replacement stale. That benefit grows with the length of the matched text — compare the two forms for the -timescale 1ns/1ps example in section 4.

3. A rename sweep was supposed to produce AXI_AWVALID and produced Uaxi_awvalid instead. What was the command, and why was there no error?

It used GNU's \U on a sed that does not implement it — almost certainly a script written on Linux and run on a Mac. BSD sed dropped the backslash, emitted a literal U, and expanded the capture unchanged, exiting 0 because nothing was syntactically wrong. The same script on GNU sed would have worked, which is what makes it so hard to find: it is correct on the author's machine.

4. Is \. escaping for sed or for the shell?

For sed. The shell and sed each have their own escaping rules, and they are easy to confuse. Inside single quotes the shell passes your text through untouched, so everything in 's/\.sv$/.svh/' is seen by sed exactly as written, and the backslash is sed's. The moment you use double quotes — usually because you want to interpolate a shell variable — the shell starts interpreting backslashes and dollar signs first, and you need to reason about two layers. Chapter 8.2 is about that; until then, single-quote your scripts.

5. You escape a dot in a pattern and the output is unchanged from the unescaped version. Does that mean the escape was unnecessary?

No — it means your test data cannot tell the difference, which is section 5's whole point. The escape changes the result only on input where some other character sits in the dot's position. If every line in your sample has a real dot there, both commands agree. Construct one line that does not, and check again before you trust the pattern on four hundred files.

9. Summary

• The delimiter is whatever character follows s. If your text contains it, sed stops reading early and reports a bad flag — exit 1.
• When the text has slashes, change the delimiter rather than escaping them. s|…|…| lets you write the path exactly as it appears in the file, so the command can be checked against the data by eye.
• & on the replacement side means the entire match. Use it to append to or wrap matched text without retyping it; escape it as \& when you want a literal ampersand.
• An unescaped & silently inserts the match, and an unescaped . silently matches any character. Both exit 0.
• \n and \t do work in the replacement on the BSD sed used here — verified at the byte level with od -c.
• GNU's \U and \L are not errors on BSD sed — they emit a literal U or L and exit 0, which is the quietest and most damaging failure in the chapter. Do case conversion with tr or awk instead.
• Rank these by whether sed warns you. One of the four mistakes here errors out; three look exactly like success. That is the argument for reviewing a diff rather than an exit status.

The ampersand showed you how to reuse the whole match. The far more powerful version is reusing part of it — keeping the stem of a signal name while changing its prefix, or keeping a corner's voltage while changing its temperature. Chapter 3.4 — Capture Groups and Backreferences is that, and it closes Module 3.