Regex Overmatching and Unexpected Replacements

Part III · Module 7 · Chapter 7.3 · Regex Overmatching and Unexpected Replacements

1. The Bug With No Error

Every failure in Part I and II announced itself somehow — a diagnostic, a non-zero status, output that was obviously empty. This one does not.

   sed ran.              exit status 0.
   the pattern matched.  the replacement happened.
   the result is wrong.

Nothing is broken. The command was correct and the instruction was too wide. That is why this is a debugging chapter and not a regex chapter: the skill is not writing patterns, it is finding out what a pattern actually touches before it touches it.

Chapter 7.2 gave you the trigger — a count that does not match your prediction. This chapter is what to do next.

2. The Case: an Unanchored Identifier

Rename an AXI signal. Here is the RTL:

cat -n rtl/bus.sv

     1	logic axi_awvalid;
     2	logic axi_awvalid_d;
     3	logic axi_awvalid_q;
     4	assign axi_awvalid = req & ~stall;
     5	logic axi_wvalid;

Five lines. Exactly two of them mention the signal axi_awvalid — lines 1 and 4. Lines 2 and 3 are different signals: axi_awvalid_d and axi_awvalid_q are the pipelined copies, and renaming them is a bug.

The obvious command:

sed 's/axi_awvalid/m_&/g' rtl/bus.sv

logic m_axi_awvalid;
logic m_axi_awvalid_d;
logic m_axi_awvalid_q;
assign m_axi_awvalid = req & ~stall;
logic axi_wvalid;

Four lines changed. Two of them should not have been.

This is Chapter 4.3's substring rule arriving with consequences. axi_awvalid appears inside axi_awvalid_d, so the pattern matched — correctly, by the rules. Line 5 was spared only because axi_wvalid is a genuinely different string.

In a five-line file you can see it. In a tree of four hundred RTL files, the diff is two thousand lines and nobody reads it closely enough to notice that three signals gained a prefix they should not have.

3. The Fix, and Its Own Trap

The tool for "this exact identifier, not a longer one containing it" is a word boundary — a zero-width assertion that the match must start and end at the edge of a word.

Chapter 4.3 promised this chapter would supply it. Here it is, in the BSD/macOS spelling:

sed 's/[[:<:]]axi_awvalid[[:>:]]/m_&/g' rtl/bus.sv

logic m_axi_awvalid;
logic axi_awvalid_d;
logic axi_awvalid_q;
assign m_axi_awvalid = req & ~stall;
logic axi_wvalid;

Five lines out, and exactly the two real occurrences renamed — lines 1 and 4. axi_awvalid_d, axi_awvalid_q and axi_wvalid are untouched. [[:<:]] matches the start of a word and [[:>:]] the end, so the match cannot extend into a longer identifier.

Now the trap. The spelling you will find in almost every example online is GNU's \b:

sed 's/\baxi_awvalid\b/m_&/g' rtl/bus.sv

logic axi_awvalid;
logic axi_awvalid_d;
logic axi_awvalid_q;
assign axi_awvalid = req & ~stall;
logic axi_wvalid;

Nothing changed. Exit status 0. BSD sed does not implement \b as a word boundary, so the pattern asked for a literal b followed by axi_awvalid, found none, and reported success.

4. The Second Cause: a Greedy .*

The other classic. A pattern meant to match one quoted string:

printf 'set x "a" ; set y "b"\n' | sed 's/".*"/"Z"/'

set x "Z"

Half the line is gone. .* is greedy — Chapter 4.2 measured it — so ".*" matched from the first quote to the last, swallowing a" ; set y "b in the middle. The replacement was faithful; the match was enormous.

The fix is the negated class, which cannot cross the delimiter:

printf 'set x "a" ; set y "b"\n' | sed 's/"[^"]*"/"Z"/'

set x "Z" ; set y "b"

[^"]* means "any run of characters that are not quotes", so it stops at the second quote instead of the last. sed has no non-greedy quantifier, so this idiom is the answer — and it is worth knowing by heart, because "my .* ate too much" is the single most common regex complaint there is.

The same shape appears with paths. s|.*/|| strips to the last slash, which is right for a basename and wrong for a first directory; s|^/\([^/]*\)/.*|\1| is the version that stops at the first.

5. The Discipline: Count Before, Count After

Three commands turn this class of bug from "found in review, maybe" into "found before it happened".

Count the lines a command would change, using the p flag so only real changes are counted:

sed -n 's/axi_awvalid/&/gp' rtl/bus.sv | wc -l

       4

Four — and you expected two. That one number is the whole diagnosis. Note the trick in that command: the replacement is &, so the substitution replaces the match with itself and changes nothing, while the p flag still fires on every line it matched. It is a pure measurement with no edit in it.

Count per file, so you know where the surprises are:

grep -rc 'axi_awvalid' rtl/ | grep -v ':0$'

rtl/bus.sv:4

grep -rc prints a count for every file including zeros; the second grep drops the files with none. On a real tree this is the list you scan for outliers — the file with 40 hits when every other has 2 is where your pattern is too wide.

Then compare. Predicted two, measured four, so stop. After the fix:

sed -n 's/[[:<:]]axi_awvalid[[:>:]]/&/gp' rtl/bus.sv | wc -l

       2

Two. Now the edit is worth running, and Chapter 7.2's diff will confirm it.

6. Narrowing a Pattern, in Order of Preference

When a pattern is too wide, these are the tools, roughly best first:

1. Anchor it — ^ or $ if the target has a known position. Portable, free, and Chapter 4.3's subject. Works only when the target is at a line edge.
2. Use a negated class instead of .* — [^"]*, [^/]*. Portable, and the fix for greediness.
3. Add context you know is there — wire *ready; rather than ready. Portable, and often the most readable: you are describing the declaration, not the word.
4. Word boundaries — [[:<:]] on BSD, \b or \< on GNU. The right tool semantically, and not portable, so in a shared script either branch on the sed or use option 3.
5. Scope with an address — /^assign/s/…/…/, from Chapter 5.4. Narrows which lines rather than what matches, and composes with all of the above.

Option 3 deserves more credit than it gets. For a declaration, s/\(logic \)axi_awvalid;/\1m_axi_awvalid;/ is portable, obvious to a reviewer, and cannot match _d or _q because of the trailing semicolon — no word-boundary syntax required.

7. Think It Through

1. What distinguishes this chapter's bugs from Chapter 7.4's?

Visibility. Here sed succeeds and the result is wrong — no message, exit 0, and the only evidence is in the output. In Chapter 7.4 sed refuses to run and prints a diagnostic. Counter-intuitively the loud ones are easier: a message tells you where to look, while an over-match tells you nothing until someone reads the diff.

2. Your rename changed 0 lines and exited 0. Is that good?

No. Section 3 measured exactly this: \b on BSD sed matched nothing and reported success. Zero is as wrong as four when you predicted two. Any count that is not your prediction means stop — including zero, which is the one people accept because it looks safe.

3. Why does s/".*"/"Z"/ destroy a line with two quoted strings?

.* is greedy, so the match runs from the first quote to the last and the replacement collapses everything between. Use s/"[^"]*"/"Z"/, which cannot cross a quote. sed has no non-greedy quantifier; the negated class is the substitute.

4. Write a portable command that renames axi_awvalid and not axi_awvalid_q, without word boundaries.

Use context you know is there. sed 's/axi_awvalid;/m_axi_awvalid;/g' handles the declaration via the semicolon; sed 's/axi_awvalid \([=&]\)/m_axi_awvalid \1/g' handles the assignment. Less elegant than [[:<:]], completely portable, and a reviewer can see what it excludes. Covering every usage this way is laborious, which is itself an argument for section 6's last callout.

5. grep -rc 'clk' rtl/ shows one file with 180 hits and the rest with about 6. What does that tell you?

Probably nothing is wrong — that file is likely a clock-generation block or a testbench. What it tells you is where to look first, because an outlier is where a wide pattern does the most damage. Open that file's dry run before the others; if the 180 includes clk_div, gclk and clk_en, your pattern needs narrowing before it goes anywhere near the tree.

6. A sweep ran last week and a bug has just surfaced. How do you find out whether the sweep caused it?

git log -p on the affected file, or git diff <commit>~ <commit> -- path, and read what the sweep actually changed there. This is the deferred payoff of Chapter 7.2's "commit first": a sweep inside a commit is a reviewable artifact weeks later, and a sweep over a dirty tree is not reconstructable at all.

8. Summary

• This is the bug class where sed succeeded and the answer is wrong. No message, exit 0, nothing to read but the output.
• An unanchored identifier matches inside longer ones. s/axi_awvalid/m_&/g renamed four lines where two were correct, because _d and _q contain the name.
• Word boundaries are the semantic fix and are not portable. [[:<:]]…[[:>:]] renamed exactly 2 lines on BSD; GNU's \b renamed 0 and exited 0 on the same sed.
• A fix that changes zero lines has not worked, however clean the exit status. Zero is as wrong as four when you predicted two.
• .* is greedy. s/".*"/"Z"/ ate the middle of the line; s/"[^"]*"/"Z"/ is the fix, because sed has no non-greedy quantifier.
• Count before you edit: sed -n 's/pat/&/gp' file | wc -l measures without changing anything, because the replacement is the match itself.
• Count per file: grep -rc 'pat' dir/ | grep -v ':0$' shows where the outliers are.
• Narrow in this order: anchor, negated class, known context, word boundary, address scope. Known context is portable and under-used.
• A language rename with a text tool has limits — sed cannot tell a comment from a declaration. Chapters 9.2 and 8.4 take that seriously.

Every bug in this chapter was silent. The opposite problem — sed refusing to run and printing something terse and unhelpful — is the last chapter of this module. Chapter 7.4 — Common sed Errors and Debugging maps each diagnostic to its cause, including the one where forgetting the script produces a loud error or a completely silent no-op depending on the first letter of your filename.