Global and Occurrence-Based Replacement

Part II · Module 3 · Chapter 3.2 · Global and Occurrence-Based Replacement

1. The Question This Chapter Answers

A substitution command can end with a flag — a character after the closing delimiter that changes how the substitution behaves:

s | /proj/old | /proj/new | g
                            │
                            └─ the flag: which matches to replace

There are three answers you will ever need:

That is genuinely the whole chapter. The rest of it is making sure you can predict each one, and showing the two ways the number form fails quietly.

2. One Line, Three Matches

Everything below runs against this filelist:

cat -n filelists/paths.f

     1	+incdir+/proj/old -y /proj/old/rtl -v /proj/old/lib.v
     2	/proj/old/rtl/alu.sv

Line 1 is a realistic compile line: an include directory, a library search path, and a library file, all under the same tree. /proj/old appears on it three times. That is what makes it a useful test — a fixture with one match per line cannot tell these flags apart.

Two notes on the commands that follow. They use | as the delimiter instead of /, because the text being matched is full of slashes; Chapter 3.3 is about that choice. And most of them start with a 1, which restricts the command to line 1 so the comparison stays readable — that is an address, and Chapter 4.4 covers addresses properly.

3. No Flag, and g

With no flag, you get the behaviour Chapter 3.1 ended on:

sed '1s|/proj/old|/proj/new|' filelists/paths.f

+incdir+/proj/new -y /proj/old/rtl -v /proj/old/lib.v
/proj/old/rtl/alu.sv

One of three. The include directory moved; the -y and -v paths did not.

Add g — for global — and every match on the line is replaced:

sed '1s|/proj/old|/proj/new|g' filelists/paths.f

+incdir+/proj/new -y /proj/new/rtl -v /proj/new/lib.v
/proj/old/rtl/alu.sv

All three. This is what a tree migration actually needs, and g is by far the most-used flag in practice — if you are rewriting a path or renaming an identifier across a file, you almost always want it.

4. A Number: Exactly One Occurrence

Instead of a flag letter, you can give a number. 2 means "replace the second match and nothing else":

sed '1s|/proj/old|/proj/new|2' filelists/paths.f

+incdir+/proj/old -y /proj/new/rtl -v /proj/old/lib.v
/proj/old/rtl/alu.sv

The -y path changed. The +incdir+ before it and the -v after it did not. And 3:

sed '1s|/proj/old|/proj/new|3' filelists/paths.f

+incdir+/proj/old -y /proj/old/rtl -v /proj/new/lib.v
/proj/old/rtl/alu.sv

Only the library file.

This is a precision instrument, and the honest advice is that you will reach for it rarely. Counting occurrences on a line is fragile: it depends on the exact spelling of everything before the part you care about, so the command breaks the moment someone adds another search path to the line. When you find yourself counting, it is usually a sign that the right answer is a more specific pattern — which is what Module 4 is for — or a capture group, which is Chapter 3.4.

Where it genuinely helps is a line with a known fixed structure: the second field of a report, the third column of a corner table, a timestamp that always appears twice.

5. A Number Too Large Changes Nothing

The number form has a failure mode worth meeting deliberately. Line 1 has three matches. Ask for the fourth:

sed '1s|/proj/old|/proj/new|4' filelists/paths.f

+incdir+/proj/old -y /proj/old/rtl -v /proj/old/lib.v
/proj/old/rtl/alu.sv

Nothing changed, and the exit status is 0.

This is the same false success Chapter 3.1 described, but it is harder to spot, because the command looks careful. 4 is a specific, deliberate-looking instruction, and there is no fourth match, so sed does nothing and reports success. If you had counted wrong — or if the line changed since you counted — you get silence instead of a complaint.

The defence is the same as before: run it with -n and the p flag first, and let empty output tell you the truth.

sed -n '1s|/proj/old|/proj/new|4p' filelists/paths.f

No output at all. Compare that with the version that works:

sed -n '1s|/proj/old|/proj/new|3p' filelists/paths.f

+incdir+/proj/old -y /proj/old/rtl -v /proj/new/lib.v

One line of output means one line matched and here is what it becomes. Empty output means your command does nothing. That distinction is invisible without -n and obvious with it.

6. Ng — Nth Onward, and Why It May Not Work

GNU sed accepts a number and g together: 2g means "from the second match onward". It is occasionally exactly what you want — leave the first occurrence alone, change all the rest.

It is not portable. On the BSD sed that macOS ships, it is a hard error:

sed '1s|/proj/old|/proj/new|2g' filelists/paths.f

sed: 1: "1s|/proj/old|/proj/new|2g
": more than one number or 'g' in substitute flags

Exit status 1. At least this failure is loud — it refuses to run rather than doing something unexpected, which puts it in a much better class than the GNU-only features that fail silently.

If you need "all but the first" and you need it to work everywhere, express it with a pattern instead of a flag. The general shape is to capture the part you want to keep and reinsert it, which Chapter 3.4 teaches. For the common real case — leaving an +incdir+ alone while migrating the rest of the line — the simplest portable answer is two commands, one scoped to the part you want to protect.

7. Seeing How Much Changed

With g in play, it becomes useful to know not just which lines changed but how many matches were replaced. sed will not tell you — Chapter 2.1 explained that it keeps no state between lines, so it cannot count.

Two outside answers. Count the matches before you start, with grep -o, which prints each match on its own line:

grep -o '/proj/old' filelists/paths.f | wc -l

       4

Four matches across the file: three on line 1, one on line 2. Now you know what a correct global substitution should consume.

Or look at the lines, with the p flag and -n:

sed -n 's|/proj/old|/proj/new|gp' filelists/paths.f

+incdir+/proj/new -y /proj/new/rtl -v /proj/new/lib.v
/proj/new/rtl/alu.sv

Two lines changed, and every /proj/old is gone from both. Between the match count and the changed lines, you have enough to say the migration is complete rather than hoping it is.

8. Think It Through

1. Predict the output of each of these on +incdir+/proj/old -y /proj/old/rtl -v /proj/old/lib.v.

• s|/proj/old|/proj/new| → only the +incdir+ path changes; the other two remain.
• s|/proj/old|/proj/new|g → all three change.
• s|/proj/old|/proj/new|2 → only the -y path changes.
• s|/proj/old|/proj/new|9 → nothing changes, exit 0.

If you got the last one wrong, that is the point of section 5 — it is the only one of the four that gives you no feedback.

2. Your tree migration script uses g and the build still fails on one missing path. Where would you look first?

Not at the flag. g handles multiple matches per line, so the remaining problem is almost certainly a line the pattern never matched — a path spelled slightly differently, perhaps /proj/old/ with a trailing slash, or /proj//old, or a relative path. Count first: grep -c '/proj/old' file before, then grep -c '/proj/old' file after should be zero. If it is not zero, grep -n shows you which lines the pattern missed.

3. Why is |3 more fragile than a more specific pattern?

Because it depends on everything to the left of the thing you want. Add one more -y path to the line and the third match becomes the fourth, so your command silently edits the wrong one — or, if the count now runs past the end, silently edits nothing. A pattern that describes the target (-v /proj/old) keeps working when the line changes shape, because it is anchored to meaning rather than to position.

4. You need to change every /proj/old except the one on the +incdir+ line. How, portably?

Not with Ng. Use an address to exclude that line and a global substitution on the rest — the negated address form from Chapter 4.4, /+incdir+/!s|/proj/old|/proj/new|g, which reads "on lines that do not contain +incdir+, replace every match". Addressing the lines is almost always cleaner than counting occurrences within them.

5. You run a global substitution and grep -c for the old string afterwards returns 0. Are you done?

You know the old string is gone, which is the easier half. You do not yet know that what replaced it is correct — a too-wide pattern can remove every trace of the old string while also rewriting things you never intended. Chapter 7.3 is about exactly that, and the diff from Chapter 3.1 section 7 is what shows it.

9. Summary

• The flag after the closing delimiter chooses which matches change. No flag means the first; g means all of them; a number N means only the Nth.
• g is the one you usually want for a path rewrite or an identifier rename, because such text often appears more than once on a line.
• A fixture where every line has one match cannot tell the flags apart. Test a flag on input that can distinguish it, or you will carry the wrong conclusion into a file that matters.
• A number is a precision tool and a fragile one — it depends on everything to its left. Prefer a more specific pattern, or an address.
• An out-of-range number changes nothing and exits 0. |4 on a three-match line was silent. -n plus the p flag turns that silence into visibly empty output.
• Ng is GNU-only and fails loudly on BSD sed with more than one number or 'g' in substitute flags, exit 1. g and a plain number are portable.
• sed cannot count matches for you. grep -o … | wc -l before, and sed -n 's/…/…/gp' after.

You can now control how many matches change. The next problem is one you have already met three times without it being explained: every command in this chapter used | instead of /, because the text was full of slashes. Chapter 3.3 — Delimiters, Escaping, and Special Characters is about the characters that fight back, and it includes the two that will corrupt your output without any error at all.