Filelist and Include-Path Migration

Part IV · Module 9 · Chapter 9.1 · Filelist and Include-Path Migration

1. Why This Is the Easy Case

A compile filelist looks like this:

cat -n filelists/top.f

     1	+define+SYNTHESIS
     2	+incdir+/proj/old/rtl
     3	+incdir+/proj/old/rtl/common
     4	-y /proj/old/lib
     5	/proj/old/rtl/alu.sv
     6	/proj/old/rtl/fifo.sv
     7	# /proj/old/rtl/disabled.sv
     8	./rtl/local.sv
     9	/proj/old/tb/env.sv

Three kinds of line: tool options starting with + or -, source paths, and a commented-out entry. Every line is independent, nothing refers to anything else, and the content is paths — not code.

That matters because the reasons sed struggles with source files (Chapter 9.3's subject: scope, macros, strings, comments that look like code) do not exist here. A filelist is exactly the shape Chapter 8.4's test describes: each change is decided by the current line alone.

So this is not sed being used as a compromise. It is the job sed is for.

2. The Procedure

   1. AUDIT    what paths are actually in there?
   2. COUNT    how many lines will change?
   3. REWRITE  two separate, anchored substitutions — dry run first
   4. APPLY    and verify nothing stale survived

Steps 1 to 3 write nothing.

3. Step 1 — Audit Before You Touch Anything

Never start from what you think the filelists contain. The registered audit pulls every include path out of every filelist:

sed -n 's|.*\(+incdir+[^[:space:]]*\).*|\1|p' filelists/*.f | sort -u

+incdir+/proj/old/rtl
+incdir+/proj/old/rtl/common
+incdir+/proj/old/tb

Three distinct include directories, and note the third is in the other filelist — the glob caught it. Reading that pattern: match anything, capture +incdir+ plus the non-whitespace that follows, match the rest, and replace the line with just the capture. sort -u collapses duplicates across files.

Now the path roots in use:

grep -oh '/proj/[a-z]*' filelists/*.f | sort | uniq -c

   9 /proj/old

Nine references, all under one root. If this had shown two roots you would have learned that the migration is not the single rewrite you assumed.

4. Step 2 — Count, and Why Basic Mode Matters Here

grep -c '/proj/old' filelists/*.f

filelists/tb.f:2
filelists/top.f:7

Nine lines total — the number to check against afterwards.

A detail that decides how you write everything below. The + in +incdir+ is an ordinary character in basic regular expressions:

sed -n '/+incdir+/p' filelists/top.f

+incdir+/proj/old/rtl
+incdir+/proj/old/rtl/common

Works as typed. Under -E the same pattern is an error, because + becomes a repetition operator with nothing to repeat:

sed -E -n '/+incdir+/p' filelists/top.f

sed: 1: "/+incdir+/p
": RE error: repetition-operator operand invalid

Exit 1. Chapter 4.1 measured this; here is where it pays off. For filelist work, stay in basic mode. The text is full of + and you need none of what -E offers.

5. Step 3 — Two Rewrites, Not One

The filelist has two different kinds of reference to the old tree, and they need different patterns.

The include options. These are mid-line — the path follows +incdir+ — so the pattern anchors to the option text rather than to the line start:

sed -n 's|+incdir+/proj/old/rtl|+incdir+/proj/new/rtl|gp' filelists/top.f

+incdir+/proj/new/rtl
+incdir+/proj/new/rtl/common

Both include lines, including the common subdirectory, because the pattern stops at rtl and whatever follows is carried along.

The source paths. These start at column one, so the pattern anchors there:

sed -n 's|^/proj/old/rtl/|/proj/new/rtl/|p' filelists/top.f

/proj/new/rtl/alu.sv
/proj/new/rtl/fifo.sv

Two source lines. Note what is absent from that output: the commented-out # /proj/old/rtl/disabled.sv on line 7, and the two +incdir+ lines. The ^ excluded all three.

Both use | as the delimiter because the text is paths — Chapter 3.3's rule, and the reason this chapter never has to escape a slash.

6. Why the Anchor Earns Its Place

Run the source-path rewrite without the ^ and compare the whole file. Anchored:

sed 's|^/proj/old/rtl/|/proj/new/rtl/|' filelists/top.f

+define+SYNTHESIS
+incdir+/proj/old/rtl
+incdir+/proj/old/rtl/common
-y /proj/old/lib
/proj/new/rtl/alu.sv
/proj/new/rtl/fifo.sv
# /proj/old/rtl/disabled.sv
./rtl/local.sv
/proj/old/tb/env.sv

Two lines changed, exactly the two source paths. Unanchored:

sed 's|/proj/old/rtl/|/proj/new/rtl/|' filelists/top.f

+define+SYNTHESIS
+incdir+/proj/old/rtl
+incdir+/proj/new/rtl/common
-y /proj/old/lib
/proj/new/rtl/alu.sv
/proj/new/rtl/fifo.sv
# /proj/new/rtl/disabled.sv
./rtl/local.sv
/proj/old/tb/env.sv

Four lines changed. Two of them should not have been:

• Line 3, +incdir+/proj/old/rtl/common, was rewritten by the source path command. That is the include rewrite's job, and now two commands both touch it — so if their replacement strings ever differ, the result depends on which ran last.
• Line 7, the commented-out entry, was rewritten too. It is disabled on purpose, probably with a note somewhere about why. Silently repointing it at a tree where the file may not exist means that when someone re-enables it, it fails for a reason unrelated to why it was disabled.

Both exit 0. Nothing reports either.

7. Step 4 — Apply, Then Prove It

The dry runs above are Chapter 7.2's procedure. To apply, add -i — which is spelled -i on GNU sed and -i '' on BSD/macOS, and Chapter 7.1 covers why they are not interchangeable. Commit first; Chapter 7.2 measured why -i.bak is the weaker option.

Then verify, in the order that catches the most:

   grep -rn '/proj/old' filelists/        nothing should match
   git diff --stat                        9 lines, matching step 2's count
   git status --short                     only the filelists you expected

The first is the one people skip and the one that catches a missed spelling — a filelist with /proj/old//rtl, a relative path, or a symlinked alias will survive both rewrites and show up here.

One more check worth running on a real migration, because a filelist is a list of files that must exist:

   grep -v '^[+-]' filelists/top.f | grep -v '^#' | grep . | while read -r f; do
       [ -e "$f" ] || echo "MISSING: $f"
   done

A filelist that is textually correct and points at files that are not there is the most common outcome of a migration done without this step.

8. Where This Stops Working

Everything above relies on the filelist being flat text with no semantics. Two steps away, that stops being true.

• A filelist referencing another filelist with -f other.f is a graph, not a list, and a rewrite has to follow it.
• A filelist built by a script rather than checked in is a generated artifact — editing it is editing the output, and the fix belongs in the generator.
• The moment you move from the filelist to the RTL it names, you are editing a language with scope, macros, strings and comments, and none of this chapter's assumptions hold.

That last one is the rest of this module. Chapter 9.2 does the identifier rename properly, and Chapter 9.3 is about the renames that should not be done with sed at all.

9. Think It Through

1. Why two rewrites rather than one unanchored one?

Because the filelist has two kinds of reference and they need different patterns. Section 6 measured the single unanchored version also rewriting an +incdir+ line and a commented-out entry — four lines changed where two were correct, exit 0, no warning. Two anchored commands each state exactly what they own.

2. Why does this chapter never use -E?

Because the text is full of +, which is literal in basic mode and a repetition operator under -E — section 4 measured /+incdir+/ failing with repetition-operator operand invalid, exit 1. Nothing in a filelist rewrite needs alternation or grouping, so there is no reason to pay that cost.

3. Your rewrite runs cleanly and the build still cannot find a header. Where do you look?

At the include paths you did not audit. Section 3's audit found three distinct +incdir+ directories including one in a different filelist; a rewrite applied to top.f alone would leave tb.f pointing at the old tree. Run grep -rn '/proj/old' filelists/ over the whole directory, not just the file you edited.

4. The migration is textually complete and the compile fails on a missing file. What did the procedure miss?

Existence. Section 7's last check is the one that catches it: a path can be rewritten perfectly and still point nowhere, because the new tree does not have that file — it was renamed, or it was never copied. sed guarantees the text, not the filesystem.

5. A filelist contains /proj/old//rtl/alu.sv with a doubled slash. Does your rewrite catch it?

No. s|^/proj/old/rtl/|...| requires exactly one slash between old and rtl. The doubled-slash line survives both rewrites and is found only by the verification grep in section 7 — which is precisely why that step exists rather than being optional. Filesystems treat // as /, so the line works today and is invisible to a test that only compiles.

6. When is editing a filelist with sed the wrong call?

When it is generated. If a script writes top.f on every build, your edit is overwritten the next time anyone runs it, and the real change belongs in the generator. Check whether the file is in version control and whether anything writes to it before editing — that is a thirty-second check that saves a confusing afternoon.

10. Summary

• A filelist is the ideal sed target: flat, line-oriented, paths rather than code, every change decided by its own line.
• Audit first. sed -n 's|.*\(+incdir+[^[:space:]]*\).*|\1|p' filelists/*.f | sort -u found three distinct include paths, one of them in a filelist you might not have thought to edit.
• Count first too — nine lines across two files, which is the number to check git diff --stat against.
• Stay in basic mode. + is literal in BRE; under -E the same pattern is repetition-operator operand invalid, exit 1.
• Use two separate rewrites: one anchored to +incdir+ for options, one anchored to ^ for source paths.
• The unanchored single rewrite changed four lines instead of two — including an +incdir+ line that belongs to the other command and a commented-out source entry. Exit 0, no warning.
• A trailing slash is load-bearing. /proj/old/rtl did not match /proj/old/rtl/, which is why the two patterns are spelled differently on purpose.
• Use | as the delimiter throughout, so no path slash ever needs escaping.
• Verify three ways: nothing stale with grep -rn, the line count against the audit, and that every path in the list actually exists.
• Check whether the filelist is generated before editing it.

A filelist is paths. The files it names are a hardware description language, where the same text can be a signal, a comment, a string or a macro, and none of those mean the same thing. Chapter 9.2 — Signal, Module, and Define Renaming does the identifier rename across real RTL, with the word boundaries Chapter 7.3 introduced and a tree-wide procedure that keeps them honest.