Basic and Extended Regular Expressions

Part II · Module 4 · Chapter 4.1 · Basic and Extended Regular Expressions

1. Two Dialects, One Command

Every pattern you have written so far was a regular expression — a small language for describing text. sed implements two versions of it:

• Basic regular expressions (BRE) — the default. No option needed.
• Extended regular expressions (ERE) — selected with -E.

They share almost everything. The difference is a short list of characters that are operators in extended mode and ordinary text in basic mode:

   (    )    |    +    ?

   basic mode  ->  just these characters, matched literally
   -E          ->  group, group, alternation, one-or-more, optional

To use them as operators in basic mode you put a backslash in front. To use them as text in extended mode you put a backslash in front. The backslash means "the other meaning" in both directions, which is the source of most confusion — and the reason it pays to know which mode you are in before you start typing.

Chapter 2.3 introduced -E as an option; this chapter is what it actually changes.

2. Start With the Case Where Basic Mode Wins

The most common sed target in a hardware flow is a compile filelist, and the most common thing to match in one is a +incdir+ or +define+ option.

cat -n filelists/top.f

     1	+define+DEBUG
     2	+incdir+/proj/old/rtl
     3	
     4	./rtl/alu.sv
     5	./rtl/fifo.sv   
     6	# a commented-out file
     7	/proj/old/tb/env.sv
     8	
     9	#another comment
    10	./rtl/top.sv

In basic mode, you write what you mean:

sed -n '/+incdir+/p' filelists/top.f

+incdir+/proj/old/rtl

The pluses are just pluses. Now the same pattern with -E:

sed -E -n '/+incdir+/p' filelists/top.f

sed: 1: "/+incdir+/p
": RE error: repetition-operator operand invalid

Exit status 1. Under -E, + means "one or more of the thing before me" — and there is nothing before the first +, so the pattern is nonsense. sed says so and refuses to run.

To get the same job done under -E, escape both pluses:

sed -E -n '/\+incdir\+/p' filelists/top.f

+incdir+/proj/old/rtl

Correct, and noisier than the basic version. This is worth stating plainly because the usual advice is "always use -E, it is cleaner" — for filelist work, it is not. If your text is full of + signs and parentheses, basic mode is the convenient one.

The same applies to parentheses. A Verilog module header contains one:

sed -n '/(/p' rtl/top.sv

module top (

Plain, unescaped, works. Under -E the same thing is an error, because ( opens a group that is never closed:

sed -E -n '/(/p' rtl/top.sv

sed: 1: "/(/p
": RE error: parentheses not balanced

Exit status 1 again. Note the pattern of these failures: using an operator character as text under -E gives a loud error. That is the good case, and it is why -E is a forgiving dialect to learn in — your mistakes stop the command.

3. Now the Case Where -E Wins

Extended mode earns its place when you want alternation — "match this or that". Here is a module header:

cat -n rtl/top.sv

     1	module top (
     2	  input  logic clk,
     3	  input  logic rst_n,
     4	  output logic [31:0] data_o,
     5	  inout  wire  sda
     6	);
     7	  logic [7:0] count;
     8	  wire        ready;
     9	endmodule

To list the input and output ports — but not inout, logic or wire — you want "a line that is some indentation, then either input or output":

sed -E -n '/^[[:space:]]*(input|output)/p' rtl/top.sv

  input  logic clk,
  input  logic rst_n,
  output logic [31:0] data_o,

Three lines, and inout wire sda correctly excluded even though inout starts with the same four letters as input — because the pattern asks for input followed by whatever, and inout has an o where input has a p.

(input|output) reads naturally: parentheses group, the bar separates alternatives. [[:space:]] means "a whitespace character" and * means "any number of them, including none" — both are Chapter 4.2's subject and are used here because an indented port list needs them.

4. The Same Pattern in Basic Mode — and a Trap

In basic mode, the grouping and alternation operators need backslashes. The textbook spelling is:

sed -n '/^[[:space:]]*\(input\|output\)/p' rtl/top.sv

Nothing. Exit status 0, no diagnostic, no output.

The reason is that \| for alternation is not part of POSIX basic regular expressions. It is a GNU extension. GNU sed implements it; the BSD sed that macOS ships does not — so the pattern was read as a group containing the literal text input|output, nothing on any line matched that, and sed reported a successful search with no results.

5. Writing Alternation Portably

Three options, in order of preference.

Use -E. It is specified by POSIX, and both GNU and BSD sed implement it:

sed -E -n '/^[[:space:]]*(input|output)/p' rtl/top.sv

  input  logic clk,
  input  logic rst_n,
  output logic [31:0] data_o,

Or use two commands. Chapter 2.3 showed that ; separates commands, and two addresses with their own p need no alternation at all:

sed -n '/^[[:space:]]*input/p;/^[[:space:]]*output/p' rtl/top.sv

  input  logic clk,
  input  logic rst_n,
  output logic [31:0] data_o,

Slightly longer, completely portable, and arguably easier to read when the alternatives are long. One caveat worth knowing from Chapter 2.2: a line matching both addresses would be printed twice — not possible here, but it is why this form is not a universal substitute.

Or use a character class when the alternatives differ by a single character, which [ ] handles in both dialects. That is Chapter 4.2.

What to avoid is \|, unless you know the script will only ever run on GNU sed — and "only ever" is a strong claim for a file that lives in a repository.

6. The Whole Difference, Measured

Every row below was run on the sed these examples were verified against.

Two things to take from the table.

The mapping is symmetric but the failure modes are not. Treating an operator as text under -E gives you an error and no output file. Treating text as an operator in basic mode — specifically \| — gives you an empty result and a success status. One of those you find in seconds; the other you find in review, if at all.

Neither dialect is the right default. -E for anything with alternation or grouping; basic mode for filelists and anything thick with + and (. Deciding per command is normal, and writing the mode you intended into the command is the point of -E being explicit.

The confirming pair, both already shown above: /+incdir+/ needs no escapes in basic mode and two under -E, while (input|output) needs two escapes in basic mode (and a GNU extension) and none under -E. The same file, the same two patterns, opposite answers.

7. A Small Practical Note on ?

The question mark is the least-used of the five and the easiest to demonstrate. In basic mode it is a literal character:

printf 'rst_n\nrst_n?\n' | sed -n '/rst_n?/p'

rst_n?

Only the line that actually contains a question mark. Under -E it makes the preceding character optional:

printf 'rst_n\nrst_\n' | sed -E -n '/rst_n?/p'

rst_n
rst_

Both lines, because the n is now optional. That is occasionally useful for matching a name that may or may not carry a suffix — rst and rst_n, clk and clk_i — and it is a good illustration of why you cannot read a pattern without knowing its dialect. The same six characters mean two different things.

8. Think It Through

1. Why does sed -E 's/+define+DEBUG/+define+RELEASE/' top.f fail, and what are two fixes?

Under -E the leading + is a repetition operator with nothing to repeat, so sed reports repetition-operator operand invalid and exits 1. Fix one: drop -E, since basic mode treats + literally and the command needs no other extended feature. Fix two: escape the pluses in the pattern — s/\+define\+DEBUG/+define+RELEASE/. Note the replacement side needs no escaping either way, because a replacement is not a pattern (Chapter 3.3 section 5).

2. A script works on your Linux machine and returns nothing on a colleague's Mac. The pattern is /^\(read_sdc\|read_upf\)/. What is wrong, and what is the smallest portable fix?

\| is a GNU extension; BSD sed treats it as a literal bar, matches nothing, and exits 0. The smallest fix is -E with plain parentheses and bar: sed -E -n '/^(read_sdc|read_upf)/p'. If you cannot change the options, two addresses separated by ; is the alternative.

3. Which of these need -E, and which are better without it?

• /+incdir+/ — better without. Basic mode matches it as written.
• /(ss_0p72v|ss_0p88v)/ — needs -E, or escaped parentheses plus a GNU extension.
• /module top (/ — better without, so the parenthesis stays literal.
• /axi_(aw|ar)[a-z]+/ — needs -E twice over: for the alternation and for the +.

The general test: count the operator characters you want as text against the ones you want as operators, and pick the dialect that needs fewer backslashes.

4. Is ? in /rst_n?/ a wildcard?

Only under -E, where it makes the n optional. In basic mode it is a literal question mark, so the pattern matches only text that really contains one. This is a common slip for people arriving from shell globbing, where ? means "any one character" — it never means that in a sed regex, in either dialect.

5. Your pattern returns no matches. How do you tell "no such text in the file" from "my pattern is in the wrong dialect"?

Simplify until it matches. Take the plainest possible substring — one with no operator characters at all — and confirm sed -n '/that/p' finds the line you expect. Then add the pattern back one piece at a time. The moment the output goes empty, the piece you just added is either wrong or in the wrong dialect, and trying it with and without -E distinguishes those two in one step.

9. Summary

• sed has two regex dialects: basic by default, extended with -E.
• Five characters differ — ( ) | + ?. They are literal text in basic mode and operators under -E; a backslash flips the meaning in either direction.
• Basic mode is better for filelists. /+incdir+/ works as written; under -E it is repetition-operator operand invalid, exit 1.
• -E is better for alternation. (input|output) reads naturally and is portable.
• \| is a GNU extension. On BSD sed it matches nothing and exits 0 — the same script returns three lines on Linux and none on a Mac, with no error either way.
• Failures are asymmetric. Misusing an operator under -E errors out loudly; misusing \| in basic mode returns an empty, successful-looking result.
• Portable alternation: -E, or two addresses separated by ;, or a character class when the alternatives differ by one character.
• A pattern cannot be read without its dialect. /rst_n?/ matches a literal question mark in one mode and an optional n in the other.

You can now choose a dialect and spell operators correctly in it. What the patterns still cannot do is describe kinds of character — an identifier, a number, whitespace, "anything but a slash" — or say how many times something repeats. Chapter 4.2 — Character Classes and Quantifiers supplies both, and it is where patterns stop being literal text and start being descriptions.