Your Own Sets: Ranges and Negation

Part II · Module 19 · Chapter 19.5 · Your Own Sets: Ranges and Negation

1. A Set of Characters

Square brackets list the characters you will accept, and the whole set matches one of them:

import re

print("one of a, b or c :", re.findall(r"[abc]", "a-b-x-c"))
print("a range          :", re.findall(r"[a-f]", "deadbeefxyz"))
print("several ranges   :", re.findall(r"[A-Za-z]", "aB1_c"))
print("ranges and literals:", re.findall(r"[A-Fa-f0-9]", "deadBEEF12xyz"))

one of a, b or c : ['a', 'b', 'c']
a range          : ['d', 'e', 'a', 'd', 'b', 'e', 'e', 'f']
several ranges   : ['a', 'B', 'c']
ranges and literals: ['d', 'e', 'a', 'd', 'B', 'E', 'E', 'F', '1', '2']

A hyphen between two characters means a range by character code: a-f is the six letters from a to f, 0-9 the ten digits, A-Z the uppercase letters.

Several ranges can sit in one set, which is how a hexadecimal digit is spelled: [0-9A-Fa-f].

2. The Identifier Class

The canonical example, and the one you will write most:

import re

for subject in ["reg_q", "u_core", "2bad", "has-hyphen", "ok_1"]:
    m = re.search(r"[A-Za-z_][A-Za-z0-9_]*", subject)
    print(f"{subject:<12} -> {m.group() if m else '(no match)'}")

reg_q        -> reg_q
u_core       -> u_core
2bad         -> bad
has-hyphen   -> has
ok_1         -> ok_1

[A-Za-z_] then [A-Za-z0-9_]* is the standard identifier shape: a letter or underscore first, then any number of letters, digits or underscores. That is exactly the rule SystemVerilog, Python and C all use for a name.

Two results worth reading. 2bad matched bad — the pattern found a valid identifier starting later in the string, because search looks anywhere. And has-hyphen stopped at the hyphen, which is correct for an identifier and wrong if your tool allows hyphens in test names — in which case the set becomes [A-Za-z0-9_-].

3. Inside Brackets, Most Metacharacters Are Ordinary

A genuine simplification after Chapter 19.3's escaping discipline:

import re

print("literal dot     :", re.findall(r"[.]", "a.b.c"))
print("dot and comma   :", re.findall(r"[.,]", "a.b,c"))
print("literal parens  :", re.findall(r"[()]", "f(x)"))
print("literal plus    :", re.findall(r"[+]", "+incdir+"))
print("literal star    :", re.findall(r"[*]", "*** FAILED ***"))

literal dot     : ['.', '.']
dot and comma   : ['.', ',']
literal parens  : ['(', ')']
literal plus    : ['+', '+']
literal star    : ['*', '*', '*', '*', '*', '*']

No backslashes. Inside a set, . ( ) + * ? { } | all lose their special meaning and stand for themselves, because a set is a list of characters and those constructs have nothing to list.

Three characters still need positional care, and knowing the rules matters for reading other people's patterns as much as writing your own:

import re

print("^ first means NOT   :", re.findall(r"[^ab]", "abc"))
print("^ elsewhere literal :", re.findall(r"[a^]", "a^b"))
print("- last is literal   :", re.findall(r"[a-]", "a-b"))
print("- first is literal  :", re.findall(r"[-a]", "a-b"))
print("- between is a range:", re.findall(r"[a-c]", "abcd"))
print("] first is literal  :", re.findall(r"[]]", "a]b"))
print("] escaped elsewhere :", re.findall(r"[\]]", "a]b"))

^ first means NOT   : ['c']
^ elsewhere literal : ['a', '^']
- last is literal   : ['a', '-']
- first is literal  : ['a', '-']
- between is a range: ['a', 'b', 'c']
] first is literal  : [']']
] escaped elsewhere : [']']

All three follow the same idea — a character is special only where being special would make sense:

• ^ negates only as the first character; anywhere else it is a literal caret.
• - is a range only between two characters; first or last there is nothing to range from, so it is literal.
• ] is literal as the first character, because an empty set would be meaningless; elsewhere it closes the set and needs \].

Note the - first is literal row: findall returned ['a', '-'] rather than ['-', 'a'], because results come back in the order they appear in the subject string, not in the order they were listed in the pattern. The set is a set; the output is a scan.

Writing [a-] instead of [a\-] is legal and common in inherited code, which is why the positional rules are worth knowing for reading as much as for writing.

4. Negation: Describe the Delimiter, Not the Field

The construct that changes how you parse. [^...] matches any character not in the set:

import re

LINE = "smoke,1234,PASS,182.4"

print("not a comma, one char :", re.search(r"[^,]", LINE).group())
print("a run of them         :", re.search(r"[^,]+", LINE).group())
print("all the fields        :", re.findall(r"[^,]+", LINE))

not a comma, one char : s
a run of them         : smoke
all the fields        : ['smoke', '1234', 'PASS', '182.4']

Look at what that pattern did not have to know. It does not describe a test name, a seed, a verdict or a runtime — four fields with four different shapes. It describes the comma, and takes everything that is not one.

That is the inversion, and it is worth stating plainly:

When a field has no fixed shape but a known delimiter, describe the delimiter. The field is whatever is not it.

It applies directly to tool output where a value is arbitrary text:

import re

LINE = "UVM_ERROR drv.sv(88) @ 1420: uvm_test_top.env.agt.drv [DRV] bad handshake"

print("file, up to the paren  :", re.search(r"[^ ]+(?=\()", LINE) is not None)
print("everything after colon :", re.search(r": ([^\n]+)", LINE) is None)
print("message, by delimiter  :", LINE.split(": ", 1)[1])
print("instance path          :", re.search(r"[A-Za-z0-9_.]+", "uvm_test_top.env.agt.drv").group())
print("bracketed tag          :", re.search(r"\[([^\]]+)\]", LINE).group(1))

file, up to the paren  : True
everything after colon : False
message, by delimiter  : uvm_test_top.env.agt.drv [DRV] bad handshake
instance path          : uvm_test_top.env.agt.drv
bracketed tag          : DRV

The last two are the pattern worth memorising. [^\]]+ inside brackets means "everything up to the closing bracket", which extracts a tag of any length and content. The instance path uses the opposite approach — an inclusive class listing the characters a hierarchical name may contain — and both are reasonable; the inclusive version rejects malformed input and the negated one is shorter.

5. Which Way Round to Describe It

A decision rule, because both forms are always available:

import re

GOOD = "slack=-0.142"
AWKWARD = "message=error: value=3 not expected"

print("inclusive, value is a number:", re.search(r"=(-?[0-9.]+)", GOOD).group(1))
print("negated, value is anything  :", re.search(r"=([^=]+)", GOOD).group(1))
print()
print("the awkward case, negated   :", re.search(r"=([^=]+)", AWKWARD).group(1))
print("the awkward case, split once:", AWKWARD.split("=", 1)[1])

inclusive, value is a number: -0.142
negated, value is anything  : -0.142

the awkward case, negated   : error: value
the awkward case, split once: error: value=3 not expected

The negated version got it wrong on the awkward line: [^=]+ stopped at the second =, returning error: value instead of the whole message, because a negated class cannot know that the second = belongs to the value rather than separating a new field.

split("=", 1) got it right, and the reason is the 1: it splits once and hands back everything after the first delimiter untouched. The negated class has no equivalent of "stop splitting now".

The reading to take: when a delimiter can appear inside a value, a negated class is the wrong tool and a split with a count is the right one. Chapter 19.19 returns to this with the full comparison.

6. ASCII-Only Sets

Chapter 19.4 found that \d matches Unicode digits. A custom set is the explicit alternative:

import re

MIXED = "٣7a"

print("\\d  shorthand :", re.findall(r"\d", MIXED))
print("[0-9] explicit:", re.findall(r"[0-9]", MIXED))
print("\\w  shorthand :", re.findall(r"\w", MIXED))
print("[A-Za-z0-9_]  :", re.findall(r"[A-Za-z0-9_]", MIXED))

\d  shorthand : ['٣', '7']
[0-9] explicit: ['7']
\w  shorthand : ['٣', '7', 'a']
[A-Za-z0-9_]  : ['7', 'a']

Use the explicit form when a downstream consumer requires ASCII — a filename, a tool argument, a CSV field another program reads. Use the shorthand when you are parsing text you control and brevity helps.

Neither is more correct; they describe different sets and the choice should be deliberate rather than a matter of which one you remembered.

7. A Negated Class Is Not Line-Bounded

The hazard, and it only appears when a log is read whole rather than line by line:

import re

ONE_LINE = "smoke,1234"
WHOLE_FILE = "smoke,1234\ncdc,5678\n"

print("on one line     :", re.findall(r"[^,]+", ONE_LINE))
print("on a whole file :", re.findall(r"[^,]+", WHOLE_FILE))
print()
print("line-bounded    :", re.findall(r"[^,\n]+", WHOLE_FILE))

on one line     : ['smoke', '1234']
on a whole file : ['smoke', '1234\ncdc', '5678\n']

That output is four lines and the program prints five — the line-bounded result is ['smoke', '1234', 'cdc', '5678'], which is the correct parse.

[^,] matched the newline, because a newline is not a comma. So 1234\ncdc came back as one field spanning two records, and a script building a results table from it gets three fields instead of four with one of them containing an embedded newline.

Chapter 19.4 noted that . excludes the newline by default; a negated class has no such exemption — it excludes exactly what you listed and nothing else. Adding \n to the set is the fix:

import re

WHOLE_FILE = "smoke,1234\ncdc,5678\n"
print("fields:", re.findall(r"[^,\n]+", WHOLE_FILE))

fields: ['smoke', '1234', 'cdc', '5678']

8. Predict, Run, Observe

import re

LINE = "u_core/reg_q[3] slack=-0.142ns"

print("A:", re.findall(r"[a-z_]+", LINE))
print("B:", re.search(r"[^/]+", LINE).group())
print("C:", re.search(r"\[([^\]]+)\]", LINE).group(1))
print("D:", re.findall(r"[0-9.]+", LINE))
print("E:", re.findall(r"[-0-9.]+", LINE))
print("F:", re.search(r"slack=([-0-9.]+)", LINE).group(1))
print("G:", re.findall(r"[^a-z]", "ab1c"))

A: ['u_core', 'reg_q', 'slack', 'ns']
B: u_core
C: 3
D: ['3', '0.142']
E: ['3', '-0.142']
F: -0.142
G: ['1']

A split on every non-lowercase character, so u_core and reg_q survived — underscores are in the set — while the digits, brackets and the sign did not.

C is the bracketed-tag pattern from Section 4, extracting 3 without describing what is inside.

D and E together are the most expensive mistake in the chapter. [0-9.]+ found 0.142 and dropped the minus sign, turning a negative slack into a positive one — a timing violation silently reported as passing. [-0-9.]+ keeps the sign, with the hyphen first so it is literal.

But E is still not usable, and this is the part worth pausing on. It returned two matches — the 3 from reg_q[3] came first, because findall scans from the left and a bracket index is made of the same characters as a slack value. A pattern describing only a shape cannot tell the two apart.

F is the fix: anchoring the value to the literal text that precedes it. slack=([-0-9.]+) says which number, not merely what a number looks like — and the parentheses that make it retrievable are Chapter 19.8's subject.

That progression from D to F is the module in miniature: a shape that is nearly right, a shape that is right but ambiguous, and a shape pinned to its context.

9. Exercises

Exercise 1 — ranges. Write classes for a hex digit, an uppercase letter and a digit-or-underscore. Test each with findall.

Exercise 2 — the identifier. Build [A-Za-z_][A-Za-z0-9_]* and run it on five names, including one starting with a digit and one containing a hyphen.

Exercise 3 — no escapes needed. Match a literal dot, parenthesis and plus inside a class without backslashes. Then find the three characters that still need care.

Exercise 4 — positional rules. Show - behaving as a range and as a literal in two patterns that differ only in where it sits.

Exercise 5 — describe the delimiter. Parse a four-field comma-separated line with one negated class. Confirm you described nothing about the fields.

Exercise 6 — the bracketed tag. Extract the contents of [...] from a UVM line using [^\]]+.

Exercise 7 — the newline leak. Run [^,]+ over a two-line string and show a field spanning both. Fix it and confirm.

Exercise 8 — the dropped sign. Extract a negative slack with [0-9.]+ and show the sign is lost. Fix it two ways and say which reads better.

10. Summary

• [...] matches one character from the listed set, and a hyphen between two characters means a range.
• Several ranges can sit in one set — [0-9A-Fa-f] is a hex digit.
• [A-Za-z_][A-Za-z0-9_]* is the identifier shape used by SystemVerilog, Python and C alike.
• Inside brackets most metacharacters are ordinary — [.], [()], [+] need no backslashes.
• Three need positional care, all by the same logic — a character is special only where being special would make sense: ^ negates only when first, - is a range only between two characters, and ] is literal when first.
• [^...] inverts the parsing problem: describe the delimiter, and the field is whatever is not it.
• Use an inclusive class when the field has a shape and a negated one when it does not; when the delimiter can appear inside the value, use split with a count instead.
• A shape alone can be ambiguous — [-0-9.]+ matches a bracket index as readily as a slack value, so pin the value to the text that precedes it.
• [0-9] and [A-Za-z0-9_] are the ASCII-only forms of \d and \w.
• A negated class is not line-bounded — it excludes exactly what you listed, so [^,] matches a newline.
• Add \n to every negated class unless crossing lines is intended; the bug is invisible until a file is read whole.

The next chapter is the other half of every pattern so far: how many.