Flags: IGNORECASE, MULTILINE, DOTALL

Part II · Module 19 · Chapter 19.14 · Flags: IGNORECASE, MULTILINE, DOTALL

1. Where a Flag Goes

Every function from Chapter 19.11 takes an optional third argument:

import re

TEXT = "Warning: timing derate not set for corner ss\n"

print("without:", re.search(r"warning", TEXT))
print("with   :", re.search(r"warning", TEXT, re.IGNORECASE).group(0))

without: None
with   : Warning

re.IGNORECASE is a constant that modifies how the pattern is interpreted. There are short aliases — re.I, re.M, re.S, re.X — and they print as their long names, which is a small mercy when you are reading a traceback:

import re

print(re.I, re.M, re.S, re.X)
print(re.IGNORECASE | re.MULTILINE)

re.IGNORECASE re.MULTILINE re.DOTALL re.VERBOSE
re.IGNORECASE|re.MULTILINE

Flags combine with |, the bitwise-or from Chapter 3.7. That is not the regex alternation | — same character, entirely different context — and combining flags is the normal case rather than the exotic one, as Section 3 is about to show.

2. IGNORECASE: The Capitalisation a Tool Changed

Tool vendors change the case of their severity labels between releases, and some tools are inconsistent within one report:

import re

TEXT = """Warning: timing derate not set for corner ss
Info: 1842 endpoints
WARNING: library mismatch on cell BUFX4
warning: scan chain unconstrained
"""

print("case-sensitive:", re.findall(r"warning", TEXT))
print("ignorecase    :", re.findall(r"warning", TEXT, re.IGNORECASE))

case-sensitive: ['warning']
ignorecase    : ['Warning', 'WARNING', 'warning']

Three spellings of one word in four lines of one report — which is not a contrived example; it is what a report looks like when a message comes from two different stages of a tool.

The case-sensitive version found one and reported a number. That is the failure shape to fear: a warning count of 1 where the truth is 3 is a plausible-looking result that nobody queries.

re.IGNORECASE is the honest fix here. The alternative, a character class per letter — [Ww][Aa][Rr][Nn][Ii][Nn][Gg] — is correct, unreadable and easy to get wrong by one letter. Chapter 19.5's classes are the right tool for a field with two spellings; a flag is the right tool for a whole pattern whose case is simply not information.

3. MULTILINE: Anchors, Per Line

Chapter 19.7 established that ^ means the start of the string and $ the end, which made ^UVM_ERROR useless on a multi-line subject. re.MULTILINE changes exactly that:

import re

TEXT = """Warning: timing derate not set for corner ss
Info: 1842 endpoints
WARNING: library mismatch on cell BUFX4
warning: scan chain unconstrained
"""

print("no flag  :", re.findall(r"^warning", TEXT))
print("MULTILINE:", re.findall(r"^warning", TEXT, re.MULTILINE))

no flag  : []
MULTILINE: ['warning']

With re.MULTILINE, ^ matches at the start of the string and immediately after every newline, and $ matches at the end and immediately before every newline. So the fourth line's lowercase warning is now reachable, where before only position 0 was.

And here is the pair of runs that justifies the canonical annotation — the same pattern, the same report, one flag at a time:

import re

TEXT = """Warning: timing derate not set for corner ss
Info: 1842 endpoints
WARNING: library mismatch on cell BUFX4
warning: scan chain unconstrained
"""

print("IGNORECASE only:", re.findall(r"^warning", TEXT, re.IGNORECASE))
print("MULTILINE only :", re.findall(r"^warning", TEXT, re.MULTILINE))
print("both           :", re.findall(r"^warning", TEXT, re.IGNORECASE | re.MULTILINE))

IGNORECASE only: ['Warning']
MULTILINE only : ['warning']
both           : ['Warning', 'WARNING', 'warning']

Read the first two lines together, because this is the whole point of the section. Each single-flag run returned exactly one result, and each one looks like it worked. IGNORECASE alone found the Warning at position 0 — the only place an unanchored-by-line ^ can match. MULTILINE alone found the one line whose spelling happened to be lowercase. Neither returned an empty list, neither raised, and both are wrong.

That is why the canonical annotation has both flags in it:

re.search(r"^warning", text, re.IGNORECASE | re.MULTILINE)

A per-line, case-insensitive anchor needs both, and the two single-flag versions are precisely the kind of bug that survives a code review.

$ gains the same per-line behaviour:

import re

TEXT = "Warning: derate not set\nInfo: 1842 endpoints\nwarning: unconstrained\n"

print("no flag  :", re.findall(r"endpoints$", TEXT))
print("MULTILINE:", re.findall(r"endpoints$", TEXT, re.MULTILINE))

no flag  : []
MULTILINE: ['endpoints']

Which makes whole-line extraction over a report a one-liner:

import re

TEXT = """Startpoint: u_core/u_alu/acc_reg[7]
Endpoint: u_core/u_fifo/wptr_reg[3]
Startpoint: u_core/u_ctl/state_reg[1]
Endpoint: u_core/u_ctl/done_reg
"""

print(re.findall(r"^Startpoint: (\S+)$", TEXT, re.MULTILINE))

['u_core/u_alu/acc_reg[7]', 'u_core/u_ctl/state_reg[1]']

Two startpoints, anchored at both ends of their own lines, from one call over the whole report.

4. \A and \Z Are Not Affected

Chapter 19.7 introduced \A and \Z as the strict string anchors. re.MULTILINE does not touch them, which is what they are for:

import re

TEXT = "Warning: one\nwarning: two\n"

print("^ with M :", len(re.findall(r"^warning", TEXT, re.I | re.M)))
print("\\A with M:", len(re.findall(r"\Awarning", TEXT, re.I | re.M)))

^ with M : 2
\A with M: 1

So \A and \Z mean the start and end of the subject, whatever the flags say, and ^ and $ mean the start and end of a line, if MULTILINE is on. If a pattern must apply to the whole subject regardless of how someone later edits the flags, use the strict pair.

5. DOTALL: Letting the Dot Cross a Newline

Chapter 19.13 noted that . matches anything except a newline, and that this is the only thing that keeps a greedy .* inside one line. Some fields genuinely span lines — a timing path's startpoint and endpoint, with the flip-flop description between them:

import re

TEXT = """Startpoint: u_core/u_alu/acc_reg[7]
  (rising edge-triggered flip-flop)
Endpoint: u_core/u_fifo/wptr_reg[3]
  slack (VIOLATED)          -0.142
"""

print("no flag:", re.search(r"Startpoint:(.*?)Endpoint:", TEXT))
print("DOTALL :", repr(re.search(r"Startpoint:(.*?)Endpoint:", TEXT, re.DOTALL).group(1)))

no flag: None
DOTALL : ' u_core/u_alu/acc_reg[7]\n  (rising edge-triggered flip-flop)\n'

Without the flag the pattern cannot get from Startpoint: to Endpoint:, because they are on different lines and . will not cross the newline between them. re.DOTALL makes . match every character including \n, and the group now contains two lines of text — newline included, which is why repr is the right way to print it.

That is the canonical annotation:

re.search(r"Startpoint:(.*?)Endpoint:", text, re.DOTALL)

And the ? in it is load-bearing. With DOTALL on, there is nothing left to contain a greedy quantifier at all:

import re

TEXT = """Startpoint: u_core/u_alu/acc_reg[7]
Endpoint: u_core/u_fifo/wptr_reg[3]
Startpoint: u_core/u_ctl/state_reg[1]
Endpoint: u_core/u_ctl/done_reg
"""

greedy = re.search(r"Startpoint:(.*)Endpoint:", TEXT, re.DOTALL).group(1)
lazy = re.search(r"Startpoint:(.*?)Endpoint:", TEXT, re.DOTALL).group(1)

print("greedy:", repr(greedy))
print("lazy  :", repr(lazy))

greedy: ' u_core/u_alu/acc_reg[7]\nEndpoint: u_core/u_fifo/wptr_reg[3]\nStartpoint: u_core/u_ctl/state_reg[1]\n'
lazy  : ' u_core/u_alu/acc_reg[7]\n'

The greedy version ran past the first Endpoint:, past the second Startpoint:, and stopped at the last Endpoint: in the subject — Chapter 19.13 Section 1's defect, scaled from one line to the whole file. On a real report that is tens of thousands of lines in one group.

The lazy version is the first path and only the first path, and finditer gives you all of them:

import re

TEXT = """Startpoint: u_core/u_alu/acc_reg[7]
Endpoint: u_core/u_fifo/wptr_reg[3]
Startpoint: u_core/u_ctl/state_reg[1]
Endpoint: u_core/u_ctl/done_reg
"""

for m in re.finditer(r"Startpoint:(.*?)Endpoint:", TEXT, re.DOTALL):
    print(repr(m.group(1).strip()))

'u_core/u_alu/acc_reg[7]'
'u_core/u_ctl/state_reg[1]'

6. VERBOSE: The Fourth Flag Worth Knowing

Not in the chapter's title and worth the two minutes. Chapter 19.12's finished pattern is right at the length where a regex stops being readable:

import re

PATTERN = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"

print(re.search(PATTERN, "  slack (MET)   0.031").group(1))

0.031

re.VERBOSE makes whitespace in the pattern insignificant and enables # comments inside it, so the same pattern can be laid out with its own symbol audit attached:

import re

SLACK = r"""
    slack \s+                      # the word, then the tool's alignment gap
    \( (?: VIOLATED | MET ) \)     # the tag -- only these two carry a number
    \s+                            # the second alignment gap
    ( -? \d+ (?: \.\d+ )? )        # signed value, fractional part optional
"""

print(re.search(SLACK, "  slack (MET)   0.031", re.VERBOSE).group(1))
print(re.search(SLACK, "  slack (VIOLATED)   -1", re.VERBOSE).group(1))

0.031
-1

Same pattern, same results, and Chapter 19.12 Section 7's table now lives next to the symbols it describes — where it cannot drift out of date.

The trap is in the definition. Whitespace is insignificant, which means a space you meant literally is gone:

import re

print("unescaped:", re.search(r"a b", "ab", re.VERBOSE) is not None)
print("escaped  :", re.search(r"a\ b", "a b", re.VERBOSE) is not None)

unescaped: True
escaped  : True

The first line is the surprise: with VERBOSE, r"a b" matched "ab", because the space was discarded. A literal space in a verbose pattern must be written \ , or put inside a class as [ ], or avoided entirely in favour of \s+ — which is what Chapter 19.12 recommended for a different reason and is the simplest habit to carry.

7. How a Flag Goes Wrong

Two ways, and neither raises.

A flag widens a pattern you had made precise. Chapter 19.7 built \bERROR\b to stop NO_ERROR being counted. Add IGNORECASE and it starts counting English:

import re

TEXT = "an error occurred earlier; UVM_ERROR real failure here"

print("exact     :", re.findall(r"\bERROR\b", TEXT))
print("ignorecase:", re.findall(r"\bERROR\b", TEXT, re.IGNORECASE))

exact     : []
ignorecase: ['error']

The case-sensitive version found nothing, correctly — UVM_ERROR has no word boundary before ERROR, as Chapter 19.7 established. The case-insensitive version found the word error in a sentence of prose. For a severity token, case is information, and throwing it away costs more than the two spellings it buys. IGNORECASE is for fields whose capitalisation is genuinely arbitrary, not for every pattern that failed once.

The wrong flag is a silent None. Flags do not validate against the pattern, so a pattern needing DOTALL and given VERBOSE simply fails to match:

import re

TEXT = "Startpoint: a\n  (flop)\nEndpoint: b\n"

print("DOTALL :", re.search(r"Startpoint:(.*?)Endpoint:", TEXT, re.DOTALL) is not None)
print("VERBOSE:", re.search(r"Startpoint:(.*?)Endpoint:", TEXT, re.VERBOSE) is not None)
print("none   :", re.search(r"Startpoint:(.*?)Endpoint:", TEXT) is not None)

DOTALL : True
VERBOSE: False
none   : False

The last two are indistinguishable from each other and from "the report does not contain a timing path". When a cross-line pattern returns None, check the flag before you check the pattern — Chapter 19.17 makes that a routine.

8. Predict, Run, Observe

import re

TEXT = "Warning: one\nInfo: two\nWARNING: three\n"

# A
print("A:", re.findall(r"^warning", TEXT, re.I))

# B
print("B:", re.findall(r"^warning", TEXT, re.M))

# C
print("C:", re.findall(r"^warning", TEXT, re.I | re.M))

# D
print("D:", re.findall(r"one$", TEXT), re.findall(r"one$", TEXT, re.M))

# E
print("E:", re.search(r"one.Info", TEXT) is not None, re.search(r"one.Info", TEXT, re.S) is not None)

# F
print("F:", len(re.findall(r"\AWarning", TEXT, re.M)))

# G
print("G:", re.search(r"W arning", TEXT, re.X) is not None)

The verified output:

A: ['Warning']
B: []
C: ['Warning', 'WARNING']
D: [] ['one']
E: False True
F: 1
G: True

A — IGNORECASE alone: ^ still only means position 0, and the first line happens to match.

B — MULTILINE alone: ^ now works per line, but no line starts with a lowercase warning. An empty list where a reader might expect two.

C — both, and both warnings are found. Section 3.

D — $ is per-line only with MULTILINE.

E — . cannot cross the newline between one and Info until DOTALL is on.

F — \A ignores MULTILINE entirely, so exactly one match. Section 4.

G — with VERBOSE the space inside the pattern is discarded, so W arning matches Warning. Section 6's trap.

9. Exercises

Exercise 1 — three spellings. Build a four-line report containing three capitalisations of one severity. Count them with and without IGNORECASE and say what a nightly summary would have reported.

Exercise 2 — class or flag. Write the same case-insensitive match twice, once with re.IGNORECASE and once with a character class per letter. Say which you would ship and why.

Exercise 3 — the single-flag bug. Reproduce Section 3's three runs on data of your own. For each single-flag run, say in one sentence why its single result looks correct.

Exercise 4 — per-line $. Extract every line of a report that ends in a specific word, using MULTILINE. Then do it with splitlines() and say which reads better for your case.

Exercise 5 — strict anchors. Show that \A ignores MULTILINE while ^ does not, and name a check where you would want the strict version.

Exercise 6 — crossing a newline. Write a pattern that captures the text between two markers on different lines. Confirm it returns None without DOTALL.

Exercise 7 — DOTALL and greedy. Take Exercise 6's pattern, make the quantifier greedy, and run it on a subject containing two marker pairs. Report how much it consumed and name the chapter that predicted it.

Exercise 8 — all of them. Use finditer with DOTALL to extract every marker pair from Exercise 7's subject, stripped.

Exercise 9 — verbose rewrite. Rewrite Chapter 19.12's slack pattern with re.VERBOSE and a comment per line. Confirm it returns the same values on both of that chapter's value rows.

Exercise 10 — the verbose space. Put a literal space in a verbose pattern unescaped and show what it matches. Then fix it three ways: \ , [ ] and \s+.

Exercise 11 — case is information. Take a \b-bounded severity pattern, add IGNORECASE, and find a line of prose it now matches. Say what you would do instead.

Exercise 12 — the silent wrong flag. Run a cross-line pattern with re.VERBOSE instead of re.DOTALL. Record the result and state the two other explanations a None would have had.

10. Summary

• A flag is an optional third argument that changes what the symbols in a pattern mean. Short aliases are re.I, re.M, re.S, re.X.
• Flags combine with |, the bitwise-or — not the regex alternation, despite the shared character.
• IGNORECASE matches a word whatever its capitalisation, which is what a report with three spellings of one severity needs.
• MULTILINE makes ^ and $ mean start and end of a line, matching after every newline and before every newline.
• A per-line case-insensitive anchor needs both flags, and each one alone returns exactly one plausible, wrong result.
• \A and \Z are unaffected by MULTILINE — use them when a check must apply to the whole subject regardless of flags.
• DOTALL lets . cross a newline, which is the only way to capture a field that spans lines.
• DOTALL plus a greedy .* consumes to the last occurrence in the file — the lazy form in the canonical annotation is load-bearing.
• VERBOSE makes pattern whitespace insignificant and allows # comments, which is how a long pattern carries its own explanation.
• A literal space in a verbose pattern must be \ , [ ] or \s+, or it silently disappears.
• A flag can widen a pattern you had made precise — IGNORECASE on \bERROR\b starts matching English prose.
• A wrong flag is a silent None, indistinguishable from a wrong pattern and from absent data.
• Inline (?i) exists and is worth it only when a pattern travels without its flags.

The next chapter covers two constructs that check the text around a match without consuming it — which is how you require context that must not end up in your group.