Tcl · Module 24

Cleaning Tool Output Before Parsing

Banners, rules, page breaks and control characters are a quarter of the file. Removing them first is cheaper than teaching every pattern to ignore them — and one piece of standard advice turns out to be a no-op here.

Intermediate18 min readtcledapart-iireportstext-processing

Part II · Module 24 · Chapter 24.5 · Cleaning Tool Output Before Parsing

The previous four chapters wrote patterns that had to survive whatever else was in the file. This one removes the whatever-else first, and the case for doing so starts with a measurement.

1. How much of a report is not content

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

set fh [open $path r]
array set kind {}
set total 0
while {[gets $fh line] >= 0} {
    incr total
    set t [string trim $line]
    if {$t eq ""}                        { incr kind(blank)   ; continue }
    if {[regexp {^[*]+$} $t]}            { incr kind(banner)  ; continue }
    if {[regexp {^[-=]+$} $t]}           { incr kind(rule)    ; continue }
    if {[string first "\f" $line] >= 0}  { incr kind(formfeed) ; continue }
    if {[regexp {^\d+$} $t]}             { incr kind(pageno)  ; continue }
    incr kind(content)
}
close $fh

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== what the lines are, counted by kind ===
    banner      2
    blank       7
    content    35
    formfeed    1
    pageno      1
    rule        2
    TOTAL      48
    content is 73% of the file

27% of this small report is decoration, and a real one is worse: page headers repeating every 60 lines, re-printed column titles, continuation markers, a trailer summarising the run.

That is the argument for a cleaning pass. Not elegance — every pattern in every parser otherwise has to be written to coexist with a quarter of the file that is not data, and §4 explains why that coexistence is where the bugs come from.

Two details in the census worth noting. array set kind {} then incr kind(blank) works because incr treats a missing element as zero, so no key needs pre-creating (Chapter 7.9). And the form-feed test uses string first rather than a regexp because \f is a single character and string first is the direct question — Chapter 5.4 over Chapter 13.2 when there is no pattern involved.

2. The carriage return everybody tells you to strip

The standard advice for parsing tool output is to strip \r first, because reports produced on or for Windows have CRLF line endings. The mock report used in this module genuinely has them.

So the advice should be testable. It is:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

foreach tr {auto lf binary} {
    set fh [open $path r]
    fconfigure $fh -translation $tr
    set l ""
    while {[gets $fh x] >= 0} { if {[string match "*slack (VIOLATED)*" $x]} { set l $x ; break } }
    close $fh
    puts [format "    -translation %-7s len %-3d ends with CR %d   anchored match %d" \
              $tr [string length $l] [expr {[string index $l end] eq "\r"}] \
              [regexp {(-?\d+\.\d+)$} $l]]
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the CR question, measured rather than assumed ===
    -translation auto    len 60  ends with CR 0   anchored match 1
    -translation lf      len 61  ends with CR 1   anchored match 0
    -translation binary  len 61  ends with CR 1   anchored match 0

On a normally-opened channel, Tcl has already removed the carriage return — so string map {"\r" ""} $line is a no-op.

Tcl channels default to -translation auto on reading, which converts CRLF (and lone CR) line endings to LF transparently. The line comes back 60 characters long with no CR, and a $-anchored pattern matches. The same is true of read and of exec output, both of which go through channels with the same default — I checked all three.

The CR survives only when translation is explicitly turned off: -translation lf or -translation binary. And then look at the consequence — anchored match 0. The pattern (-?\d+\.\d+)$ silently stops matching, because $ cannot match before the CR. No error, no warning, just a field that is suddenly never found.

So the folklore is not wrong about the danger; it is wrong about when. Stripping CR unconditionally costs nothing and documents nothing. Knowing that the hazard appears exactly when a channel is opened binary — which parsers do, for instance when they also need to detect a null byte or count bytes exactly — is the useful form of the knowledge.

This chapter's registry brief carries the unconditional version of the advice, and it is kept here as what it is: a harmless habit whose stated reason does not apply to a normally-opened channel on this interpreter. Verified on Tcl 8.5.9 — a vendor interpreter with a different default could behave otherwise, and the three-line probe above is how to find out rather than assume.

3. The form feed does survive

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

    formfeed    1

Unlike the CR, the page-break character is not touched by translation — it is not a line terminator, so gets returns it as part of a line. In this report it is the only thing on its line, which means a parser sees a one-character line of \f that matches nothing and belongs to nothing.

It is harmless there, and it is not harmless when a tool writes it mid-line, immediately before a repeated page header. Then the line is \fReport : timing, and a pattern anchored with ^Report does not match it — the same silent non-match as §2, from a different cause.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

set line [string map [list "\r" "" "\f" ""] $line]

One string map with both replacements, which is cheaper and clearer than two passes (Chapter 5.9). Control characters generally are worth stripping as a class if a tool emits them:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

regsub -all {[\x00-\x08\x0b-\x1f]} $line "" line

That range deliberately excludes \x09 (tab) and \x0a (newline), and includes \x0b through \x1f, so form feed and carriage return are removed while tabs — which can be legitimate column separators — survive. Being deliberate about which control characters are decoration and which are structure is the whole of it.

4. Clean first, then match

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc clean_line {line} {
    set line [string map [list "\r" "" "\f" ""] $line]
    set line [string trim $line]
    if {$line eq ""}              { return "" }
    if {[regexp {^[*]+$} $line]}  { return "" }
    if {[regexp {^[-=]+$} $line]} { return "" }
    if {[regexp {^\d+$} $line]}   { return "" }
    return $line
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== a cleaner, and what it leaves ===
    kept 35, dropped 13
      Report : timing
      -path full
      -delay max
      -max_paths 3
      Design : core_top
      Version: V-2099.03-TEACHING-MOCK

35 kept, 13 dropped — matching the census exactly, which is a small cross-check of the cleaner against the counter.

The ordering is the point of the section.

A pattern written to tolerate decoration is a pattern that will match decoration somewhere else.

Suppose a parser skips the cleaning pass and instead writes its patterns loosely enough to work anyway — unanchored, tolerant of leading junk, happy with extra whitespace. Each of those relaxations is a widening of what the pattern accepts, and the file has 13 lines of decoration for it to accidentally accept.

That is not hypothetical. Chapter 24.4 §3's data-row pattern works because it is anchored at both ends and describes an exact shape. Loosen it to survive a page header appearing mid-table and it starts accepting the header row, the rule lines, and the total.

So the two are in tension: the dirtier the input, the tighter the pattern must be; and the tighter the pattern, the more it needs the input clean. Cleaning first resolves the tension in the direction that keeps patterns specific.

And a cleaner that runs after a pattern has failed is useless — which sounds obvious, and is the actual shape of the bug when a parser handles decoration inside each if branch rather than before the chain.

5. Where cleaning goes too far

Honest limits, because every rule in §4's cleaner can destroy real data.

string trim destroys column positions. A fixed-width table — and the group table in the power report is one — is parsed by character offset or by counting whitespace-separated fields. Trimming is safe for the second and fatal for the first, and a report with an empty column in the middle cannot be split on whitespace at all, because the separator and the missing value look identical.

^[-=]+$ drops a legitimate row. A table whose endpoint column contains - as a placeholder, or an area report row for a cell literally named ---, is deleted by the rule meant for horizontal rules. Unlikely here; not unlikely across every report format a flow produces.

^\d+$ drops a legitimate row too. It is meant for page numbers. It also deletes a line that is a bare count — and a summary line consisting only of a number is a perfectly normal thing for a report to contain.

The cleaner is now a second parser. Every rule in it is a claim about the format, and those claims go stale exactly like the ones in the main patterns do. Chapter 24.7 tests the parser against a renamed field; the cleaner deserves the same treatment and rarely gets it, because nobody thinks of it as logic.

The resolution is to keep the cleaner narrow and reversible:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc clean_line {line} {
    # Remove characters that are never data. Nothing else.
    return [string map [list "\r" "" "\f" ""] $line]
}

Strip the control characters, which are unambiguously decoration, in the cleaner. Leave the line-level decisions — this is a rule, this is a banner — to the main parser's shape-matched patterns, where they are visible next to the data patterns they coexist with rather than hidden in a helper two files away.

That is the narrower claim, and the one worth defaulting to.

6. Exercises

Exercise 1. What fraction of the measured report was decoration, and why does that justify a cleaning pass?

Exercise 2. Why is string map {"\r" ""} a no-op on a normally-opened channel here?

Exercise 3. When does the CR actually survive, and what is the symptom?

Exercise 4. Why does the form feed survive when the CR does not?

Exercise 5. Why does the control-character range exclude \x09 and \x0a?

Exercise 6. Explain the tension between dirty input and tight patterns.

Exercise 7. Give two ways string trim can break a parser.

Exercise 8. Why keep the cleaner narrow and leave line-level decisions in the main parser?

Answers.

1. 27% — 13 of 48 lines. It justifies the pass because otherwise every pattern in the parser must be written to coexist with a quarter of the file that is not data.

2. Because Tcl channels default to -translation auto, which converts CRLF to LF on read. The CR is gone before the line reaches your code. read and exec behave the same way.

3. Only when translation is explicitly lf or binary. The symptom is that a $-anchored pattern silently stops matching, because $ cannot match before the CR — no error, just a field that is never found.

4. Because a form feed is not a line terminator, so translation does not touch it. gets returns it as part of a line.

5. Because \x09 is tab and \x0a is newline. Tabs can be legitimate column separators, so removing them as "control characters" would destroy structure rather than decoration.

6. The dirtier the input, the tighter a pattern must be to avoid accidentally matching decoration; but the tighter the pattern, the more it depends on the input being clean. Cleaning first resolves it in the direction that keeps patterns specific, which is the direction that keeps them correct.

7. It destroys column positions, which a fixed-width table parser depends on; and it cannot help a whitespace-split parser distinguish an empty middle column from the separator around it.

8. Because the cleaner is itself a second parser full of claims about the format, and those claims go stale. Keeping it to characters that are never data, and leaving "this line is a rule" to the shape-matched patterns, puts each claim next to the data pattern it coexists with instead of hiding it in a helper.

Remember.

A quarter of a report file is decoration — 13 of 48 lines measured here, and worse in a real one.
string map {"\r" ""} is a no-op on a normally-opened channel: -translation auto already removed it, and read and exec are the same.
The CR survives only under -translation lf or binary — and then a $-anchored pattern silently stops matching.
The form feed does survive, because it is not a line terminator. Strip \r and \f in one string map.
Exclude tab and newline from any control-character sweep — they can be structure.
Dirty input demands tight patterns; tight patterns demand clean input. Clean first.
A pattern loosened to tolerate decoration will eventually match decoration.
string trim breaks fixed-width parsing, and ^[-=]+$ or ^\d+$ can delete a legitimate row.
Keep the cleaner to characters that are never data. Line-level judgements belong in the parser, visible beside the data patterns.
Measure the channel's behaviour rather than inheriting the folklore — the three-line fconfigure probe above settles it on any interpreter.

Where this fits

Part of the Tcl curriculum.