Tcl · Module 24

Writing CSV Summaries

Getting data out is the easy half. Getting it out so the next tool reads it back correctly is where a library call that looks right corrupts a row — and says nothing.

Advanced21 min readtcledapart-iireportscsvdata-exchange

Part II · Module 24 · Chapter 24.6 · Writing CSV Summaries

Five chapters of parsing produce a list of dicts in memory, which vanishes when the script exits. This chapter writes it somewhere the next thing in the chain can read — a spreadsheet, a diff against last night's run, a plotting script, a dashboard.

CSV is the usual answer, and it is almost trivial. The "almost" is the chapter.

1. The four hazards

A CSV field needs quoting when it contains the separator, the quote character itself, or a line break. Each of those turns up in real report data:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc csv_field {s} {
    if {[regexp {[",\n\r]} $s]} {
        return "\"[string map [list "\"" "\"\""] $s]\""
    }
    return $s
}
proc csv_row {fields} {
    set out {}
    foreach f $fields { lappend out [csv_field $f] }
    return [join $out ","]
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== one field per hazard ===
    plain         u_mem/addr_reg_7/D
    has-a-comma   "core_clk, io_clk"
    has-a-quote   "sized 2x ""for setup"""
    has-a-newline "line one
line two"

The rules, which are RFC 4180's:

A field needs quoting if it contains ", ,, newline or carriage return. The character class [",\n\r] is the whole test.
Inside a quoted field, a quote is doubled, not backslash-escaped. string map does that in one pass. This is the rule people get wrong most often, because every other format in the world escapes with a backslash.
Otherwise emit the field as-is. Quoting everything unconditionally is also valid CSV and some writers do it; it makes files larger and diffs noisier, and it hides which fields actually needed it.

Where these fields come from in practice: a clock-group list is comma-separated inside one field (Chapter 20.7), an ECO reason is free text an engineer typed and may contain anything, and a multi-line field appears whenever a tool's own text block is carried through verbatim.

2. Check it against a library

Tcl ships tcllib, which has a csv package. On this interpreter it is present, so the hand-written quoter can be compared against it field by field:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

package require csv
foreach h $hazards {
    set v [lindex $h 1]
    set same [expr {[csv_row [list $v]] eq [::csv::join [list $v]]}]
    puts [format "    %-13s agree %d" [lindex $h 0] $same]
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== against tcllib's ::csv::join (version 0.7.2) ===
    plain         agree 1
    has-a-comma   agree 1
    has-a-quote   agree 1
    has-a-newline agree 0
    mine   : "line one<NL>line two"
    tcllib : line one<NL>line two

Three agreements and one disagreement. When your code and a shipped library disagree, the library is usually right — so this needs settling rather than assuming.

3. Settling it: three independent confirmations

First, an independent reader. Both files were written and then read by Python's csv module, which implements RFC 4180 and has no connection to either writer:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  mine.csv -> 3 row(s)
      ['a', 'core_clk, io_clk', '1']
      ['b', 'has "quotes"', '2']
      ['c', 'line one\nline two', '3']
  theirs.csv -> 4 row(s)
      ['a', 'core_clk, io_clk', '1']
      ['b', 'has "quotes"', '2']
      ['c', 'line one']
      ['line two', '3']

Three rows were written. The hand-quoted file reads back as 3 rows with the newline preserved inside field 3. The library-quoted file reads back as 4 rows, with row c split in half and line two promoted to the first column of a row that does not exist.

That is not a formatting difference. The data is corrupt: a row appeared, a field moved column, and a value that was one string is now two.

Second, the library's own source. info body on the procedure shows what the quoting test is:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if {[string match "*\[${delChar}$sepChar\]*" $val]} {

A string match glob whose character class contains exactly two characters: the delimiter (") and the separator (,). Newline and carriage return are not in it, and because the class is built from the two arguments the procedure takes, there is no option that adds them. The behaviour is unconditional, not a default.

Third, the shape of the agreement. ::csv::join is correct for the comma and correct for the quote — the two cases everyone tests. It is wrong only for the case that is rare in test data and routine in real data. That is exactly the profile of a defect that survives for years.

The honest scope of this claim.

This is csv 0.7.2 with Tcl 8.5.9, which is what exists on the machine these chapters were verified on. A later tcllib may behave differently, and the two-minute check is:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

package require csv
puts [::csv::join [list "a\nb"]]

If the output has no quotes around it, the version you have shares the behaviour. That probe is worth running rather than trusting this paragraph — Chapter 17.1 is the general argument for establishing what your interpreter actually is instead of assuming.

And the practical advice is not "avoid tcllib". It is do not assume a vendor EDA interpreter even has it. Tool shells routinely ship a stripped Tcl with no tcllib at all, so a parser that depends on package require csv fails to load inside the one environment it was written for. A 6-line csv_field that works everywhere is worth more than a library call that works on your workstation.

4. Writing the real table

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

set rows [parse_violators reports/violators.rpt]
set fh [open out/violations.csv w]
puts $fh [csv_row {endpoint check group required actual slack}]
foreach r $rows {
    puts $fh [csv_row [list [dict get $r endpoint] [dict get $r check] [dict get $r group] \
                            [dict get $r required] [dict get $r actual] [dict get $r slack]]]
}
close $fh

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== write the parsed violation table ===
endpoint,check,group,required,actual,slack
u_mem/addr_reg_7/D,max_delay/setup,core_clk,4.8200,4.8930,-0.0730
u_mem/dout_reg_3/D,max_delay/setup,core_clk,4.8200,4.8390,-0.0190
u_ctrl/state_reg_0/D,max_delay/setup,core_clk,4.8200,4.8410,-0.0210
u_dec/sel_reg_2/D,max_delay/setup,core_clk,4.8200,4.9510,-0.1310
u_io/u_sync/sync_reg_1/D,min_delay/hold,core_clk,0.1200,0.0890,-0.0310

Not one field in that output needed quoting — which is precisely why the quoting must be in the writer rather than decided by inspection. The day an endpoint name contains a comma, or an engineer's comment field is added, the writer handles it and nobody has to remember.

Three decisions worth stating.

The header row comes from the same csv_row. Column names are fields and can need quoting too, and routing them through the same function means the header can never disagree with the rows about the quoting convention.

The column order is written down once, in the header, and the loop follows it. Taking the order from dict keys instead would make it depend on insertion order — stable in practice, but a silent coupling between the parser's internals and the file's columns.

The values go out as strings, exactly as parsed. 4.8200 is written as 4.8200, not as 4.82. Passing a parsed field through expr or format on the way out would normalise it and destroy the significant figures the tool chose to print, which is a small lie about precision and a real nuisance when diffing two runs.

5. Reading it back

A writer is not verified until something reads it. Within Tcl:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc csv_split {line} {
    set fields {} ; set cur "" ; set inq 0
    set n [string length $line]
    for {set i 0} {$i < $n} {incr i} {
        set c [string index $line $i]
        if {$inq} {
            if {$c eq "\""} {
                if {[string index $line [expr {$i + 1}]] eq "\""} {
                    append cur "\"" ; incr i
                } else { set inq 0 }
            } else { append cur $c }
        } else {
            if {$c eq "\""}     { set inq 1 } \
            elseif {$c eq ","}  { lappend fields $cur ; set cur "" } \
            else                { append cur $c }
        }
    }
    lappend fields $cur
    return $fields
}

Round-tripped against the writer over five rows — including an empty field, a field beginning with a comma, a field ending with one, and a field whose own content is quoted:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== writer then reader, field by field ===
    round-tripped 5 of 5 rows, 0 mismatch(es)

Worth looking at what makes that reader longer than the writer. The doubled-quote rule means a " inside a quoted field is only a terminator if the next character is not also a ", so the reader needs one character of lookahead and an explicit incr i to consume the pair.

There is no way to express that with split:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== and why split is not a CSV reader ===
    the line        : u_dec/sel_reg_2/D,"core_clk, io_clk",-0.1310
    csv_split gives : 3 field(s)
    split , gives   : 4 field(s)

Three fields, and split reports four. It has no concept of quoting, so the comma inside "core_clk, io_clk" is a separator to it. Every field after that point is then off by one — which is a row whose columns have silently shifted, the same corruption as §3 arriving from the reading side instead of the writing side.

And the reader above still does not handle an embedded newline, because by the time it sees a line, the line has already been split on the newline by gets. A fully correct CSV reader cannot be line-oriented — it has to read characters and decide where rows end, because a quoted newline is data and an unquoted one is a row boundary. That asymmetry is why §3's corruption was invisible to the writer and fatal to the reader.

6. CSV is a poor format, and what to do about it

Being honest, since the registry's purpose is that the next tool "reads the file rather than guessing at it", and CSV forces a certain amount of guessing.

No types. -0.0730 is a string. The reader decides whether it is a number, and a reader that guesses from the first row gets it wrong when a later row says N/A.

No units. This is Chapter 24.3's entire lesson thrown away at the output stage. A column headed slack with -0.0730 in it does not say nanoseconds, and a column headed power is worse. Put the unit in the column name — slack_ns, leakage_w — because it is the only place CSV has to put it.

No provenance. Which run, which corner, which netlist, which tool version. Chapter 24.2 §6 argued that a comparison without provenance subtracts numbers that are not comparable, and a CSV file of bare numbers is exactly that trap handed to the next person. Either add columns for it or write a sibling metadata file next to the CSV, named for the same run — the pairing argument from Chapter 23.6 §6.

No schema. A consumer cannot tell a renamed column from a missing one, which is Chapter 24.7's problem reappearing one layer downstream. If you control both ends, write the column list into the metadata file and have the reader assert it.

When not to use CSV at all.

Hierarchical results — per-corner, per-mode, per-group — flatten into CSV badly, either with a combinatorial explosion of columns or with a key column whose meaning is in a comment somewhere. JSON is the better target, and json::write is in tcllib with the same availability caveat as §3.
A file only Tcl will read needs no CSV at all. A list of dicts written with puts $fh $rows and read back with source or read is exact, typed in Tcl's terms, needs no quoting rules, and round-trips perfectly — because list quoting is Tcl's own, which is Chapter 6.2's point.
Anything a spreadsheet will open should be CSV, because that is the one thing CSV is unambiguously good at.

The reason to write CSV is that everything reads it. That is a real and sufficient reason, and it is the only one.

7. Exercises

Exercise 1. What three things make a CSV field need quoting?

Exercise 2. How is a quote escaped inside a quoted field, and why is that surprising?

Exercise 3. ::csv::join agreed on the comma and the quote and disagreed on the newline. Why is that pattern typical of a long-lived defect?

Exercise 4. Name the three independent ways the tcllib finding was confirmed.

Exercise 5. After reading the corrupted file, three written rows came back as four. What exactly happened to the third row?

Exercise 6. Why is "do not assume a vendor interpreter has tcllib" better advice than "do not use tcllib"?

Exercise 7. Why route the header row through csv_row as well?

Exercise 8. Why write 4.8200 rather than passing it through format?

Exercise 9. Why can a fully correct CSV reader not be line-oriented?

Exercise 10. Give two things CSV cannot express that a report parser knows.

Answers.

1. The separator (,), the quote character ("), or a line break (newline or carriage return).

2. By doubling it — "". It is surprising because almost every other format escapes with a backslash, so the backslash is the habit people bring and it produces a file that reads back wrongly.

3. Because the comma and the quote are the cases everyone writes a test for, and an embedded newline is rare in test data and routine in real data. A defect that passes every obvious test survives.

4. Behaviour (agreement on three hazards, disagreement on the fourth); an independent RFC 4180 reader in another language recovering 3 rows from one file and 4 from the other; and the library's own source, whose quoting test is a character class containing only the delimiter and the separator with no option to extend it.

5. It was split at the embedded newline. ['c', 'line one'] became one row and ['line two', '3'] became another — so a row appeared, a field changed column, and one string became two.

6. Because tcllib is fine where it exists; the problem is that EDA tool shells routinely ship a stripped Tcl without it, so a package require csv fails to load in the one environment the parser was written for.

7. Because column names are fields too and can need quoting, and sharing the function makes it impossible for the header and the rows to disagree about the convention.

8. Because the tool chose those significant figures and format would normalise them away. It is a small lie about precision and a real nuisance when diffing two runs.

9. Because a quoted newline is data and an unquoted one is a row boundary, so where rows end cannot be decided until the quoting state is known. gets has already split on newlines before the reader sees anything.

10. Any two of: types (everything is a string), units (which is 24.3's lesson discarded at the output stage), provenance (run, corner, netlist, tool version), and schema (a renamed column is indistinguishable from a missing one).

Remember.

Quote when a field contains ", ,, newline or CR — [",\n\r] is the whole test.
A quote is escaped by doubling it, not with a backslash.
tcllib csv 0.7.2 does not quote an embedded newline, so the file it writes reads back with an extra row and a field in the wrong column. Confirmed by behaviour, by an independent RFC 4180 reader, and by the library's own source.
Do not assume a vendor interpreter has tcllib at all. Six lines that work everywhere beat a library call that works on your workstation.
Put the writer's quoting in the writer, not in a judgement about whether this data needs it — today's data does not and tomorrow's will.
Route the header through the same function, and write the column order down rather than taking it from dict keys.
Emit values exactly as parsed. 4.8200 is not 4.82.
split $line "," is never a CSV reader, and a correct reader cannot be line-oriented, because a quoted newline is data.
CSV carries no types, units, provenance or schema — put units in the column names and provenance in a sibling file named for the same run.
The reason to write CSV is that everything reads it. That is the only reason, and it is enough.

Where this fits

Part of the Tcl curriculum.