Tcl · Module 26

Reporting Per Scenario

One directory per scenario is the easy half. The hard half is aggregation — and the obvious way reads the filesystem, which records what happened rather than what was supposed to.

Advanced22 min readtcledapart-iimcmmreportingsignoff

Part II · Module 26 · Chapter 26.6 · Reporting Per Scenario

Eight scenarios produce eight sets of reports. If they land in one directory they overwrite each other; if they land in eight directories the question becomes how to get one answer back out.

Both halves have a defect waiting, and the second one is the module's closing finding.

1. One directory per scenario

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc scenario_dir {root mode corner check} {
    return [file join $root "${mode}__${corner}__${check}"]
}

The name carries the full scenario identity — mode, corner and check — for exactly the reason Chapter 25.5 §3 established: a name built from fewer fields than the thing it identifies collides, and the collision overwrites evidence silently. There the missing field was the binary; here it would be the check, and functional__ss_0p72v_125c without it would have setup and hold reports fighting over one directory.

The double underscore is a small deliberate choice: corner names already contain single underscores, so __ is a separator that cannot be confused with one inside a field. A name that cannot be split back apart unambiguously is a name that will eventually be split wrongly.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== running the plan, one directory per scenario ===
    functional__ss_0p72v_125c__setup             written
    functional__tt_0p80v_25C__setup              SKIPPED (no such corner: tt_0p80v_25C)
    functional__ff_0p88v_m40c__setup             written
    functional__ff_0p88v_m40c__hold              written
    ran 3, skipped 1, of 4 planned

Four planned, three written, one skipped — the Chapter 26.5 §4 loop doing its job. Remember that 3 of 4, because §3 is about what happens to it.

2. What goes in the directory

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

set fh [open [file join $dir timing.rpt] w]
puts $fh "mode   : $mode"
puts $fh "corner : $corner"
puts $fh "check  : $check"
puts $fh "library: [dict get [dict get $::mcmm::corners $corner] library]"
puts $fh "spef   : [dict get [dict get $::mcmm::corners $corner] spef]"

Every report carries a provenance header generated from the dicts the analysis was configured from — so the header cannot disagree with what actually ran, which is Chapter 26.2 §6's argument.

The library and spef lines matter most. A slack number is meaningless without knowing which characterised data produced it, and "which library" is precisely the question asked six months later when two runs disagree. Chapter 24.2 §6 made this argument for area comparisons and Chapter 23.6 for netlists and ECO logs; this is the third place it applies, and it is the same argument each time: a number without its provenance cannot be compared with anything.

What else belongs in a real scenario directory: the full report_timing, the all-violators report (Chapter 22.7), the constraint-check output, the tool log, and the exact command line or do file that produced them.

3. Aggregating by scanning the directory

Eight directories, one answer wanted. The obvious approach reads what is there:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

foreach d [lsort [glob -nocomplain -directory $ROOT -type d *]] {
    set w [read_worst $d]
    lappend found $w
}
set worst [lindex [lsort -real $found] 0]

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== aggregating by scanning the directory ===
    functional__ff_0p88v_m40c__hold               -0.0208
    functional__ff_0p88v_m40c__setup               1.8980
    functional__ss_0p72v_125c__setup              -0.4800
    worst across 3 scenario(s) found : -0.4800
    VERDICT: the design's worst setup slack is -0.4800

That output is wrong in two independent ways, and neither is visible in it.

First: it says 3 scenario(s) found and the plan had 4. The scan reports a confident verdict over whatever directories exist. It has no idea that functional__tt_0p80v_25C__setup was supposed to be there, because the filesystem records what happened, not what was supposed to happen. A scenario that failed, crashed, ran out of licence or was never launched leaves no directory — and a directory scan cannot distinguish "did not need to run" from "did not run".

This is Chapter 24.7 §2's empty-report argument and Chapter 25.4 §6's NO-VERDICT in a third costume: absence of evidence silently becoming evidence of absence.

Second, and unrelated: −0.0208 is a hold slack. The lsort -real ran over every number it found, mixing the hold result from functional__ff_0p88v_m40c__hold in with the setup results — and then the verdict line calls the minimum "the design's worst setup slack".

The scan never parsed the directory names it was reading, so it does not know which check each number came from. It happens to get the right answer here because −0.4800 is below −0.0208 anyway, which is the worst kind of luck: the defect is invisible in this run and will produce a wrong headline the first time a hold slack is the most negative number present.

Ranking setup against hold on one axis is exactly Chapter 24.4 §5's error. Two different physical quantities, one lsort.

4. Aggregating from the plan

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

foreach s $PLAN {
    lassign $s mode corner check
    set d [scenario_dir $ROOT $mode $corner $check]
    set w [read_worst $d]
    if {$w eq ""} { lappend missing [file tail $d] } else { lappend have $w }
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== aggregating from the PLAN instead ===
    functional__ss_0p72v_125c__setup              -0.4800
    functional__tt_0p80v_25C__setup              MISSING
    functional__ff_0p88v_m40c__setup               1.8980
    functional__ff_0p88v_m40c__hold               -0.0208
    3 of 4 planned scenario(s) produced a report
    REFUSED to aggregate: 1 scenario(s) missing:
        functional__tt_0p80v_25C__setup

3 of 4, the missing scenario named, and no verdict at all.

The change is one line of thinking: iterate the plan and look up the directory, rather than iterating the directories. The plan is the statement of what was supposed to happen; the filesystem is the record of what did. Only the first can detect a gap.

This is the fifth appearance of Chapter 24.7 §5's principle, and the clearest: the aggregation needs a second, independent measurement — the plan's length against the number of reports found — and nothing in the reports themselves can supply it.

And "REFUSED to aggregate" is the right behaviour, not an over-reaction.

The temptation is to aggregate the three and note the gap in a warning. That produces a number, and a number gets quoted — at which point "worst slack −0.48 across the corners" enters a review slide, with the absent corner unmentioned.

Refusing means somebody has to decide: rerun the missing scenario, or explicitly record that it is not required. Both are fine; silently reporting over a subset is not, because the result reads identically to a complete run.

The second defect fixes the same way. Group by check before comparing:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

set by_check [dict create]
foreach r $results { dict lappend by_check [dict get $r check] [dict get $r worst] }
dict for {check vals} $by_check {
    puts [format "  worst %-6s %.4f  over %d scenario(s)" \
              $check [lindex [lsort -real $vals] 0] [llength $vals]]
}

One worst per check, each with its denominator — Chapter 24.4 §5's grouping and Chapter 22.7's denominator rule, which have now appeared in every module of Part II.

5. The summary a reviewer can act on

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc summarise {results} {
    set by_check [dict create]
    foreach r $results {
        if {[dict get $r status] ne "OK"} { continue }
        dict lappend by_check [dict get $r check] $r
    }
    dict for {check rows} $by_check {
        set worst "" ; set worst_row ""
        foreach r $rows {
            if {$worst eq "" || [dict get $r worst] < $worst} {
                set worst [dict get $r worst] ; set worst_row $r
            }
        }
        puts [format "  %-6s worst %8.4f  at %s / %s  (over %d scenario)" \
                  $check $worst [dict get $worst_row mode] \
                  [dict get $worst_row corner] [llength $rows]]
    }
}

The summary reports, per check: the worst number, which scenario produced it, and how many scenarios it was taken over.

That middle field is the one that makes the summary actionable. "Worst setup −0.48" sends somebody to look at the design. "Worst setup −0.48 at functional / ss_0p72v_125c" sends them to the right report, which is in a directory named after that scenario, with a header naming the library that produced it. The chain from the summary number back to the raw report is unbroken — which is the registry's phrase for this chapter and the thing the whole module is for.

And the status filter is deliberate: only OK rows contribute a number, because a SETUP-FAILED row has no number to contribute. But the failed rows must still be counted somewhere, which is §4's job — a summary that silently filters failures is the directory scan again, wearing a different hat.

6. The review

Before trusting an MCMM result:

Does every scenario have its own directory, named for mode, corner and check?
Does every report carry its provenance — mode, corner, library, spef, tool version?
Is the aggregation driven by the plan rather than by the filesystem?
Does it refuse to produce a verdict when a planned scenario is missing?
Are setup and hold aggregated separately, never through one lsort?
Does each summary line carry the scenario that produced it and the count it was taken over?
Can a number in the summary be traced to a file on disk — and is that traceability tested, not assumed?
Is the scenario list the same object the driver walked, the reporter wrote and the aggregator read?

That last point closes Module 26. The plan is the single artefact that every stage shares, and every defect in this chapter came from a stage inventing its own idea of what the run consisted of.

7. Exercises

Exercise 1. Why does the directory name carry the check as well as the mode and corner?

Exercise 2. Why __ as the separator?

Exercise 3. Why generate the report's provenance header from the dicts rather than writing it literally?

Exercise 4. The directory scan reported 3 scenario(s) found. What can it not know, and why?

Exercise 5. What is the scan's second, unrelated defect?

Exercise 6. Why is the second defect's invisibility in this run the worst case?

Exercise 7. What single change fixes the first defect?

Exercise 8. Why refuse to aggregate rather than warn?

Exercise 9. Why does the summary name the scenario that produced the worst number?

Exercise 10. Why does the summary filter to OK rows, and what must happen to the others?

Answers.

1. Because a scenario's identity is mode, corner and check. Without the check, the setup and hold reports for one mode-corner pair would collide in one directory and overwrite each other.

2. Because corner names already contain single underscores, so __ is a separator that cannot be confused with one inside a field. A name that cannot be split back apart unambiguously will eventually be split wrongly.

3. So the header cannot disagree with what actually ran. A literal header goes stale the first time a period or a library changes, and then every report in the directory is confidently mislabelled.

4. Which scenarios are missing. The filesystem records what happened, not what was supposed to happen, so a scenario that failed or never launched leaves no directory — and a scan cannot distinguish "did not need to run" from "did not run".

5. It lsort -reals over setup and hold slacks together, so its "worst setup slack" verdict was computed from a set that included a hold number.

6. Because it happens to give the right answer here, so nothing flags it — and it will produce a wrong headline the first time a hold slack is the most negative number present.

7. Iterate the plan and look up each directory, rather than iterating the directories. The plan is the statement of what was supposed to happen.

8. Because a number gets quoted. Reporting over a subset produces a result that reads identically to a complete run, with the absent scenario unmentioned. Refusing forces a decision: rerun it, or record that it is not required.

9. Because that is what makes it actionable — it points at the directory, which has the report, which has the header naming the library. The chain from summary to raw data stays unbroken.

10. Because a failed scenario has no number to contribute. The failures must still be counted by the plan-driven check, or the summary becomes the directory scan again in a different form.

Remember.

One directory per scenario, named for mode, corner and check — the full identity, with a separator that cannot occur inside a field.
Every report carries its provenance, generated from the dicts the analysis used so it cannot go stale.
A number without its library and spef cannot be compared with anything.
Aggregating by scanning the directory is wrong twice: it cannot know what is missing, and it compared hold slacks against setup slacks.
The filesystem records what happened; the plan states what was supposed to. Only the plan detects a gap.
Refuse to produce a verdict when a planned scenario is missing — a partial number reads exactly like a complete one.
Group by check before taking a worst, and carry the denominator.
Name the scenario that produced each worst number, so the chain back to the raw report is unbroken.
Filter the summary to successful rows, but count the failures elsewhere.
One scenario list, shared by the driver, the reporter and the aggregator — every defect in this chapter came from a stage inventing its own.

Where this fits

Part of the Tcl curriculum.