Automating GLS Triage & Reporting

Chapter 13 · Section 13.5 · Simulation Performance & Regression Strategy

Project thread — the mini-SoC's GLS regression is only sustainable with automated triage. This lesson closes the performance chapter; Chapter 14 walks the case studies end-to-end (RTL → clean GLS signoff).

1. Why Should I Learn This?

A GLS regression at scale is unmanageable by hand — automation is what makes it sustainable.

• Auto-classify known-expected-waived vs new/real.
• Auto-detect the first divergence (12.2); track waivers (justified, not wholesale).
• Report new-vs-known so engineers see only what's new.

This operationalizes the debug methodology (Ch12) across a regression — the last piece of affordability.

2. Real Silicon Story — engineers drowning in known Xs

A team's GLS regression produced thousands of Xs every run — mostly known, expected ones (power-down corruption, intentional notifier firings) — and engineers manually re-triaged the same thousands each morning, missing the occasional new one in the flood.

Automation fixed it: a classifier waived the known-expected Xs by rule (each with a justification), auto-detected first divergence, and the report surfaced only the handful of new findings. The new ones — the ones that mattered — were now visible instead of buried, and manual effort dropped from hours to minutes.

Lesson: automate GLS triage — classify known-expected-waived Xs away (with justification) and surface only the new/real. Manual re-triage of known noise buries the new signal.

3. Concept — automating triage & reporting

The problem: a GLS regression floods with Xs/violations/mismatches — most are known/expected, but the new/real ones are what matter, and they're buried.

(1) Auto-classify each finding:

• Known-expected-waived — a power-down corruption X (10.2), a notifier X on an intentionally-checked path (8.5), a crossing X (9.2), or a previously-triaged waiver → classify away (with justification).
• New / real — not matching any waiver → surface for human review.

(2) Auto-detect first divergence (12.2):

• Per failure, find the earliest diverging signal/time → a wall of downstream symptoms collapses to one root.

(3) Manage waivers (justified, tracked):

• A waiver database — each entry has a reason, owner, scope (a specific signal/condition), and is reviewable.
• NOT wholesale check-disabling (8.4/12.6) — waivers are narrow and justified; new findings are never auto-waived.

(4) Report clearly:

• Pass/fail, new-vs-known failures, an X report (classified), and what ran (tier/fidelity, 13.4).
• A dashboard: known noise classified away, new signal surfaced.

The goal:

• Reduce manual triage to only the new/real — automate the funnel's real-vs-artifact and first-divergence steps (12.1/12.2), leaving judgment to humans on the new findings.

Scope (accuracy):

• Automation classifies and surfaces — humans review new findings; waivers are justified and tracked, never a wholesale silencing (8.4). GLS stays dynamic (0.3).

4. Mental Model — a spam filter for regression noise

Automated triage is a spam filter for your GLS regression.

• Thousands of messages arrive each run (Xs/violations) — most are known (newsletters you subscribed to: expected corruption/notifier/crossing Xs).
• The filter classifies the known ones into folders (waived, with a reason) so your inbox isn't buried.
• New senders (new/real findings) land in the inbox — the few things you actually read.
• The filter uses rules you wrote and can review (the waiver DB) — not a blanket "delete everything" (wholesale check-disable), which would also trash a real message.
• You still read the inbox (humans review the new) — the filter just removes the known noise.

A reviewable filter that surfaces new signal — not a delete-all that hides real messages.

5. Working Example — the automation flow

The auto-triage and report (tool-neutral):

# Automated GLS triage — REPRESENTATIVE (tool-neutral):
# for each finding (X / violation / mismatch):
#   1) auto-detect FIRST divergence (12.2) -> one root per failure
#   2) auto-CLASSIFY against the waiver DB:
#        matches a known-expected-WAIVED rule (corruption 10.2 / notifier 8.5 / crossing 9.2)? -> classify away
#        no match -> NEW/real -> surface for human review
#   3) report

# Waiver DB entry — REPRESENTATIVE (justified + tracked, NOT wholesale disable, 8.4):
#   signal:  u_dut.u_blk.* (power domain PD_BLK outputs)
#   finding: corruption X during power-down (10.2)
#   reason:  expected -- domain powered down; isolation verified (10.3)
#   owner:   <name>   scope: PD_BLK power-down window   review: <date>
# -> NEW findings NEVER auto-waived; each waiver is narrow, justified, reviewable.
 
# Regression report — REPRESENTATIVE:
#   PASS: 412  FAIL: 3  (NEW: 1, KNOWN-WAIVED: 2)   <- engineers look at the 1 NEW
#   X-report: 1840 X (1837 known-waived, 3 NEW)      what-ran: nightly zero-delay subset (13.4)

Practical context (representative, tool-neutral):

# Triage automation checklist (tool-neutral):
#   [ ] auto-detect first divergence per failure (12.2)
#   [ ] auto-classify vs a JUSTIFIED, TRACKED waiver DB (reason/owner/scope) -- NOT wholesale disable (8.4)
#   [ ] known-expected (corruption 10.2 / notifier 8.5 / crossing 9.2) -> classify away
#   [ ] NEW/real -> surface for human review (never auto-waive new)
#   [ ] report: pass/fail, NEW-vs-KNOWN, X-report, what-ran (tier/fidelity, 13.4)
#   goal: engineers see ONLY the new/real

Known (classified-away) vs new (surfaced) failures across runs, as a real waveform:

6. Debugging Session — manual triage drowning in known Xs

7. Common Mistakes

• Manually re-triaging known noise every run — slow, error-prone, buries the new.
• Wholesale check-disabling to quiet the flood (8.4) — masks real violations.
• Auto-waiving new findings. New is never auto-waived — surface it.
• Untracked waivers. Each needs a reason/owner/scope, reviewable.
• No new-vs-known reporting. Engineers must see what's new at a glance.

8. Industry Best Practices

• Auto-classify findings (known-expected-waived vs new/real) against a justified waiver DB.
• Auto-detect first divergence (12.2) per failure.
• Track waivers (reason/owner/scope, reviewable) — never wholesale disable (8.4).
• Never auto-waive new — surface it for human review.
• Report new-vs-known (+ X-report, what-ran) on a dashboard.

Senior Engineer Thinking

• Beginner: "There are 1,800 Xs again — I'll go through them like yesterday."
• Senior: "Most are the same known-expected ones — the automation should classify those away (with justification) and show me only the new. If a new X is hiding in that flood, manual re-triage will miss it. Automate the funnel; review the new."

The senior automates classification of known noise and reviews only the new/real findings.

Silicon Impact

At scale, automated triage is what keeps a GLS regression trustworthy. A flood of known-expected Xs (corruption, notifier, crossing — mostly expected modeling, 12.6) re-triaged by hand is not just slow — it buries the new/real finding that could be a reset gap, a timing effect, a low-power leak, letting it reach silicon (0.3) simply because it was lost in the noise. Auto-classifying the known away (with justification), auto-detecting first divergence, and surfacing only the new makes the regression manageable and the real bugs visible — while tracked, narrow waivers (never a wholesale silencing, 8.4) keep the automation honest. This closes the performance chapter: 13.1 (why slow) → 13.2 (speed) → 13.3 (what to run) → 13.4 (pipeline) → 13.5 (automation) together make GLS fast, well-scoped, well-placed, and sustainable — affordable enough to run every tape-out, and trustworthy enough to gate it.

Engineering Checklist

• Auto-classified findings (known-expected-waived vs new/real) via a justified waiver DB.
• Auto-detected first divergence (12.2) per failure.
• Tracked waivers (reason/owner/scope) — not wholesale disable (8.4).
• Never auto-waived new — surfaced for human review.
• Reported new-vs-known (+ X-report, what-ran) on a dashboard.

Try Yourself

1. Take a GLS regression's X/violation log and separate known-expected (corruption/notifier/crossing) from new.
2. Observe: the vast majority are known — the same every run — and a new one is easy to miss by hand.
3. Change: write classification rules (a waiver DB with reasons) to waive the known and surface only the new; add first-divergence detection.
4. Expect: the report collapses to a handful of new findings — the ones that matter, now visible. Manual triage drops from hours to minutes.

Triage automation is scripting around any simulator's output. No paid tool required for the concept.

Interview Perspective

• Weak: "We go through the GLS failures manually each run."
• Good: "We auto-classify known-expected Xs and surface the new ones for review."
• Senior: "We automate the funnel's first steps: auto-detect first divergence, auto-classify each finding against a justified, tracked waiver DB (corruption/notifier/crossing Xs classified away with reasons), and surface only the new/real for human review — new is never auto-waived. The report breaks down new-vs-known. Waivers are narrow and justified, never a wholesale check-disable — that keeps the automation honest and the real bugs visible."

9. Interview / Review Questions

10. Key Takeaways

• A GLS regression floods with Xs/violations/mismatches — most known-expected — so automate triage to surface only the new/real (automating the funnel's first steps, 12.1/12.2).
• Auto-classify each finding: known-expected-waived (corruption 10.2, notifier 8.5, crossing 9.2, prior waivers) → classify away (with justification); new/real → surface for human review (never auto-waive new).
• Auto-detect the first divergence (12.2) per failure — collapse downstream symptoms to one root.
• Manage waivers in a justified, tracked, reviewable database (reason/owner/scope) — not a wholesale check-disable (8.4/12.6).
• Report pass/fail, new-vs-known, an X-report, and what ran (13.4) — so engineers review only what's new. This closes the chapter (13.1 slow → 13.2 speed → 13.3 what → 13.4 pipeline → 13.5 automation), making GLS affordable and sustainable; it stays dynamic (0.3). This closes Chapter 13; next, Chapter 14 — industry case studies (RTL → clean GLS signoff).

Quick Revision

Automate GLS triage (the funnel's first steps, 12.1/12.2): auto-detect first divergence, auto-classify vs a justified, tracked waiver DB — known-expected (corruption 10.2 / notifier 8.5 / crossing 9.2) classified away, new/real surfaced (never auto-waive new). Report pass/fail + new-vs-known + X-report + what-ran (13.4). Waivers = justified/tracked/narrow, NOT wholesale disable (8.4). Goal: engineers see only the NEW. Chapter 13 complete; next: Chapter 14 — industry case studies.