Tcl · Module 29

Reviewing a Flow for Debuggability

Debuggability is not a quality you assess by reading a flow. It is three questions, each with a test you can run today on last night's failure.

Advanced20 min readtcledapart-iidebuggingreviewsignoff

Part II · Module 29 · Chapter 29.6 · Reviewing a Flow for Debuggability

A flow is debuggable when a failure can be located, reproduced, and attributed. Those are the registry's three words for this chapter, and they are worth separating because a flow can be good at one and useless at the other two.

Each has a test. The test is what makes this a review rather than an opinion.

1. The three questions

Question	The test	Answered by
Can it be located?	Does the first line of the failure name the step?	29.4
Can it be reproduced?	Does a replay from the record alone give the same result?	29.5
Can it be attributed?	Does the log distinguish "did nothing" from "did its job"?	29.1, 29.3

Take last night's failure and try all three. The answers are rarely what the flow's authors expect.

2. Located

The test: does the first line of the failure name the step?

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

    rtl-path-wrong       STOPPED: gate after 'read' failed: no input file was readable
                                  phases completed: 1
    sdc-path-wrong       STOPPED: gate after 'constrain' failed: no constraints were applied
                                  phases completed: 3

gate after 'read' and gate after 'constrain' — the step, in the first line, with no log reading.

And the phases-completed count is a second, independent locator: 1 and 3 say which phase failed even if the message is truncated, buried in a 40,000-line log, or lost to a crash. Two signals about one fact, which is Chapter 24.7 §5's principle applied to a failure report rather than to a parser.

What to look for when the answer is no:

A failure reported at the end rather than at the step — Chapter 29.1 §6's locality problem, where "the report is empty" is three phases from the cause.
A message with no location, such as operation failed or a bare stack trace.
No count of what completed, so a truncated log is unlocatable.
A log written only on success, which is the one case you do not need it.

3. Reproduced

The test: does a replay driven only by the record give the same result?

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== replay, reading every argument from the record ===
    original : cells 4  sized 2  viol 2  FAIL
    replay   : cells 4  sized 2  viol 2  FAIL
    identical: yes

And before attempting it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== is it replayable? ===
    inputs 2  missing 0  changed 0

That second check is the one worth having, because "can this still be reproduced" is asked far more often than "reproduce it" — and it costs two digests.

What to look for when the answer is no:

No seed recorded for a randomised step, which makes the run unrepeatable in principle (Chapter 25.5 §1).
Input identity by timestamp rather than digest, which cannot answer the question within one second (Chapter 29.2 §2).
No tool version, so a replay that disagrees cannot be explained.
A record stored away from the results, which gets separated from what it describes.
A record nobody has ever replayed from. This is the common case: the record exists, looks complete, and has never been exercised. An untested record is an untested claim.

4. Attributed

The test: can the log tell "this step did nothing" from "this step did its job"?

This is the hardest of the three and the one that fails most often, because the two look identical.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

    typo-in-pattern  a query that matched nothing                 cells 4  sized 0  viol 2  FAIL

That row is Chapter 29.1 §4's case, and it is the sharpest possible illustration: the verdict is unchanged. FAIL with two violations, exactly as the correct run. The ECO did nothing and the only evidence is sized 0 in a column nobody reads.

And the session that exposes the related case:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

% p
  1. read 1 of 1 file(s)
  2. elaborated 4 cell(s)
  3. constraints applied: 0
  4. optimised 4 cell(s)
  5. reported 0 violation(s)
5 phase(s)

constraints applied: 0, between two correct lines, reading as ordinary progress.

What to look for when the answer is no:

A step that logs "done" rather than a count. "Applied constraints" cannot be distinguished from applying none; "constraints applied: 0" can.
A count with no denominator. read 0 is ambiguous between a wrong path and an empty list; read 0 of 1 is not (Chapter 29.4 §2).
No assertion that the step's effect took hold — the return value and the resulting state are different claims (Chapter 29.1 §5).
A verdict computed from something the step does not affect, which is why the typo case goes unnoticed.

5. The three are independent, and that is the finding

A flow can answer one of the three questions well and be expensive to work with anyway. The combination is what matters, and the cost of each gap is specific.

Can locate	Can reproduce	Can attribute	What every failure costs you
yes	yes	yes	read one line, fix, re-run
yes	yes	no	a live debug session to find what silently did nothing
yes	no	yes	you know the step and cannot re-run it — debug in production
no	yes	yes	bisection: re-run with phases removed until it changes
yes	no	no	the step name, and nothing you can do with it
no	no	no	argument, and eventually a decision taken on a number nobody trusts

The last row is not rhetorical. Chapter 29.2 §7 named its real cost: a review spends an hour discussing a violation that was fixed three weeks ago, because nothing in the report says which netlist it describes.

And the middle rows are the common ones. Most flows locate reasonably well — a stack trace at least names a file — and most cannot reproduce or attribute at all. That is why the usual experience of debugging a flow is a live session with a senior engineer who remembers what the step is supposed to do, which is exactly the knowledge that should have been in the log.

6. What debuggability is not a property of

Being honest about the limit, because a review of the flow is only half of it.

Debuggability is a property of the flow and of what was kept. A perfectly instrumented flow whose run directory was deleted by a cleanup job is not debuggable. The retention policy is part of the answer, and it is usually owned by somebody who has never read the flow.

It is not a property of the code alone. Two of the three tests — reproduced and attributed — are run against a failure, not against a script. You cannot assess them by reading, which is why this chapter is three tests rather than a code review.

And it degrades silently. A flow that was debuggable last year stops being so when a step is added without a gate, when a record gains a field nobody populates, or when the log format changes and the count disappears. Nothing fails when debuggability is lost — which makes it exactly the class of problem Chapter 29.1 is about, applied to the flow's own instrumentation.

7. The review

Does a failure name its step in the first line?
Is there a count of phases completed, independent of the message?
Is the log written on failure, with the full phase sequence?
Does every step log a count with its denominator, not "done"?
Is there an assertion that each step's effect took hold, separate from its return value?
Is the verdict computed from something every step affects — or can a step do nothing without changing it?
Is the seed recorded for anything randomised?
Is input identity a digest, not a timestamp?
Is the tool version recorded?
Does the record live beside the results?
Has anybody ever replayed from the record?
Can a stage be entered on its own for interactive reproduction (Chapter 29.3 §6)?
Does the exit status distinguish a broken flow from a failing design (Chapter 29.4 §6)?
How long are run directories kept?
When was the last deliberate failure?

That last question closes Module 29. Everything in this module has been one idea in different forms: a flow's value is bounded by what it can tell you about itself, and the night it fails is the wrong time to discover the answer is nothing.

8. Exercises

Exercise 1. What are the three questions, and why separate them?

Exercise 2. Give the test for each.

Exercise 3. What is the second, independent locator in 29.4's output, and why have two?

Exercise 4. Why is replayable more valuable day to day than the replay?

Exercise 5. What is the common failure of a run record that looks fine?

Exercise 6. Why is "attributed" the hardest of the three?

Exercise 7. The typo case kept the verdict at FAIL. Why does that make it the worst case for attribution?

Exercise 8. Why is a count without a denominator insufficient?

Exercise 9. A flow can locate and reproduce but not attribute. What does every failure then cost?

Exercise 10. A flow can locate and attribute but not reproduce. What does it cost?

Exercise 11. Name two things debuggability depends on that are not in the flow's code.

Exercise 12. Why does debuggability degrade silently, and what is the defence?

Answers.

1. Can a failure be located, reproduced, and attributed. They are separate because a flow can be good at one and useless at the other two, and the gaps cost different things.

2. Located: does the first line name the step. Reproduced: does a replay from the record alone give the same result. Attributed: can the log distinguish "this step did nothing" from "this step did its job".

3. The phases-completed count — 1 versus 3. Two signals about one fact, so a truncated, buried or lost message still leaves the failure locatable.

4. Because "can this still be reproduced" is asked far more often than "reproduce it", and it costs two digests rather than a run.

5. That nobody has ever replayed from it. It exists, looks complete, and is an untested claim.

6. Because "did nothing" and "did its job" look identical in a log that reports completion rather than effect.

7. Because the verdict is unchanged — FAIL with two violations, exactly as a correct run — so there is nothing anomalous to notice. The only evidence is sized 0.

8. Because read 0 cannot distinguish a wrong path from an empty input list, and those have different fixes. read 0 of 1 can.

9. A live debug session to find what silently did nothing — which needs somebody who remembers what each step is supposed to do.

10. You know which step failed and cannot re-run it, so the debugging happens in production.

11. The retention policy for run directories, and the fact that two of the three tests are run against a failure rather than against a script.

12. Because nothing fails when it is lost — a step added without a gate, a record field nobody populates, a log format change that drops a count. The defence is to break something deliberately and periodically, and check all three tests still pass.

Remember.

Three questions: located, reproduced, attributed — and each has a test you can run on last night's failure.
Located: the first line names the step, plus a phases-completed count as an independent second signal.
Reproduced: a replay from the record alone gives the same result — and replayable answers the question that is actually asked.
The common record failure is that nobody has ever replayed from it.
Attributed is the hardest, because "did nothing" and "did its job" look identical unless the log reports an effect.
Log counts with denominators, not "done". read 0 of 1 and read 0 of 0 are different failures.
Assert that the effect took hold, separately from the return value.
Check whether a step can do nothing without changing the verdict. The typo case is the one nobody finds.
The three are independent, and each gap has a specific cost — a live session, debugging in production, or bisection.
Debuggability depends on retention too, and on a failure to test against.
It degrades silently, which makes it the same class of problem as the rest of this module.
Break something on purpose, periodically. A flow that has not failed recently has untested error paths.

Where this fits

Part of the Tcl curriculum.