Debugging with a Method, Not a Reflex

Part I · Module 25 · Chapter 25.5 · Debugging with a Method, Not a Reflex

1. Why Re-reading Fails

A bug is a disagreement between what the code does and what you believe it does. Re-reading the source asks you to find that disagreement by inspecting one side of it — and you will inspect it using the same belief that produced the bug. That is not a lack of care; it is the structure of the situation. The belief is the error, and the belief is also the instrument.

A method works because it asks the program instead. Every question you put to a running program is answered by the thing that is actually true, so the answer is independent of your model, and a sequence of such answers converges whether or not you have the insight.

The method, in one line each:

1. Reproduce it reliably, smallest input first.
2. State a falsifiable hypothesis about where the state first becomes wrong.
3. Ask the program, at that point, for the value that decides it.
4. Narrow — the answer moves the boundary; repeat.
5. Fix the cause, then confirm the symptom is gone and understand why.

The rest of this chapter is one bug through those five steps, with the commands.

2. The Bug

A decoder for bus transactions, and a count of error responses:

static int decodeResp(std::uint32_t raw)
{
    return (int)((raw >> 1) & 0x3);     // the defect
}
static Txn decode(std::uint32_t word)
{
    Txn t{};
    t.addr   = word & 0xFFFFF000u;
    t.master = (int)((word >> 8) & 0xF);
    t.resp   = decodeResp(word);
    return t;
}
static int countErrors(const std::vector<std::uint32_t>& words)
{
    int n = 0;
    for (std::uint32_t w : words) { const Txn t = decode(w); if (t.resp != 0) ++n; }
    return n;
}

Four inputs encoding OKAY, SLVERR, DECERR and OKAY. The symptom:

  errors counted: 1   (expected 2: SLVERR and DECERR)
    word 0x40001000 -> addr 0x40001000 master 0 resp 0
    word 0x40001001 -> addr 0x40001000 master 0 resp 0
    word 0x40001002 -> addr 0x40001000 master 0 resp 1
    word 0x40002000 -> addr 0x40002000 master 0 resp 0

Step 1 is already done by that dump, and it is worth noticing how much it bought. The count being wrong tells you almost nothing; the per-word dump tells you which word is misdecoded — 0x40001001, whose low bits are 01, reports resp 0. One input, reproducible, and every subsequent step is cheaper for it. Printing the intermediate values is not a lesser technique than a debugger; it is step 1, and skipping it is why debugging sessions wander.

3. A Hypothesis That Can Be Wrong

Step 2. The hypothesis has to name a location and a claim, so that a single observation can refute it. Here:

"decodeResp returns the wrong value for raw = 0x40001001."

That is falsifiable: look at what it returns for that input. Compare it with the vaguer "something is wrong with the decoding", which no observation can settle.

Where the hypothesis comes from matters less than its shape. Here it comes from the dump — the field that is wrong is resp, and one function computes resp. In a less obvious case, binary search over the pipeline supplies it: check the state halfway along, and whichever half is wrong is the next hypothesis. Chapter 25.1 §7's lesson applies in the other direction too — the value of a hypothesis is that it can be refuted, and a refuted hypothesis is progress because it removes a region.

4. Asking the Program

Step 3. lldb is present on this machine; gdb and valgrind are not, so every transcript here is lldb. The commands translate directly — b for break, p for print, bt for backtrace — and the method is identical.

The first useful command is a conditional breakpoint, so the program stops only on the input you care about:

breakpoint set -f bug.cpp -l 12 -c "raw == 0x40001001"

Without the condition you stop four times and step through three uninteresting ones. With it, the program runs to exactly the failing case. On a loop over ten thousand transactions this is the difference between a method and an afternoon.

Then bt, which answers "how did I get here":

    frame #0: 0x0000000100002130 bug`decodeResp(raw=1073745921) at bug.cpp:12:19
  * frame #0: 0x0000000100002130 bug`decodeResp(raw=1073745921) at bug.cpp:12:19
    frame #1: 0x0000000100000924 bug`decode(word=1073745921) at bug.cpp:20:16
    frame #2: 0x0000000100000780 bug`countErrors(words=size=4) at bug.cpp:27:51
    frame #3: 0x0000000100000564 bug`main at bug.cpp:35:77
    frame #4: 0x0000000183883e00 dyld`start + 6992

The symptom was in main's output and the fault is three frames down. The backtrace prints the argument values at each level — raw=1073745921 is 0x40001001 — which also confirms the value arrived intact, eliminating the whole class of "something corrupted it on the way in".

5. The Step That Decides It

This is the technique worth taking from the chapter. At the fault point, evaluate the expression the code computes and the expression you believe it should compute, side by side:

(int) $0 = 0
(int) $1 = 0
(int) $2 = 1
(int) $3 = 0

The first pair is the breakpoint's first hit, on word 0x40001000 — a genuine OKAY. raw & 0x3 is 0 and (raw >> 1) & 0x3 is 0. They agree, which proves nothing. That is an important half of the measurement: on a passing input the correct and incorrect expressions are indistinguishable, which is exactly why the bug survived testing.

The second pair is the failing input, 0x40001001. raw & 0x3 is 1 — the correct answer. (raw >> 1) & 0x3 is 0 — what the code computes. The expressions disagree, and the disagreement is the defect: it names the wrong operator, at the right line, with no source re-reading at all.

In lldb:

expression (int)(raw & 0x3)
expression (int)((raw >> 1) & 0x3)

Three properties make this better than stepping.

It is a direct test of the hypothesis, so it cannot waste a step. Either the expressions agree — hypothesis refuted, move on — or they disagree and you are done.

It does not require the fix to be in the program. You are evaluating a candidate expression in the debugger, which is much cheaper than editing, rebuilding and re-running, and on a slow build it is the difference between minutes and a coffee break.

It works on the data the program actually has, so it accounts for the input you would not have thought to construct.

6. Narrowing When You Have No Hypothesis

Step 4. The case above had an obvious candidate. When nothing is obvious, three techniques supply one, in increasing cost.

Bisect the data. Halve the input until the smallest failing case remains. A one-transaction reproduction removes every question about accumulated state, and it is often enough on its own — this chapter's four-word input is already that.

Bisect the pipeline. Check the state at the midpoint of the transformation. Here: is word correct on entry to decode? Is t correct on return? Whichever half contains the first wrong value is the next hypothesis. Two or three iterations localise a fault in a long chain, and each iteration needs no insight.

Bisect history. If it worked last week, git bisect finds the commit mechanically. The prerequisite is a script that exits non-zero on the failure — which is Chapter 23.8 §6's contract, and the reason that chapter insisted the exit status be the verdict.

And a watchpoint, for the case where a value changes and you do not know what wrote it:

watchpoint set variable n

lldb reports the old and new value and the frame that did it. That is the right tool for Chapter 25.3's out-of-bounds corruption, where the write goes through a different name than the variable it damages — re-reading the corrupted variable's code cannot find it, and the watchpoint names the instruction.

7. Commands Worth Knowing

A small set covers almost everything:

The last row is worth more than it looks. A batch invocation is reproducible, scriptable and quotable, which turns a debugging session into something you can put in a bug report or re-run after a change. Every transcript in this chapter came from one.

Two build requirements: compile with -g, and prefer -O0 for the session. Chapter 24.7 §4 is why — at -O2 variables live in registers and are reported as optimised out, and inlined frames are absent from the backtrace, so bt stops matching the source. If the bug only occurs at -O2, -Og -g is the compromise, and Chapter 25.1 §3 is the reason a bug can be genuinely optimisation-specific.

8. The Last Step Is Not the Fix

Step 5. Changing (raw >> 1) & 0x3 to raw & 0x3 makes the count read 2. That is necessary and it is not the end.

Confirm you understand why the fix works, in a sentence. "The response occupies bits 1..0, and the shift moved the field one place too far, so SLVERR (01) read as 0 and DECERR (10) read as 1." A fix you cannot explain is a fix that might be a coincidence, and the next edit will undo it.

Add the test that would have caught it, which §7's callout identifies exactly: a word whose low two bits are 01. That is a stronger position than "add a test" — you know which input discriminates, because you measured the two expressions agreeing everywhere else.

Look for siblings. A field-extraction mistake in one decoder is evidence about the others. master in the same function is (word >> 8) & 0xF — is 8 the right shift, and is 0xF the right mask? The same bug shape, in code written at the same time by the same person, is the cheapest place to find the second defect.

And check whether the symptom could have been silent. Here the count was wrong, which somebody noticed. If DECERR had decoded to 0 instead of 1, the count would have been 0 and the run would have looked clean — which is Chapter 23.7 §2's measured disaster, a gate passing a failing run. A defect that could have been silent deserves a loud check, not just a fix.

9. What to Actually Do

• Reproduce it smallest-first, and print the intermediate values. The per-word dump in §2 located the failing input before any debugger was opened.
• State a hypothesis that one observation can refute, naming a location and a claim.
• Use a conditional breakpoint so the program stops only on the failing input.
• Read bt before anything else. The symptom was three frames from the cause, and the backtrace showed the argument arriving intact.
• Evaluate both candidate expressions at the fault point. The disagreement names the defect; the agreement tells you which inputs can detect it.
• Bisect when you have no hypothesis — data, then pipeline, then history with git bisect.
• Use a watchpoint when a value changes and you do not know what wrote it, which is the only workable tool for Chapter 25.3's corruption.
• Compile -g, debug at -O0, and use -Og when the bug needs optimisation to appear.
• Use batch mode so the session is reproducible and quotable.
• Finish properly: explain the fix, add the input that discriminated, look for siblings, and ask whether the symptom could have been silent.

10. Mistakes Worth Making Once

Re-reading the function. You read it the same way you did when you wrote it.

Stepping from main. Three frames of uninteresting correctness before the fault.

An unconditional breakpoint in a loop. Four stops here; ten thousand on real input.

A hypothesis nothing can refute. "Something's wrong with the decoding" cannot be tested.

Collecting values without comparing them. Each one confirms a problem and discriminates nothing.

Debugging at -O2 by accident. Variables optimised out, frames missing, bt not matching the source.

Fixing without explaining. A fix you cannot state in a sentence may be a coincidence.

Stopping at "the symptom is gone". The test that would have caught it is now identifiable exactly — adding it is the cheapest it will ever be.

Not checking the sibling fields. The same mistake shape, written the same afternoon.

11. Interview Relevance

The frozen curriculum marks this chapter High, and it is usually asked as "walk me through how you'd debug this" — a question about method, where naming a sequence and sticking to it is most of the answer.

The sequence to give is reproduce smallest-first, state a falsifiable hypothesis, ask the program at that point, narrow by bisection when there is no hypothesis, and finish by explaining the fix and adding the discriminating test.

The technique worth volunteering, because few candidates mention it, is the comparison in §5: at the fault point, evaluate the expression the code computes and the expression you think it should compute, against the same data. One observation leaves three candidates — bad input, bad expression, bad expectation — and the comparison eliminates two of them in a single step.

The detail that shows real experience is what the agreeing pair means: on a passing input the correct and incorrect expressions give the same answer, which is both why the bug survived testing and a precise specification of the test that was missing.

12. What You Should Be Able to Say Now

• Why re-reading the source fails: the belief that produced the bug is also the instrument you are using to look for it.
• The five steps, and that each one narrows whether or not you have insight.
• That printing intermediate values is step 1 — the per-word dump located the failing input before any debugger ran.
• That a hypothesis must name a location and a claim so one observation can refute it.
• That a conditional breakpoint — -c "raw == 0x40001001" — is what makes a loop debuggable.
• That bt showed the fault three frames below the symptom and confirmed the argument arrived intact.
• The deciding step: evaluate both candidate expressions at the fault point. Measured, raw & 0x3 gave 1 and (raw >> 1) & 0x3 gave 0 on the failing input, and both gave 0 on a passing one.
• That the agreeing pair explains why the bug survived testing and specifies the missing test exactly.
• That watchpoints are the tool for a value changed by code that does not name it.
• That you debug at -O0 -g, and that batch mode makes a session reproducible.

13. Exercises

Exercise 1 — the session. Build §2's program and reproduce the symptom. Then reproduce the conditional breakpoint, the backtrace and the two expression evaluations.

Exercise 2 — the agreeing pair. Find every 32-bit word on which raw & 0x3 and (raw >> 1) & 0x3 agree. Then explain, in terms of that set, why a test suite of mostly-OKAY traffic cannot fail.

Exercise 3 — the discriminating test. Add the test the callout identifies, confirm it fails before the fix and passes after, and check whether it also catches a plausible neighbouring bug such as the wrong mask.

Exercise 4 — bisect the pipeline. Delete the per-word dump so you have only the wrong count. Localise the fault using only breakpoints and frame variable, and count how many steps it takes.

Exercise 5 — the watchpoint. Set a watchpoint on n in countErrors and record which frame increments it and when. Then do the same for Chapter 25.3's after member and confirm the writing instruction is in the array code.

Exercise 6 — the siblings. Check master's shift and mask in the same function against the field layout. Report whether the second defect exists.

14. Summary

• Re-reading the source fails structurally: the belief that caused the bug is the instrument you are searching with. A method asks the program, whose answers are independent of your model.
• Five steps: reproduce smallest-first, state a falsifiable hypothesis, ask the program at that point, narrow by bisection, and fix the cause — then explain it and add the test.
• Printing intermediate values is step 1, not a lesser technique. The per-word dump identified word 0x40001001 as the failing input before any debugger was opened.
• A conditional breakpoint — breakpoint set -f bug.cpp -l 12 -c "raw == 0x40001001" — stops only on the failing case, and bt showed the fault three frames below the symptom with the argument intact.
• The deciding step is a comparison, not an observation. At the fault point, raw & 0x3 evaluated to 1 and the code's (raw >> 1) & 0x3 to 0. One observation leaves three candidates — bad input, bad expression, bad expectation — and the comparison eliminates two at once.
• The agreeing pair is the other half of the finding. On a passing word both expressions gave 0, so correct and incorrect code are indistinguishable there — which is why the defect shipped, and which specifies the missing test precisely: a word whose low two bits are 01.
• Bisect when there is no hypothesis — data, pipeline, then git bisect, which needs a script whose exit status is the verdict.
• Watchpoints are the only workable tool when a value is changed by code that does not name it.
• Debug at -O0 -g, use -Og when the bug needs optimisation, and run lldb in batch mode so the session is reproducible.

Chapter 25.6 takes the tools that answer these questions without being asked, measures what each one catches, and finds that one of them reports a defect while letting the run succeed.