Debugging a C Program

Part I · Module 13 · Chapter 13.9 · Debugging a C Program

1. The Problem With Rereading

Open the file, start at the top, read until you see it. Everyone does this, and in a short function it often works.

It fails for three reasons this curriculum has already measured, and all three are the normal case in C rather than the exception.

The fault is usually not where the symptom is. Chapter 9.11 §4 established the distance between cause and effect for an invalid pointer; Chapter 13.6 §2 measured that an out-of-range write lands on storage you cannot predict and that the layout changes with a compiler flag. Rereading the function that misbehaved means rereading code that is correct.

The source is not the program that ran. Chapter 13.2 §3 measured a null check present at -O0 and absent at -O1 and above. Reading the source tells you what you asked for, not what was built.

And the symptom may be no symptom at all. Here is a program whose fault produced nothing to investigate:

    static int average(const int *v, int n)
    {
        int t = 0;
        for (int i = 0; i < n; i++) { t += v[i]; }
        return t / n;                      /* n is 0 */
    }

sum_to(5) = 15
average of nothing = 0
  exit status 0

No crash. A plausible number. A success status. Nothing in that output suggests a bug, and rereading would have to start from the suspicion that something is wrong at all.

Rereading is a search with no way to narrow. Every line stays a candidate until you happen to notice the one that matters, which means the method's cost grows with the size of the file and its success depends on luck.

2. The Method

3. Step One: Reproduce It

A bug you cannot trigger on demand cannot be investigated. It can only be guessed at.

Everything else depends on this, because every later step needs to compare a change against a baseline, and you cannot compare against something that happens sometimes.

What to pin down:

The exact input. The file, the arguments, the sequence. Chapter 12.6 §4 measured that an argument can be an empty string, and Chapter 13.5 §5 measured a loop that misbehaves only for an empty string — input you did not think to record is often the input that matters.

The exact build. Flags, optimisation level, compiler version. Chapter 13.2 §5 listed six things that change a program containing undefined behaviour, and -O is one of them.

The smallest version that still fails. Cutting the program down is not a separate chore; it is step 3, started early.

If a bug reproduces only sometimes, the first task is to find what varies — uninitialised memory (Chapter 9.12), an allocation address (Chapter 9.10 §3), input ordering, timing. "Intermittent" is usually "deterministic on a variable I have not identified".

4. Step Two: Run the Tools Before Reading

This is the step the instinct skips, and the measurement is unambiguous.

The division-by-zero program from §1 gave no signal. Under a sanitizer:

$ cc -std=c17 -g -fsanitize=undefined d1.c -o d1u && ./d1u
d1.c:15:14: runtime error: division by zero
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior d1.c:15:14

File, line, column, and what happened — before any source had been read.

The second case is stronger, because it is the one rereading cannot solve. A structure with an 8-byte name field and an int limit; one function copies a 19-byte string into the name, and a different function later reads the limit:

    struct config { char name[8]; int limit; };

    static void set_name(struct config *c, const char *s)
    {
        strcpy(c->name, s);                 /* the bug */
    }

    static int use_limit(const struct config *c)
    {
        return c->limit * 2;                /* where the wrong value shows up */
    }

$ cc -std=c17 -g -Wall -Wextra -Wpedantic d2.c -o d2
0 warnings

$ cc -std=c17 -g -fsanitize=address d2.c -o d2a && ./d2a
==63349==ERROR: AddressSanitizer: stack-buffer-overflow
WRITE of size 19 at 0x00016bd8268c thread T0
Address 0x00016bd8268c is located in stack of thread T0 at offset 44 in frame
SUMMARY: AddressSanitizer: stack-buffer-overflow d2.c:8 in set_name

It named set_name — the function containing the bug — not use_limit, where a developer investigating a wrong limit would have started. The write's size, the storage it hit, and the line.

Run the sanitizers before you read anything. They cost a recompile, they frequently name the line, and — in the case that matters most — they point at the cause rather than the symptom. Chapter 13.10 covers which to use and what each sees.

5. Step Three: Bisect

When the tools do not hand you the line, narrow the region. The principle is the same one that makes binary search fast, applied to code:

Find a point where the state is still correct and a point where it is wrong, then halve the distance between them. Each test eliminates half the remaining candidates, so twenty tests cover a million lines.

The axis can be anything that divides:

Position in the code. Check an invariant halfway through the suspect region. Correct there means the fault is later; wrong means earlier.

Input. Halve the data until the failure stops. Chapter 13.5 §5's bug needed an empty string; chopping the input is how you find that.

Time. On the tenth iteration or the ten-thousandth? A counter and a condition answer it.

History. If it worked last month, bisect the commits. Most version-control systems automate this, and it is the same halving.

Configuration. Chapter 13.2 §5's list — optimisation level, compiler, flags. If it fails at -O2 and not -O0, that is not a mystery to solve later; it is a strong signal of undefined behaviour and the answer to "where do I look".

The thing that makes bisection work is that each step must give a definite answer. "It still seems wrong" is not a result. Before starting, decide what correct looks like at the midpoint — which is step 4.

6. Step Four: Observe What Is Actually There

The debugger's job, and the point at which the two tools diverge.

A debugger — lldb or gdb — runs your program under supervision. The commands worth knowing are few:

Two of those are worth more than the rest.

bt answers "how did I get here", which is the single most useful fact when a crash is in a function you did not expect to be running.

A watchpoint answers "what changed my variable" — and that is the direct tool for Chapter 13.6's problem. When a value is being corrupted by something unrelated, setting a watchpoint on it and running makes the program stop at the instruction that wrote it, whichever function that turns out to be.

Always build with -g. It adds debug information — line numbers, variable names, types — without changing the generated code. There is no reason to omit it, and -g -O2 is a legitimate combination for debugging a release build, with the caveat that optimised code makes stepping confusing because statements have been reordered or removed (Chapter 13.2 §3).

An honest note about this chapter: the environment these pages are measured in does not permit a debugger to attach — lldb reports "Not allowed to attach to process" — so there is no captured debugger session here. The commands above are listed; no transcript of them is shown, because none was produced. Every other measurement in this chapter was run.

And printing still works. A printf at the right place is a legitimate instrument, not a lesser one — it is sometimes the only option, in code that cannot be run under supervision at all. Two rules make it effective:

    fprintf(stderr, "at %s:%d  count=%zu limit=%zu\n", __FILE__, __LINE__, count, limit);

Print to stderr, because Chapter 12.4 §4 measured that stdout is buffered and a program that crashes may never flush it — the output you most need is the output you lose. And print the state, not a marker. "here 3" tells you control arrived; the values tell you whether it should have.

7. Step Five: One Hypothesis at a Time

The step that separates debugging from guessing.

A hypothesis must predict something you have not yet observed. State it, derive a prediction, test the prediction.

Hypothesis:  limit is being overwritten by the name copy.
Prediction:  a name of 7 characters or fewer leaves limit intact;
             8 or more changes it.
Test:        run with both.

If the prediction holds, the hypothesis survives and you know where to fix. If it fails, the hypothesis was wrong and you have learned something — which is why a prediction is required. A hypothesis that predicts nothing cannot be refuted and testing it teaches nothing either way.

Three failures of this step are worth naming because they are the common ones.

Changing several things at once. If the symptom goes away you do not know which change did it, and one of them may have hidden the bug rather than fixed it.

Changing code to see what happens. That is generating new programs, not investigating the existing one. Observe first; change only to test a stated prediction.

Stopping at the first thing that looks wrong. Code can contain several defects, and the one you found may not be the one causing the symptom. The test is whether fixing it makes the predicted difference — not whether it was wrong.

And the step after the fix, which is the one most often skipped:

Explain the whole symptom. If the fix works but you cannot say why the program printed that number, you have probably suppressed the symptom rather than removed the cause. Chapter 9.10 §4's measurement is the warning: memory bugs very often appear to work, so "it stopped happening" is weak evidence.

8. What You Should Be Able to Say Now

1. Rereading is a search with no way to narrow, and it fails for three measured reasons.
2. The fault is usually not where the symptom is (9.11 §4, 13.6 §2 — the layout even moved with a flag).
3. The source is not the program that ran — 13.2 §3's check was present at -O0 and gone at -O1.
4. Measured: a division by zero produced a plausible number and exit status 0 — no symptom to investigate.
5. The method: reproduce · run the tools · bisect · observe · one hypothesis. The order matters.
6. Reproducing means pinning the input, the build, and the smallest failing case. "Intermittent" usually means "deterministic on a variable you have not found".
7. Measured: UBSan named d1.c:15:14: division by zero where the program said nothing.
8. Measured: with 0 warnings, ASan named set_name — the cause — not use_limit, where the symptom would appear.
9. Bisect on any axis that divides: position, input, time, history, configuration — and each step must give a definite answer.
10. bt answers how you got here; a watchpoint answers what changed your variable. Always build with -g. (No debugger session is transcribed here — one could not be run.)
11. Print to stderr, because stdout is buffered (12.4 §4), and print state, not markers.
12. A hypothesis must make a prediction. Change one thing; explain the whole symptom; and treat disappears with a printf, differs at -O2, and appeared after an upgrade as undefined behaviour until proven otherwise.

9. Exercises

Exercise 1 — what does the tool say. For each, name the sanitizer you would run and what it would report.

• a program that crashes intermittently in free
• a total that is occasionally absurd
• a program that works at -O0 and fails at -O2
• a function that returns a different value on the second call with the same arguments

Exercise 2 — reproduce it. A colleague reports "it crashes sometimes on big files". List six things you would establish before investigating, and say which one Chapter 13.5 §5 suggests is most likely to be the trigger.

Exercise 3 — choose the axis. For each, say which bisection axis you would use and roughly how many steps it would take.

• a 4000-line file, fault somewhere in it
• a 2 MB input file, fault triggered by some record
• a loop of 1,000,000 iterations, wrong after some iteration
• a repository with 300 commits since it last worked

Exercise 4 — write the observation. Given the struct config bug in §4, write the single fprintf you would add and where, and say why stderr rather than stdout.

Exercise 5 — hypothesis and prediction. A program prints a total that is too large by exactly the value of the first element. State one hypothesis and the prediction that would test it.

Exercise 6 — the moving bug. A bug vanishes when a diagnostic printf is added. Say what this almost certainly means, what you would run next, and why "leave the printf in" is not a fix.

Exercise 7 — the watchpoint. A counter holds the wrong value and nothing in the obvious code path writes it. Describe, in debugger commands, how you would find what does, and which chapter explains why the culprit may be an unrelated function.

Exercise 8 — explain the whole symptom. You fix a bug and the crash stops, but you cannot explain why the program previously printed -294967296. Say why you should keep investigating, naming the chapter that measured a bug that "appears to work".

10. Summary

• Rereading cannot narrow, and fails because the fault is elsewhere, the source is not what ran, and sometimes there is no symptom — measured: division by zero, plausible output, exit status 0.
• Reproduce · tools · bisect · observe · hypothesis, in that order.
• Pin the input, the build, and the smallest failing case.
• Measured: UBSan gave d1.c:15:14: division by zero; ASan named set_name, the cause, with 0 compiler warnings.
• Bisect on position, input, time, history or configuration — every step must give a definite answer.
• bt for how you got here, a watchpoint for what changed a variable, always -g. No session is transcribed here; one could not be run.
• stderr, and print state rather than markers.
• One hypothesis, one prediction, one change — and explain the whole symptom.
• Vanishes with a printf · differs at -O2 · appeared after an upgrade = undefined behaviour.

Two chapters have now reached for the same instrument at the decisive moment. Chapter 13.1 measured sanitizers taking the detection rate from nine defects to eleven; this chapter used one to name a line a program had said nothing about, and another to point at the function that caused a corruption rather than the one that displayed it. They have been used throughout as instruments and never explained: what they actually do to a program, what each kind can and cannot see, what they cost, and the one limitation that matters more than all their capabilities — that a tool watching a running program reports only the faults your inputs actually reach. The next chapter is that account.