Memory Debugging Tools

Part I · Module 25 · Chapter 25.7 · Memory Debugging Tools

1. What Is Available, and What Is Not

So the sanitizer route to leak detection, which Chapter 25.6 §1 inventoried, is closed here, and the platform's own allocator instrumentation is the route that works. On Linux, -fsanitize=address with detect_leaks=1 or -fsanitize=leak does this job and valgrind's memcheck does it too; the method in §§3–5 transfers directly, because every one of those tools answers the same three questions.

2. The Model

A long-running model that records a trace per transaction and keeps one in a hundred:

struct Trace { int id; double samples[8]; };
static std::vector<Trace*> kept;

static void onTransaction(int i)
{
    Trace* t = new Trace{i,{}};            // recorded...
    if (i % 100 == 0) kept.push_back(t);   // ...but only 1 in 100 is kept
}

pid … transactions 1000  kept 10  sizeof(Trace) 72

Nine hundred and ninety allocations are never freed and never referenced again. The program is correct in every other respect: it produces the right output, exits 0, and in a thousand-transaction test nobody notices 79 kilobytes. At ten million transactions it is 790 megabytes, which is the shape of the real failure — a model that works in a short run and is killed by the operating system overnight.

3. Getting Permission to Look

Before any report, two obstacles, and both produce output that looks like a tool failure rather than a permissions problem.

The process must be debuggable. Running leaks against an ordinary binary gives:

Process … is not debuggable. Due to security restrictions, leaks can only show or save contents of readonly memory of restricted processes.

— followed by a long list of loaded libraries, which is easy to mistake for a report. Two routes work, and both were verified:

# 1. attach to a live process
MallocStackLogging=1 ./build/model &
leaks $!

# 2. sign the binary with the get-task-allow entitlement, then use --atExit
codesign -s - -f --entitlements ent.plist ./build/model
MallocStackLogging=1 leaks --atExit -- ./build/model

The entitlement file is three lines and grants com.apple.security.get-task-allow. Route 2 is better for a scripted regression because it needs no timing; route 1 is better for a model you want to inspect at a chosen moment, and it is the only option for a process you did not build.

The allocator must be recording. MallocStackLogging=1 is what makes the allocator keep a stack for every block. Without it leaks still counts the leaked blocks, and it cannot tell you where any of them came from — which is the entire value. Set it, and remember it costs memory and time, so it belongs in the leak-hunting run rather than every run.

4. The Report

Process …: 990 leaks for 79200 total leaked bytes.
STACK OF 990 INSTANCES OF 'ROOT LEAK: <malloc in onTransaction(int)>':

Two properties make that immediately actionable.

It groups. Nine hundred and ninety leaks, one stack, one line of output. A tool that printed 990 separate leaks would be useless; the grouping is what turns a leak report into a list of sites, usually a handful even in a badly leaking program.

It names the allocating function — onTransaction(int), demangled. That is the answer to "where did this come from", and for a leak it is usually enough, because a leak's cause is the allocation that had no owner.

Then the arithmetic, which does not reconcile the obvious way:

The report is 11% larger than the program requested, because the allocator rounds each 72-byte request into an 80-byte bucket and reports what it reserved. Chapter 23.6 §2's point about sizeof applies one level further out: the object is 72 bytes, the allocation is 80, and the gap is the allocator's quantisation rather than padding in your struct.

The practical consequence is a rule: do not try to match a leak total against your own sizeof arithmetic. Match the count — 990 blocks against 990 unfreed allocations — which is exact and which is what identifies the defect.

5. Reading a Trace That Ends Inside a Library

This is the frozen purpose's second half. malloc_history on one of the leaked addresses:

ALLOC 0xADDR-0xADDR [size=80]:  0xADDR (dyld) start | 0xADDR (longrun) main  longrun.cpp:15 | 0xADDR (longrun) onTransaction(int)  longrun.cpp:10 | 0xADDR (libc++abi.dylib) operator new(unsigned long) | 0xADDR (libsystem_malloc.dylib) _malloc | 0xADDR (libsystem_malloc.dylib) _malloc_zone_malloc_instrumented_or_legacy

Read it from the wrong end. The trace is printed outermost-first, so it runs dyld start → main → onTransaction → operator new → _malloc → the allocator's internals. The frames people find discouraging — three of the six are library code — are the innermost ones, and they are the same three in every allocation in the program. They carry no information at all.

The method is therefore mechanical:

Find the deepest frame that is in your own binary. Here (longrun) onTransaction(int) longrun.cpp:10. Everything below it is the allocator doing its job; everything above it is how you got there. That one frame is the answer.

Use the frames above it only if the site is ambiguous. onTransaction is called from one place, so main longrun.cpp:15 adds nothing. If the allocating function were a shared helper called from forty places, the caller chain is what distinguishes the leaking call site from the thirty-nine that are fine.

Ignore the innermost library frames entirely. operator new, _malloc and the zone functions are present in every trace and are never the bug.

That generalises past leaks to any trace, including Chapter 25.6 §4's sanitizer reports and a crash backtrace. A stack trace whose top frames are all library code is not an unhelpful trace; it is a trace you are reading from the wrong end. And when a binary has been stripped so there are no names, atos -o ./model <address> recovers the symbol and line from the debug information — which is why Chapter 24.7 §6 keeps -g in the release configuration.

6. The Other Tools, and What Each Is For

vmmap shows the process's address space by region, with the malloc zones broken out:

MALLOC_TINY   <range>  [ size  resident  dirty  swap] rw-/rwx SM=PRV  DefaultMallocZone_…
MALLOC_SMALL  <range>  [ size  resident  dirty  swap] rw-/rwx SM=PRV  DefaultMallocZone_…

Use it for the question leaks cannot answer: is the growth even in the heap? A model whose footprint grows without leaks reporting anything is usually growing in a container that is still reachable — which is not a leak, it is unbounded retention, and §7 is the distinction. vmmap's totals also separate your heap from memory-mapped files and from the runtime's own allocations.

heap gives a histogram of live blocks by class name, which answers "what is the memory made of". For a C++ model the useful rows are usually the container node types, and a count that matches your expected object count is a quick sanity check.

atos turns an address into a symbol and a line, for traces that lost their names.

dtrace is present and is largely unusable under System Integrity Protection, which §8 covers alongside the profiler.

7. A Leak Is Not the Only Way to Grow

Worth separating, because leaks reports nothing for the second case and the symptom is identical.

A leak is memory that is allocated and unreachable. leaks finds it.

Unbounded retention is memory that is allocated and still reachable — a log vector that is never trimmed, a cache with no eviction, a map keyed by transaction id that grows for the life of the run. Every block has an owner, nothing is lost, and the process still dies. leaks reports zero, correctly.

The second is more common in simulation models than the first, because the first is largely prevented by Chapter 14's unique_ptr while the second is a design decision nobody revisited. Chapter 22.9 §3 measured the arithmetic — per-cycle logging at a hundred million cycles produces 9 GB — and Chapter 22.9 §4's ring buffer is the standard fix: a bounded structure that keeps the last N entries and discards the rest.

So when a model's footprint grows: run leaks first, because it is cheap and definitive. If it reports nothing, the question changes from "what did I forget to free" to "what is still holding this, and should it be", and heap plus a count of your own containers is the next measurement.

8. The Profiler Is Not Available Here

Stated in this chapter because it belongs with the platform inventory, and measured in Chapter 25.8 §1: sample attaches to a process and produces a report whose call graph is empty, because System Integrity Protection prevents it reading the stacks. instruments is not installed, and perf and gprof are Linux tools that are absent.

Chapter 25.8 therefore builds a sampling profiler into the program itself, which works because a process may always sample its own stacks. That is a real technique rather than a workaround, and it is the honest alternative to a chapter that showed invented profiler output.

9. What to Prevent Rather Than Find

The tools in this chapter find leaks. Chapter 14 removes them:

// leaks: no owner
Trace* t = new Trace{i,{}};
if (i % 100 == 0) kept.push_back(t);

// does not leak: the owner is explicit, and the un-kept ones die at the end of scope
auto t = std::make_unique<Trace>(i);
if (i % 100 == 0) kept.push_back(std::move(t));

The second version cannot leak: if the unique_ptr is not moved into kept, it is destroyed when onTransaction returns. There is no delete to forget and no path that skips it, which is Chapter 22.4 §2's measured guarantee.

The three rules that between them remove almost all of this class:

Every heap object has a named owner — unique_ptr, shared_ptr, or a container by value. A raw new whose result is not immediately given to an owner is the defect.

Every container that grows per event has a bound. A ring buffer, an eviction policy, or a periodic trim. Chapter 22.9 §4 is the mechanism.

Run leaks on a short run, in the regression. It is cheap, it is definitive, and it finds the defect while it is still 79 kilobytes.

10. The Table

11. What to Actually Do

• Run leaks on a short run as part of the regression. It finds the defect at 79 KB rather than 790 MB.
• Set MallocStackLogging=1, or the report can count but not tell you where.
• Give the process permission to be inspected — attach to a live pid, or codesign with get-task-allow and use --atExit. Without it you get a library list that looks like a report.
• Act on the block count, not the byte total. 990 blocks mapped exactly onto 990 unfreed allocations; the bytes were 11% out because of allocator buckets.
• Read a stack trace from the wrong end: the deepest frame in your own binary is the answer, and the library frames below it are in every trace.
• Keep -g in release so atos can recover symbols.
• When the footprint grows and leaks is clean, look for retention — ask what still holds the memory and whether it should.
• Bound every per-event container with a ring buffer or an eviction policy.
• Give every heap object an owner at the point of allocation, which removes the class rather than detecting it.
• Know your platform's inventory before planning a CI matrix — the sanitizer route exists on Linux and not here.

12. Mistakes Worth Making Once

Reading the library list as a leak report. The real line is is not debuggable.

Forgetting MallocStackLogging=1. A count with no site, which is the half you cannot act on.

Trying to reconcile the byte total with sizeof. 79,200 against 71,280, and the gap is the allocator's.

Reading a trace top-down and giving up at _malloc. The informative frame is further out.

Expecting ASan to find leaks on this platform. It says so explicitly.

Concluding there is no problem because leaks is clean. Unbounded retention is reachable memory and reports nothing.

Waiting for the overnight run to fail. The leak was findable in a one-second run.

A raw new whose result is not handed to an owner on the next line.

A stripped release binary. No symbols, and atos has nothing to work from.

13. Interview Relevance

The frozen curriculum marks this chapter High, and the useful question is "how would you find a leak in a long-running model?"

The answer that lands starts by separating the two growth modes, because most candidates conflate them: a leak is allocated and unreachable and a tool finds it; unbounded retention is allocated and still reachable, reports nothing, and is more common in simulation code. Then the method — run the leak tool on a short run, because the defect is findable at kilobytes and fatal at megabytes, and the run that shows the symptom is the one you did not instrument.

The detail that marks experience is reading the trace: a stack whose frames are mostly library code is read from the innermost end outward, and the deepest frame in your own binary is the answer, with the caller chain mattering only when the allocating function has many call sites.

And the figure worth quoting is the discrepancy: leaks reported 79,200 bytes where the program had requested 71,280, because the allocator reports its 80-byte buckets — so you act on the block count, which was exactly 990 and mapped onto the defect in one step.

14. What You Should Be Able to Say Now

• That -fsanitize=leak is rejected and ASan's detect_leaks is unsupported on arm64 macOS, so leak detection here is the platform's allocator instrumentation.
• That leaks needs the process to be debuggable — attach to a live pid or codesign with get-task-allow — and that otherwise it prints a library list that resembles a report.
• That MallocStackLogging=1 is what makes the allocation stacks available.
• That the report groups — 990 leaks under one stack — and names the allocating function.
• That the byte total does not reconcile with sizeof arithmetic: 79,200 reported against 71,280 requested, because of an 80-byte bucket for a 72-byte object — so the count is the figure to act on.
• How to read a library-heavy trace: find the deepest frame in your own binary; the operator new / _malloc frames are in every trace and carry nothing.
• That atos recovers symbols from addresses, which is why -g stays in release.
• That a leak has no incorrect behaviour, which is why no test can see it and a tool must, and why it is only visible at a scale where it already hurts.
• The difference between a leak and unbounded retention, and that leaks correctly reports nothing for the second.
• That vmmap and heap answer the questions leaks cannot, and that the profiler is blocked by SIP.

15. Exercises

Exercise 1 — permission. Run leaks against an unsigned binary and read the whole output. Find the is not debuggable line. Then make it work both ways — live attach, and codesign plus --atExit.

Exercise 2 — the report. Reproduce §2's model and confirm the count is exactly transactions - kept. Then compute the requested bytes from sizeof and explain the discrepancy with the reported total.

Exercise 3 — the stack. Run malloc_history on one leaked address. Identify the deepest frame in your own binary, and the frames that would appear in every allocation in the program.

Exercise 4 — without the stacks. Run leaks without MallocStackLogging=1 and compare what you get. Say which half of the report is missing and why it is the half that matters.

Exercise 5 — retention, not a leak. Change the model to keep every trace in kept. Confirm that leaks now reports nothing and that the footprint still grows. Then bound it with a ring buffer and measure the footprint again.

Exercise 6 — remove the class. Convert the allocation to make_unique and confirm leaks reports zero without any delete in the source.

16. Summary

• Leak detection here is not a sanitizer. -fsanitize=leak is rejected on arm64 macOS and ASan reports detect_leaks is not supported on this platform, so the tools are leaks, malloc_history, vmmap and heap. On Linux the sanitizer route works and the method below is unchanged.
• Two obstacles come before any report. The process must be debuggable — attach to a live pid, or codesign with get-task-allow — or leaks prints is not debuggable and a library list. And MallocStackLogging=1 is what records the allocation stacks; without it you get a count with no site.
• The report is grouped and named: 990 leaks for 79200 total leaked bytes under one STACK OF 990 INSTANCES OF 'ROOT LEAK: <malloc in onTransaction(int)>'.
• The byte total does not reconcile with your arithmetic. 990 × sizeof(Trace) 72 = 71,280 requested against 79,200 reported — 990 × 80, the allocator's bucket. Act on the count, which was exact and mapped straight onto the defect.
• Read a library-heavy trace from the innermost end outward. The deepest frame in your own binary — onTransaction(int) longrun.cpp:10 — is the answer; operator new and _malloc appear in every trace. atos recovers symbols when names are gone, which is why -g stays in release.
• A leak has no incorrect behaviour to observe, so no test can detect it and the property is only visible at a scale where it already hurts — which is why you run the tool on a short run rather than waiting for the long one to fail.
• A leak is not the only way to grow. Unbounded retention is reachable memory, reports nothing, and is the commoner defect in simulation models; vmmap and heap are the next measurements, and Chapter 22.9 §4's ring buffer is the fix.
• Prevention beats detection: every heap object gets an owner at the point of allocation, and every per-event container gets a bound.

Chapter 25.8 turns from memory to time, and has to solve the same platform problem — the sampling profiler on this machine returns an empty call graph.