The 12-GiB Leak: Tracing an OOM Bug Through CUDA Memory Lifetimes in a Groth16 Proving Engine

Introduction

In the high-stakes world of Filecoin proof generation, every millisecond of GPU idle time is lost revenue. The cuzk SNARK proving engine had already undergone eight optimization phases—from parallel synthesis dispatch through dual-GPU-worker interlocks—each peeling away layers of inefficiency. Phase 9 targeted the PCIe transfer bottleneck: the 6 GiB of polynomial data (a, b, c) that had to be uploaded from host memory to the GPU for each partition of the Groth16 proof. The optimization was straightforward: pin the host pages with cudaHostRegister, allocate device buffers, and issue asynchronous cudaMemcpyAsync transfers on a dedicated stream with event-based synchronization, overlapping the upload with computation rather than blocking on synchronous transfers.

But when the assistant ran the first benchmark with the intended dual-worker configuration (gpu_workers_per_device=2), every single proof failed with a cascade of CUDA out-of-memory errors. The daemon logs told a fragmented story: prestage_setup=fallback err=2 (cudaErrorMemoryAllocation), panics in dev_ptr_t constructors, and a GPU left in an unrecoverable state. Message 2429 captures the moment of diagnosis—a deep, methodical tracing exercise where the assistant works backward from the error logs to reconstruct the exact sequence of allocations and frees that led to the OOM. This article examines that message in detail, exploring the reasoning process, the assumptions that proved incorrect, and the subtle memory-lifetime bug that was uncovered.

The Message in Context

[msg 2429] is an assistant message from deep within the Phase 9 implementation cycle of the cuzk SNARK proving engine. The conversation leading up to it spans dozens of messages across multiple optimization phases. By this point, the assistant has already:

The Opening Puzzle: Domain Size Mismatch

The message opens with a deceptively simple question: why does the log say domain=134217728 (2^27) when the assistant expected 2^26? The assistant traces through the lg2 function:

Same logic, different return type (uint32_t vs int). With n = 67108862 = 2^26 - 2: - n >>= 1 → 33554431 → ret = 1 - Continue... eventually ret = 25, n = 1, n >>= 1 → 0, loop exits. - Returns 25. Then 25 + 1 = 26. So domain should be 2^26 = 67108864, not 2^27. But the log says 134217728.

This is a critical moment. The assistant is checking whether a bug in the lg2 function itself could explain the OOM—if the domain size were computed incorrectly, the allocation sizes would be wrong. The lg2 function is a simple bit-counting loop that returns the floor of the base-2 logarithm. The assistant manually simulates the loop for n = 67108862 and confirms it returns 25, yielding domain size 2^26. So the function is correct.

But then comes the realization:

Wait — let me re-check. points_h.size() — what IS this value? Let me look at what npoints - 1 would be. If npoints = 67108865 (= 2^26 + 1), then npoints - 1 = 67108864 = 2^26, and lg2(2^26) = 26, domain = 2^27 = 134217728. That would explain the log!

The H SRS (Structured Reference String) for the Filecoin PoRep circuit has domain_size + 1 points. Since domain_size = 2^26 = 67108864, the H SRS has 67108865 points. The pre-staging code computes lg2(points_h.size() - 1) + 1, which becomes lg2(67108864) + 1 = 26 + 1 = 27. The domain size is 2^27 = 134217728 elements—not the 2^26 the assistant initially assumed.

This is the first major insight: the pre-staging code is allocating for a domain of 128 million elements, not 64 million. At 32 bytes per element (a BLS12-381 field element in fr_t), that means:

The Critical Insight: Memory Lifetime Bug

The assistant then reconstructs the exact execution flow for the dual-worker configuration. This is where the reasoning becomes most impressive—a careful simulation of the interleaved execution of two GPU workers competing for the same device.

With gpu_workers_per_device = 2, two separate calls to generate_groth16_proofs_c are interleaved via a static mutex (mtx_ptr). Worker 0 acquires the mutex first, pre-stages its 12 GiB of buffers, runs the NTT+MSM kernels, and eventually releases the mutex. Worker 1 waits at the mutex acquisition and then proceeds.

The assistant traces worker 0's execution step by step:

  1. Pin host pages (before mutex) — OK
  2. Acquire mutex
  3. Pre-stage: allocate d_a (4 GiB) + d_bc (8 GiB) + issue async uploads
  4. Launch per_gpu thread → NTT+MSM using prestaged buffers → batch_add → tail_MSMs
  5. Join per_gpu thread
  6. Release mutex
  7. Free d_bc (8 GiB), events, stream
  8. Unregister host Then worker 1's flow after acquiring the mutex:
  9. Acquire mutex
  10. Pre-stage: allocate d_a + d_bc — should succeed since worker 0 freed d_bc at step 7 But wait—the assistant catches a subtle issue:
Wait — but d_a in the pre-staged path is handed to gpu_ptr_t which is freed when the scope exits inside the per_gpu thread. So both d_a AND d_bc are freed before the mutex is released. Let me re-verify...

This is the moment of doubt. The assistant realizes it needs to check the actual code structure, not just the conceptual flow. It reads the file to verify the cleanup order.

The key finding comes from examining the code structure:

// Inside the per_gpu thread:
if (prestage_ok) {
    gpu_ptr_t<fr_t> d_a{d_a_prestaged};
    // ... use d_a ...
}  // d_a freed here (gpu_ptr_t destructor)

// After the per_gpu thread joins:
gpu_lock.unlock();
// Phase 9 cleanup:
if (d_bc_prestaged) { cudaFree(d_bc_prestaged); ... }

The d_a buffer is wrapped in a gpu_ptr_t RAII wrapper, so it gets freed when that scope exits—before the mutex is released. But d_bc is freed after the mutex is released, in the Phase 9 cleanup section.

This is the root cause of the OOM. When worker 1 acquires the mutex, worker 0's d_bc (8 GiB) is still allocated because it hasn't been freed yet. Worker 1 then tries to allocate its own d_a (4 GiB) + d_bc (8 GiB) = 12 GiB. Combined with worker 0's lingering d_bc (8 GiB), the total is 20 GiB on a 16 GiB GPU. OOM is inevitable.

The Assumptions That Failed

This diagnosis reveals several assumptions that proved incorrect:

Assumption 1: "Pre-staging inside the mutex serializes allocations." The assistant had previously moved the pre-staging allocation inside the mutex to prevent both workers from allocating simultaneously. While this serialized the start of allocation, it didn't account for the fact that one worker's buffers might still be alive when the next worker starts—because cleanup happened after the mutex release.

Assumption 2: "RAII wrappers handle all cleanup before scope exit." The gpu_ptr_t RAII wrapper for d_a worked correctly, freeing the buffer when the per_gpu thread scope exited. But d_bc was managed manually with raw cudaFree calls in the cleanup section after the mutex release. This asymmetry—one buffer RAII-managed, the other manually managed—created the lifetime gap.

Assumption 3: "The domain size is 2^26." The assistant initially expected a domain size of 2^26 (64M elements) based on the PoRep circuit parameters. But the H SRS has domain_size + 1 points, and the pre-staging code computes the domain from points_h.size(), yielding 2^27 (128M elements). This doubled the expected allocation from 6 GiB to 12 GiB.

Assumption 4: "The first partition's success means the approach works." The first partition succeeded with prestage_setup=ok and fast NTT+MSM times (840 ms). But this was a misleading signal—the OOM only manifested when the second worker tried to pre-stage while the first worker's d_bc was still alive.

The Thinking Process: A Model of Systematic Debugging

What makes [msg 2429] particularly valuable as a case study is the structure of the reasoning. The assistant doesn't jump to conclusions or apply random fixes. Instead, it works through the problem in layers:

Layer 1: Validate the math. Before assuming the allocation logic is wrong, the assistant manually simulates the lg2 function to confirm it produces the correct result. This eliminates one potential root cause.

Layer 2: Re-examine the input data. When the math checks out but the log contradicts expectations, the assistant re-examines the input—specifically, what is points_h.size()? This leads to the realization that the H SRS has domain_size + 1 points, not domain_size.

Layer 3: Calculate the actual allocation sizes. With the correct domain size (2^27), the assistant computes the memory footprint: 4 GiB for d_a, 8 GiB for d_bc, 12 GiB total.

Layer 4: Simulate the execution flow. The assistant traces through the exact sequence of operations for both workers, paying careful attention to the order of allocations, deallocations, mutex acquisitions, and releases.

Layer 5: Identify the asymmetry. The critical insight is that d_a is freed inside the per_gpu thread scope (before mutex release) while d_bc is freed after the mutex release. This creates a window where worker 1's allocation attempt sees worker 0's d_bc still occupying 8 GiB.

Layer 6: Verify by reading the code. Rather than continuing to reason from memory, the assistant reads the actual source file to confirm the cleanup order.

This layered approach—validate the math, check the inputs, compute the numbers, simulate the execution, identify the structural issue, verify against the code—is a textbook example of systematic debugging. Each layer builds on the previous one, and each eliminates a class of potential causes.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

CUDA memory model: Understanding pinned vs. pageable host memory, the distinction between synchronous cudaMalloc/cudaFree and asynchronous pool-based allocation, and the fact that cudaFree is a synchronous operation that blocks until the device has finished using the memory.

GPU architecture for dual-worker configurations: The concept of multiple GPU workers sharing a single device via a mutex, where each worker takes turns executing kernels while the other waits. This is the Phase 8 dual-GPU-worker interlock architecture.

Groth16 proof structure for Filecoin PoRep: The a/b/c polynomials that must be uploaded for each partition, the H SRS (Structured Reference String) that provides the domain points, and the relationship between domain size and polynomial length.

The lg2 function and its semantics: A simple bit-counting function that returns the floor of the base-2 logarithm. Understanding that lg2(n) + 1 gives the number of bits needed to represent n, which is used to compute the NTT domain size.

RAII patterns in C++ CUDA code: The use of gpu_ptr_t as a smart pointer that calls cudaFree in its destructor, versus raw cudaFree calls for manually managed buffers.

Output Knowledge Created

This message produces several important pieces of knowledge:

Root cause identification: The OOM is caused by d_bc being freed after the mutex release, creating a window where worker 1's allocation collides with worker 0's lingering buffer.

Specific fix: Move the d_bc cleanup (the cudaFree(d_bc_prestaged) call) to before the mutex release, ensuring all pre-staging buffers are freed before the next worker can acquire the mutex.

Domain size clarification: The pre-staging code allocates for a 2^27 domain (128M elements), not 2^26, because the H SRS has domain_size + 1 points. This means the pre-staging allocation is 12 GiB, not 6 GiB as might be naively expected.

Memory accounting for the dual-worker case: With two workers sharing a 16 GiB GPU, the peak memory usage is: worker 0's d_bc (8 GiB, still alive) + worker 1's d_a (4 GiB) + worker 1's d_bc (8 GiB) = 20 GiB. This exceeds the 16 GiB VRAM, explaining the OOM.

Design principle: When using a mutex to serialize GPU access across workers, all cleanup of GPU memory must happen before the mutex release, not after. Otherwise, the next worker inherits a fragmented memory landscape.

The Broader Significance

This message is a microcosm of the entire optimization project. The cuzk proving engine's journey from a naive single-worker implementation to a sophisticated multi-worker pipeline had progressively revealed deeper and deeper layers of complexity. Each optimization phase—from the initial parallel synthesis dispatch (Phase 7) through the dual-worker interlock (Phase 8) to the PCIe transfer optimization (Phase 9)—had uncovered new bottlenecks and new failure modes.

The Phase 9 OOM bug is particularly instructive because it represents a failure mode that only appears at the intersection of two optimizations: the dual-worker interlock (Phase 8) and the PCIe transfer optimization (Phase 9). Neither optimization alone would have caused this bug. The dual-worker interlock without pre-staging would have had both workers allocating and freeing buffers within the mutex scope, with no lingering allocations. The pre-staging optimization without the dual-worker interlock would have had only one worker at a time, so the cleanup-after-unlock issue wouldn't matter.

This is a classic systems integration problem: individually correct components can fail when combined, because their interactions create new states that neither component was designed to handle. The assistant's debugging approach—tracing the exact sequence of operations across both workers—is precisely the kind of cross-component reasoning needed to diagnose such issues.

Conclusion

Message 2429 captures a pivotal moment in the optimization of a high-performance GPU proving engine. What began as a puzzling OOM failure—with contradictory signals (first partition succeeds, subsequent partitions fail)—was systematically decomposed into a clear root cause: a memory lifetime mismatch where d_bc was freed after the mutex release, creating a window of double allocation for the next worker.

The assistant's reasoning process demonstrates several virtues of systematic debugging: validating assumptions about basic functions before moving to higher-level analysis, carefully simulating execution flows rather than guessing, and verifying against actual source code rather than reasoning from memory. The fix—moving the d_bc cleanup before the mutex release—is simple once the root cause is understood, but getting to that understanding required tracing through the mathematics of domain size calculation, the structure of the SRS, the CUDA memory allocation patterns, and the interleaved execution of two GPU workers.

For anyone working on high-performance GPU systems, this message offers a valuable case study in how subtle memory lifetime bugs can emerge at the intersection of concurrency and resource management—and how systematic reasoning can cut through the complexity to find the root cause.