Diagnosing the Phantom OOM: How CUDA Memory Pool Asymmetry Nearly Derailed Phase 9

In the high-stakes world of Filecoin proof generation, every millisecond counts. The cuzk SNARK proving engine had been steadily optimized through eight phases, each targeting a specific bottleneck in the Groth16 proof pipeline. Phase 9 promised the biggest leap yet: PCIe transfer optimization that would eliminate GPU idle gaps by pre-staging polynomial data on the GPU and overlapping host-to-device transfers with computation. But when the dual-worker benchmark ran, something went terribly wrong. The first partition completed in a blistering 1,590 milliseconds — a 58% improvement over the Phase 8 baseline — and then everything collapsed. Partition after partition failed with out-of-memory errors, corrupting the CUDA context and cascading into total failure.

Message [msg 2447] captures the moment of diagnosis. It is a masterclass in systems-level debugging: the assistant, confronted with a baffling OOM failure on a GPU that should have had 14 GiB free, systematically traces through CUDA memory allocation internals, reference-counted smart pointer destructors, and the subtle asymmetry between CUDA's synchronous and asynchronous memory pools. What emerges is a root cause that is invisible to nvidia-smi and undetectable by conventional memory accounting: CUDA's cudaMallocAsync and cudaMalloc draw from separate memory pools, and memory freed via the async path is not returned to the synchronous pool. The consequence is a phantom OOM — the GPU reports 14 GiB free, but the synchronous allocator sees none of it.

The Context: Phase 9's Two-Pronged Optimization

To understand the diagnostic journey in [msg 2447], we must first understand what Phase 9 was trying to accomplish. The Phase 8 baseline had identified two root causes of GPU utilization dips during proof generation. The first was that 6 GiB of a/b/c polynomial data was being uploaded from non-pinned host memory while the GPU mutex was held, wasting precious milliseconds on slow, unpinned transfers. The second was that the Pippenger multi-scalar multiplication (MSM) issued a hard sync() call after every batch, preventing GPU compute from overlapping with device-to-host result transfers.

Phase 9's implementation, described in [msg 2440] and the surrounding conversation, addressed both issues. Change 1 (Tier 1) pinned the host memory buffers with cudaHostRegister, allocated device-side buffers for the a/b/c polynomials, and issued asynchronous cudaMemcpyAsync transfers on a dedicated stream with CUDA event-based synchronization. This moved the 6 GiB upload out of the GPU mutex, allowing one worker to transfer data while another worker held the GPU for computation. Change 2 (Tier 3) introduced double-buffered host result buffers for the Pippenger MSM, deferring the sync() call to the next iteration so that GPU compute could overlap with result retrieval.

The initial dual-worker benchmark with gpu_workers_per_device=2 and concurrency=3 failed catastrophically ([msg 2438]). All five proofs failed with OOM errors. The first partition succeeded — prestage_setup=ok, gpu_total_ms=1588 — but every subsequent partition failed with prestage_setup=fallback err=2, and even the fallback path (which avoided pre-staging entirely) also OOM'd. The GPU was left with 10 GiB of leaked memory after the failed runs, confirming that CUDA allocations were not being properly cleaned up after errors.

The Diagnostic Process: Following the Memory Trail

Message [msg 2447] opens with the assistant summarizing the state of affairs: partition 0 succeeds with spectacular speed (1,590 ms GPU time), but partition 1 onwards OOM. The OOM corrupts the CUDA context, dooming all subsequent partitions. The first partition's pre-staging allocated 12 GiB (4 GiB for d_a + 8 GiB for d_bc), ran to completion, freed d_bc inside the worker thread, freed d_a via the gpu_ptr_t smart pointer destructor, and released the mutex. The GPU should have been clean. Yet partition 1's pre-staging failed.

The assistant then embarks on a meticulous line-by-line analysis of the CUDA memory lifecycle. This is not a casual glance at error messages — it is a deep forensic examination of the C++ source code in gpu_t.cuh, tracing each allocation and deallocation path to understand where memory might be leaking.

The first hypothesis is that cudaFree from the gpu_ptr_t destructor doesn't return memory to the allocator immediately. But gpu_ptr_t uses synchronous cudaFree(ptr->ptr), which should be immediate. The assistant then examines dev_ptr_t and gpu.Dmalloc/Dfree, which use cudaMallocAsync/cudaFreeAsync with stream synchronization. These are also synchronous — Dfree calls cudaFreeAsync followed by zero.sync(), which should complete the free before returning.

But then the assistant has a critical insight. Quoting directly from the message:

cudaMallocAsync uses CUDA memory pools. These pools cache freed memory and don't necessarily release it back to cudaMalloc's pool. So cudaFreeAsync followed by cudaMallocAsync works (same pool), but cudaFreeAsync followed by cudaMalloc (used by dev_ptr_t) might not see the freed memory!

This is the root cause. The MSM (msm_t) allocates its buckets via gpu.Dmalloc() which calls cudaMallocAsync. After the MSM completes, gpu.Dfree() calls cudaFreeAsync, returning the memory to CUDA's async memory pool. But the next partition's dev_ptr_t or pre-staging code uses synchronous cudaMalloc — a different allocator that draws from a different pool. The async pool's free memory is invisible to the synchronous allocator. The GPU appears to have 14 GiB free (as reported by nvidia-smi), but the synchronous allocator sees only what remains in its own pool, which may be far less.

The Assumptions That Had to Be Broken

This diagnosis required the assistant to challenge several implicit assumptions. The first assumption was that cudaFree is cudaFree — that freeing GPU memory through any API path makes it available to all subsequent allocations. CUDA's memory pool architecture violates this assumption. The cudaMallocAsync API, introduced in CUDA 11.2, maintains a per-stream memory pool that caches freed blocks for reuse within the same pool. Memory freed via cudaFreeAsync goes back to this pool, not to the global free memory that cudaMalloc draws from.

The second assumption was that nvidia-smi reports usable free memory. The assistant had checked nvidia-smi earlier ([msg 2440]) and found 5,697 MiB free out of 16,303 MiB total — plenty of room for a 12 GiB allocation. But nvidia-smi reports physical VRAM free, not allocator-visible free. If the async pool holds 8 GiB of cached-but-not-released memory, nvidia-smi shows it as "used" (because the memory is still allocated to the CUDA context), but the synchronous allocator can't touch it even if it were freed.

The third assumption was that the memory accounting in the C++ code was correct. The assistant spent considerable effort tracing through gpu_ptr_t reference counting, lambda capture semantics, and destructor ordering to verify that worker 0's cleanup was complete. This analysis, spanning several paragraphs of the message, confirms that the memory management code is correct — the freed memory simply goes to the wrong pool.

The Reasoning Process: A Window Into Systems Debugging

What makes [msg 2447] particularly valuable as a case study is the transparency of the assistant's reasoning process. The message reads like a debugger's stream of consciousness, with hypotheses proposed, tested against code evidence, and either refined or discarded.

The assistant begins with the high-level observation ("partition 0 succeeds, partition 1 onwards OOMs") and immediately frames the puzzle: "The first partition's pre-staging allocates 12 GiB. It succeeds. It runs, frees d_bc (8 GiB) inside the thread, d_a (4 GiB) via gpu_ptr_t. GPU should be clean. But then partition 1's pre-staging fails with err=2."

The next paragraph introduces the pool hypothesis: "The issue is likely that the cudaFree from gpu_ptr_t destructor doesn't actually return memory to the allocator immediately — CUDA's memory pool might hold onto it." The assistant then checks the code to confirm that gpu_ptr_t uses synchronous cudaFree, which should return memory immediately. But then notices that dev_ptr_t and gpu.Dmalloc/Dfree use cudaMallocAsync/cudaFreeAsync with stream sync.

This is the moment of insight. The assistant quotes the relevant source code from gpu_t.cuh lines 209-217 and 71-75, showing the Dmalloc/Dfree wrappers and the underlying stream_t::Dmalloc. The key observation: gpu.Dmalloc calls cudaMallocAsync, and gpu.Dfree calls cudaFreeAsync. The MSM uses gpu.Dmalloc for its bucket allocations. When the MSM destructor frees these buckets, they go to the async pool. But the next partition's dev_ptr_t uses synchronous cudaMalloc — a different pool.

The assistant then crystallizes the diagnosis: "This is the root cause. The msm_t constructor uses gpu.Dmalloc() → cudaMallocAsync pool. After the MSM completes, gpu.Dfree() → cudaFreeAsync — returns to the async pool. But then the next partition's dev_ptr_t or pre-staging uses cudaMalloc (synchronous) — different allocator, doesn't see the async pool's free memory."

Input Knowledge Required

Understanding this message requires significant background knowledge spanning multiple domains. First, one must understand the Groth16 proof generation pipeline for Filecoin PoRep, including the role of NTT (number-theoretic transform), MSM (multi-scalar multiplication), and the partition-based proving architecture. The cuzk engine processes proofs in partitions, each handling a subset of the circuit's constraints, and the GPU workers execute the heavy linear algebra kernels.

Second, one must understand CUDA memory management: the distinction between cudaMalloc (synchronous, global pool) and cudaMallocAsync (per-stream, pool-based), the behavior of cudaFree vs cudaFreeAsync, and the fact that CUDA memory pools cache freed blocks rather than releasing them to the global allocator. This is a subtle point that even experienced CUDA developers may not fully appreciate.

Third, one must understand the C++ memory management patterns in the codebase: gpu_ptr_t (a reference-counted smart pointer that calls cudaFree in its destructor), dev_ptr_t (a simpler RAII wrapper using cudaMalloc/cudaFree), and the lambda capture semantics that determine whether d_bc_prestaged is properly nullified after cleanup.

Fourth, one must understand the concurrency architecture: the per-GPU mutex that serializes access to the GPU, the dual-worker model where two GPU workers share one device, and the pre-staging logic that allocates device buffers before the mutex is acquired.

Output Knowledge Created

This message produces several important pieces of knowledge. First and most concretely, it identifies the root cause of the OOM failures: CUDA memory pool asymmetry between cudaMallocAsync and cudaMalloc. This is not a bug in the application code — it is a fundamental property of CUDA's memory management that the code was unknowingly relying on.

Second, the message establishes that the memory-aware allocator approach proposed by the user in [msg 2445] — querying cudaMemGetInfo, subtracting a safety margin, and sizing allocations accordingly — is the correct fix. The assistant explicitly states this at the end of the message: "The fix for the memory-aware approach: use cudaMemGetInfo to check actual free memory, and size our pre-staging accordingly."

Third, the message demonstrates a rigorous debugging methodology. The assistant does not guess at the cause — it traces through every allocation and deallocation path, verifies reference counting, checks destructor ordering, and considers edge cases like CUDA context corruption after errors. This methodology is itself a reusable artifact.

Fourth, the message implicitly documents a performance-critical property of the cuzk engine: the first partition's GPU time of 1,590 ms represents a dramatic improvement over the Phase 8 baseline of 3,746 ms, confirming that the PCIe transfer optimization is working correctly when it can allocate memory. The OOM issue is a memory management problem, not a performance architecture problem.

Mistakes and Incorrect Assumptions

The message contains one notable incorrect assumption that the assistant later corrects. Early in the analysis, the assistant considers whether the cudaHostRegister call is causing the failure — specifically, whether two workers processing different partitions of the same proof would try to register the same host memory pages simultaneously, getting cudaErrorHostMemoryAlreadyRegistered (error 712). The assistant walks through this scenario in detail:

Both workers enter generate_groth16_proofs_c and: 1. Both do cudaHostRegister (before mutex) — the second one might fail with cudaErrorHostMemoryAlreadyRegistered (err=712) because the SAME host memory buffers are being registered by worker 0 already!

This is a reasonable hypothesis, but the assistant ultimately abandons it because the fallback path (which skips host registration entirely) also fails with OOM. If the only problem were double-registration, the fallback path would work — it just wouldn't get the performance benefit of pinned memory. The fact that the fallback also OOMs points to a deeper issue.

The assistant also spends considerable time verifying that the C++ memory management is correct — tracing gpu_ptr_t reference counts, lambda captures, and destructor ordering. This analysis is correct but ultimately irrelevant to the root cause. The memory management is fine; the problem is that the freed memory goes to the wrong CUDA pool. This is a valuable lesson: even when your code is correct, the underlying system may not behave as you expect.

The Broader Significance

Message [msg 2447] represents a turning point in the Phase 9 implementation. Before this message, the assistant was operating under the assumption that CUDA memory allocation is a simple "malloc/free" model where freed memory is immediately available for any subsequent allocation. After this message, the assistant understands that CUDA's memory pool architecture introduces subtle pool-affinity constraints that can cause phantom OOMs.

The fix — using cudaMemGetInfo to query actual free memory and sizing pre-staging allocations accordingly — is implemented in subsequent messages ([msg 2448] onwards). The single-worker benchmark in [msg 2444] had already shown that Phase 9's optimizations work spectacularly when memory allocation succeeds: NTT+MSM time dropped 71.6%, tail MSM dropped 34.4%, and overall GPU time per partition dropped 50-61%. The throughput improved from 37.4 s/proof to 32.1 s/proof, a 14.2% gain. The memory-aware allocator would unlock these gains for the dual-worker configuration as well.

This message also illustrates a recurring theme in systems optimization: the deepest bugs are often at the boundaries between abstraction layers. The CUDA memory pool API is designed to improve performance by caching allocations, but this optimization becomes invisible to the programmer who expects a uniform memory model. The assistant had to pierce this abstraction, reading the actual cudaMallocAsync and cudaFreeAsync calls in the source code, to understand why the GPU appeared to have free memory but allocations still failed.

Conclusion

Message [msg 2447] is a remarkable piece of diagnostic reasoning. In the span of a single message, the assistant identifies a root cause that is invisible to standard monitoring tools, requires deep knowledge of CUDA internals, and explains a baffling failure mode that had derailed an otherwise successful optimization. The message demonstrates the value of systematic debugging: tracing every allocation path, verifying every destructor, and challenging assumptions until the evidence forces a conclusion.

The CUDA memory pool asymmetry diagnosed in this message is not a bug that can be fixed — it is a fundamental property of the hardware abstraction layer that must be worked around. The memory-aware allocator that the assistant proposes is the correct mitigation: rather than assuming that freed memory is available, query the allocator directly and adapt. This principle — never assume, always measure — is the hallmark of production-grade systems engineering.