The Devil in the Cleanup Order: Tracing a GPU Memory Leak in Phase 9 of the cuzk SNARK Proving Engine

In the high-stakes world of Filecoin Proof-of-Replication (PoRep) proving, every millisecond of GPU idle time is a direct hit to throughput. The cuzk SNARK proving engine, a custom Groth16 implementation optimized for the SUPRASEAL_C2 pipeline, had already undergone eight phases of optimization — from parallel synthesis to dual-worker GPU interlocking — each shaving seconds off the proof generation time. Phase 9 targeted the final frontier: PCIe transfer optimization, aiming to eliminate the GPU idle gaps caused by blocking host-to-device (HtoD) memory transfers. But when the first dual-worker benchmark ran, it crashed with a cascade of CUDA out-of-memory (OOM) errors. This article examines message [msg 2430], the pivotal moment when the assistant identified and fixed a subtle VRAM lifecycle bug that had been silently corrupting the optimization.

The Phase 9 Optimization: What Was Attempted

Phase 9 addressed two root causes of GPU utilization dips identified in the Phase 8 baseline ([msg 2425]). The first change (Tier 1) targeted the 6 GiB of non-pinned a/b/c polynomial uploads. In the baseline, these uploads happened inside the GPU mutex as blocking cudaMemcpy calls, forcing the GPU to idle while data trickled over PCIe. The optimization pinned host memory with cudaHostRegister, pre-allocated device buffers (d_a at 4 GiB and d_bc at 8 GiB), and issued async cudaMemcpyAsync transfers on a dedicated stream with event-based synchronization. The second change (Tier 3) eliminated per-batch hard sync stalls in the Pippenger MSM by introducing double-buffered host result buffers, deferring the sync() call to the next iteration.

The single-worker benchmark results were spectacular: NTT+MSM time dropped from ~2430 ms to ~690 ms (a 71.6% reduction), and overall throughput improved from 37.4 s/proof to 32.1 s/proof (+14.2%). But the true test was the dual-worker configuration (gpu_workers_per_device=2), where two workers would interleave their GPU access through the Phase 8 mutex-based interlock. When the assistant launched this benchmark, every proof failed.

The OOM Cascade

The daemon logs told a grim story. The first partition of the first proof succeeded — prestage_setup=ok with domain=134217728 (128M elements) and lot_of_memory=1. The NTT+MSM completed in 840 ms, dramatically faster than the Phase 8 baseline. But then the second partition failed: prestage_setup=fallback err=2 (where err=2 is cudaErrorMemoryAllocation). The fallback path then also OOM'd inside dev_ptr_t at gpu_t.cuh:331. Subsequent partitions panicked with unrecoverable CUDA errors.

The assistant's initial analysis in [msg 2419] correctly identified the first problem: the pre-staging allocation happened before the GPU mutex, so both workers tried to allocate 12 GiB simultaneously on a 16 GiB GPU. The fix was to move the pre-staging inside the mutex, serializing the allocations. But after rebuilding and retesting, the OOM persisted. This led to the deeper investigation in [msg 2427] through [msg 2429], where the assistant performed a meticulous forensic analysis of the GPU memory lifecycle.

The Target Message: A Moment of Insight

Message [msg 2430] captures the moment the assistant traced the OOM to its true root cause. The message reads:

I need to split the cleanup: free d_bc and GPU resources BEFORE unlocking the mutex (they use GPU resources), but unregister host pages AFTER (those are CPU-only). Actually, the d_bc memory is no longer needed after the NTT phase (which ends early in the per_gpu thread). I could free it right after the NTT completes but before batch_add. However, the simplest correct fix is: move ALL GPU resource cleanup before the unlock.

This is a deceptively simple statement that encodes hours of careful reasoning. The key insight is that the cleanup order — specifically, when d_bc (the 8 GiB device buffer for the b and c polynomials) is freed relative to the mutex unlock — determines whether the dual-worker pipeline can function at all.

The Reasoning Chain: Tracing the Memory Lifecycle

To understand why this matters, we must trace the exact sequence of events in the dual-worker configuration. The assistant reconstructed this flow in [msg 2429]:

Worker 0 (Partition 0):

  1. Pin host pages (before mutex) — OK
  2. Acquire GPU mutex
  3. Pre-stage: allocate d_a (4 GiB) + d_bc (8 GiB) = 12 GiB, issue async uploads
  4. Launch per_gpu thread → NTT+MSM using pre-staged buffers → batch_add → tail_MSMs
  5. Join per_gpu thread
  6. Release mutex
  7. Free d_bc (8 GiB), events, stream
  8. Unregister host pages Worker 1 (Partition 1) — waiting at step 2:
  9. Acquire mutex (after worker 0 releases it)
  10. Pre-stage: allocate d_a + d_bc = 12 GiB The critical observation: in the original code, step 7 (freeing d_bc) happened after step 6 (releasing the mutex). This meant that when Worker 1 acquired the mutex at step 2, Worker 0's d_bc was still allocated — 8 GiB of VRAM that should have been freed but wasn't. Worker 1 then tried to allocate its own 12 GiB, bringing the total to 20 GiB on a 16 GiB GPU. OOM was inevitable. The assistant's analysis in [msg 2429] is a masterclass in systematic debugging. It walks through the code, examining the scope of gpu_ptr_t<fr_t> d_a (which is freed by its destructor when the scope exits inside the per_gpu thread) versus the raw pointer d_bc_prestaged (which was freed in the outer cleanup block after the mutex release). The mismatch between these two lifetimes — one managed by RAII, the other by manual cleanup — was the bug.

The Fix: Two Layers of Correction

Message [msg 2430] proposes the first layer of correction: move all GPU resource cleanup before the mutex unlock. But the assistant immediately recognizes an even better optimization in the follow-up messages ([msg 2431], [msg 2432]): free d_bc right after the NTT phase completes, inside the per_gpu thread. The d_bc buffer holds the b and c polynomials, which are only needed for the NTT and coefficient-wise multiplication. After those kernels complete, the data is consumed and the 8 GiB can be released, making room for the MSM bucket allocations and the batch_add working memory.

This two-tier approach — free d_bc early within the per_gpu thread, and ensure all remaining GPU resources are freed before the mutex unlock — solved the OOM. The assistant applied the edit and rebuilt, and subsequent benchmarks (in later messages) confirmed the fix.

Assumptions and Mistakes

Several assumptions were challenged during this debugging session. The initial Phase 9 design assumed that pre-staging could happen outside the mutex, with the two workers' allocations coexisting peacefully. This was based on a memory budget calculation that assumed 7.5 GiB for the active worker plus 6 GiB for the pre-staging worker = 13.5 GiB, fitting comfortably in 16 GiB. But this calculation failed to account for the scenario where both workers pre-stage simultaneously (before either holds the mutex), consuming 12 GiB each.

A subtler assumption was that the cleanup order in the original code was correct. The code freed d_bc after the mutex release because it seemed like a "cleanup" operation that didn't need the GPU lock. But this overlooked the fact that the freed memory becomes available for the next worker's allocation — and the next worker can't proceed until it acquires the mutex and sees free VRAM. By freeing after the unlock, the code created a window where the next worker saw full VRAM even though the previous worker's buffers were logically done.

Input Knowledge Required

To fully understand this message, one needs knowledge of: CUDA memory management (cudaMalloc, cudaFree, cudaHostRegister, cudaMemGetInfo), GPU VRAM budgeting for large-scale SNARK proving (where individual buffers can be 4-8 GiB), the Phase 8 dual-worker mutex interlock design, the NTT/MSM pipeline phases and their memory requirements, RAII patterns in C++ (gpu_ptr_t vs raw pointers), and the specific domain size calculations for the PoRep-32g circuit (where the H SRS has 67,108,865 points, yielding a domain of 2^27 = 134,217,728 elements).

Output Knowledge Created

This message produced a corrected understanding of the GPU memory lifecycle in the dual-worker configuration. It established the principle that GPU resource cleanup must happen before the mutex release, not after. It also demonstrated that d_bc can be freed immediately after the NTT phase (not at the end of the per_gpu thread), freeing 8 GiB for downstream allocations. The edit applied in this message, combined with the follow-up in [msg 2432], created a working Phase 9 implementation that would later achieve 14.2% throughput improvement in single-worker mode and reveal PCIe bandwidth contention as the next bottleneck in dual-worker mode.

The Thinking Process

What makes this message remarkable is the visible thinking process. The assistant doesn't just state the fix — it walks through the tradeoffs. It considers splitting cleanup into GPU resources (freed before unlock) and CPU resources (host page unregistration, freed after unlock). It evaluates whether to free d_bc immediately after NTT or at the mutex release boundary. It chooses "the simplest correct fix" — moving all GPU cleanup before the unlock — while acknowledging that a more aggressive approach (freeing d_bc right after NTT) would be even better for memory pressure.

This is debugging at its finest: not just fixing the symptom (OOM), but understanding the exact memory lifecycle that causes it, and designing a fix that is both correct and minimally invasive. The message captures the moment of synthesis — when scattered observations about timing logs, domain sizes, allocation patterns, and mutex behavior coalesce into a precise diagnosis and a clean fix.