The 8-GiB Ghost: How a Single Pointer Lifetime Bug Nearly Derailed a GPU Optimization

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, memory management is not merely a housekeeping chore — it is the central architectural challenge. When the cuzk SNARK proving engine's Phase 9 PCIe Transfer Optimization began crashing with out-of-memory (OOM) errors during dual-worker benchmarks, the root cause turned out to be a single raw pointer whose lifetime extended just a few lines too long. Message <msg id=2432> captures the precise moment when the assistant identified and resolved this critical bug — a deceptively simple edit that freed 8 GiB of GPU VRAM at exactly the right moment, salvaging an optimization that had already demonstrated 14.2% throughput gains in single-worker mode.

The Context: Phase 9's Promise and Its OOM Crisis

The Phase 9 optimization targeted two root causes of GPU idle gaps identified in the Phase 8 baseline. The first change moved 6 GiB of non-pinned a/b/c polynomial uploads out of the GPU mutex by pinning host memory with cudaHostRegister, allocating device buffers, and issuing async cudaMemcpyAsync transfers on a dedicated stream with event-based synchronization. The second change eliminated per-batch hard sync stalls in the Pippenger MSM by introducing double-buffered host result buffers. Together, these changes promised to dramatically reduce the time the GPU spent waiting for data transfers.

Initial single-worker benchmarks were spectacular: NTT+MSM time dropped from ~2430 ms to ~690 ms (a 71.6% reduction), tail MSM from ~125 ms to ~82 ms (34.4% reduction), and overall GPU time per partition from ~3746 ms to ~1450–1900 ms (50–61% reduction). Throughput improved from 37.4 s/proof to 32.1 s/proof — a 14.2% gain. But when the assistant scaled up to the intended dual-worker configuration (gpu_workers_per_device=2), the system crashed with cudaErrorMemoryAllocation errors.

The Debugging Chain: Tracing the OOM

The debugging trail began in message <msg id=2419>, where the assistant first diagnosed the core issue: with two workers, both tried to pre-stage their 12 GiB allocations simultaneously, exceeding the 16 GiB GPU VRAM. The initial fix moved pre-staging inside the GPU mutex, serializing the allocations across workers. But even after this change, OOM errors persisted.

Message <msg id=2426> revealed a deeper problem: the first partition succeeded (prestage_setup=ok), but subsequent partitions failed. The assistant traced through the code flow meticulously in <msg id=2429>, reconstructing the exact sequence of allocations and deallocations:

Worker 0 acquires mutex, pre-stages d_a (4 GiB) + d_bc (8 GiB) = 12 GiB
Worker 0 runs NTT+MSM, d_a is freed (gpu_ptr_t destructor)
Worker 0 releases mutex
d_bc is freed AFTER mutex release — 8 GiB still allocated!
Worker 1 acquires mutex, tries to pre-stage its own 12 GiB
8 GiB (worker 0's d_bc) + 12 GiB (worker 1's pre-stage) = 20 GiB → OOM The critical discovery was that d_bc_prestaged — an 8 GiB raw pointer — was being freed in the cleanup section after the mutex release, not before. This meant that when worker 1 acquired the mutex, worker 0's d_bc was still consuming VRAM, leaving insufficient room for worker 1's pre-stage allocation.

The Subject Message: A Precise Surgical Fix

Message <msg id=2432> reads in its entirety:

I need to free d_bc_prestaged right after execute_ntt_msm_h_prestaged returns (the NTT phase is done, d_bc is no longer needed). The problem is d_bc_prestaged is a raw pointer allocated in the outer scope. I need to free it AND null it so the cleanup code doesn't double-free.

But d_bc_prestaged is captured by reference in the lambda. Modifying it inside the lambda will update the outer variable. Let me add the free: [edit] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu Edit applied successfully.

This message is deceptively brief, but it encapsulates a profound understanding of the system's memory lifecycle. The assistant recognized that d_bc_prestaged was needed only during the NTT phase of execute_ntt_msm_h_prestaged — after the coefficient-wise multiply kernel completed, the buffer's data was fully consumed. In the original non-pre-staged code path, this was handled naturally by dev_ptr_t<fr_t> d_b(...) being a local variable that went out of scope. But in the pre-staged path, d_bc_prestaged was a raw pointer allocated in the outer function scope, with its lifetime managed manually in the cleanup section after the mutex release.

The fix required two simultaneous actions: calling cudaFree(d_bc_prestaged) to release the 8 GiB, and setting d_bc_prestaged = nullptr to prevent the cleanup code from double-freeing. The assistant verified that because d_bc_prestaged was captured by reference in the lambda (the per-GPU thread), modifying it inside the lambda would correctly update the outer scope variable — a subtle but crucial C++ semantics detail.

Input Knowledge Required

To understand this message, one needs deep knowledge of several domains. First, the Groth16 proof generation pipeline for Filecoin PoRep, specifically the role of the a/b/c polynomials and their NTT (Number Theoretic Transform) processing. The d_bc buffer holds the b and c polynomials combined, consuming 8 GiB of VRAM at 32 bytes per element across 128M elements (domain size 2^27). Understanding that d_bc is only needed during the NTT phase — and not during the subsequent H MSM (Multi-Scalar Multiplication), batch addition, or tail MSM phases — is essential.

Second, CUDA memory management: the distinction between gpu_ptr_t (RAII wrapper that frees on scope exit) and raw cudaMalloc pointers, the semantics of cudaFree, and the importance of nullifying freed pointers to prevent double-free. The assistant also needed to understand that cudaFree is synchronous and immediately releases VRAM back to the pool.

Third, C++ lambda capture semantics: the assistant verified that d_bc_prestaged was captured by reference (&d_bc_prestaged), meaning modifications inside the lambda affect the outer variable. This is critical because the lambda executes in a separate thread (the per-GPU thread), and the outer scope's cleanup code runs after the thread joins.

Fourth, the dual-worker interlock architecture from Phase 8: two GPU workers share a single physical GPU via a mutex, interleaving their kernel executions. The mutex ensures only one worker runs CUDA kernels at a time, but the VRAM allocations persist across mutex acquisitions unless explicitly freed.

Output Knowledge Created

This message created a corrected memory lifecycle for the pre-staged buffers. The immediate output was a one-line edit to groth16_cuda.cu that inserted cudaFree(d_bc_prestaged); d_bc_prestaged = nullptr; after the execute_ntt_msm_h_prestaged call inside the per-GPU thread lambda. But the knowledge created extends far beyond that single line.

The message established a critical design principle for the pre-staging architecture: pre-staged buffers must be freed at the earliest possible point in the pipeline, not at the end of the mutex scope. This principle directly contradicts the natural inclination to centralize cleanup code. The assistant recognized that d_bc's 8 GiB footprint was too large to carry through the entire proof generation pipeline, and that the original code's pattern (local dev_ptr_t going out of scope after NTT) should be replicated in the pre-staged path.

Furthermore, the message created a reusable pattern for managing raw CUDA pointers in a lambda-based threading model: capture by reference, free at the earliest safe point, null the pointer, and let the outer cleanup code check for null before attempting to free again. This pattern was subsequently applied to other pre-staged buffers and became part of the Phase 9 implementation's memory management strategy.

Assumptions and Mistakes

The debugging chain reveals several incorrect assumptions that the assistant had to correct. The most significant was the assumption that moving pre-staging inside the mutex was sufficient to prevent OOM. Message <msg id=2419> initially proposed this fix, but the assistant later discovered in <msg id=2429> that the real issue was d_bc's lifetime extending beyond the mutex release. This is a classic example of a "fix that fixes the symptom but not the cause" — the mutex serialization prevented simultaneous allocation, but the cumulative VRAM pressure from overlapping worker lifetimes remained.

Another incorrect assumption was about the domain size computation. In <msg id=2428>, the assistant initially suspected a bug in the lg2 function, computing domain sizes and comparing them against the log output. After tracing through the binary arithmetic, the assistant realized the domain was correctly 2^27 (128M elements) because the H SRS has domain_size + 1 points, not domain_size - 1. This detour, while ultimately a red herring, demonstrates the assistant's thoroughness in verifying every possible source of error.

The assistant also initially assumed that the OOM was caused by CUDA memory pool fragmentation (cudaMallocAsync/cudaFreeAsync not releasing to the synchronous pool). While this was a contributing factor in some failure modes, the primary cause was the simpler lifetime issue. The assistant's willingness to consider multiple hypotheses — and to systematically test each one against the log evidence — is a hallmark of effective debugging.

The Thinking Process: A Window into Systematic Debugging

The reasoning visible in the surrounding messages reveals a methodical, hypothesis-driven approach. The assistant begins with the observable symptom (OOM errors in dual-worker mode), formulates a hypothesis (simultaneous pre-staging), implements a fix (move pre-staging inside mutex), tests it (fails), examines the new evidence (first partition succeeds, subsequent ones fail), and refines the hypothesis (d_bc lifetime extends beyond mutex release).

What is particularly striking is the assistant's willingness to trace through the code manually, computing exact memory sizes and allocation sequences. In <msg id=2429>, the assistant walks through the entire 8-step flow for worker 0 and worker 1, computing cumulative VRAM usage at each step: 12 GiB during pre-stage, 8 GiB after d_a freed, 20 GiB when worker 1 tries to allocate while worker 0's d_bc is still alive. This manual trace is the intellectual core of the debugging effort — it transforms a vague "OOM error" into a precise, quantified understanding of memory pressure.

The subject message itself represents the final synthesis of this debugging chain. The assistant has already identified the root cause (d_bc freed too late) and verified the code structure (d_bc_prestaged is a raw pointer in outer scope, captured by reference in lambda). The message is the moment of action — the assistant confirms the fix strategy and applies it. The brevity of the message belies the depth of reasoning that preceded it.

The Broader Significance

This episode illustrates a fundamental tension in GPU optimization: the conflict between maximizing overlap (by keeping buffers alive for async transfers) and minimizing peak memory (by freeing buffers as early as possible). The Phase 9 optimization attempted to overlap H-to-D transfers with compute by pre-staging buffers outside the mutex. But the dual-worker configuration introduced a second dimension of overlap — two workers interleaving their GPU time — which created cumulative memory pressure that the single-worker design had not anticipated.

The fix in message <msg id=2432> resolves this tension by freeing d_bc at the earliest possible point — immediately after the NTT phase completes — rather than at the end of the mutex scope. This reduces the peak memory of the pre-staged path from 12 GiB + working memory to 4 GiB + working memory during the H MSM and later phases, making room for the other worker's pre-stage allocation.

The subsequent benchmark results (documented in chunk 1) confirmed the fix: the dual-worker configuration completed successfully with a system throughput of 41.0 s/proof, and detailed GPU timing logs showed kernel-level performance consistent with the single-worker results. The gap between single-worker (32.1 s) and dual-worker (41.0 s) throughput now points to PCIe bandwidth contention and CPU-side processing as the next bottlenecks — a testament to the iterative, data-driven optimization methodology that characterizes this entire project.

Conclusion

Message <msg id=2432> is a masterclass in surgical debugging. In one line of code, the assistant resolved an OOM crisis that threatened to derail a 14.2% throughput optimization. But the true value of this message lies not in the edit itself, but in the reasoning that produced it — a systematic, evidence-based investigation that traced a GPU memory error through multiple layers of abstraction, from CUDA error codes to C++ lambda semantics to the precise arithmetic of domain sizes and pointer lifetimes. It is a reminder that in systems programming, the difference between success and failure often comes down to a single pointer, freed at exactly the right moment.