Diagnosing Memory Accumulation in a Split GPU Proving Pipeline

In the high-stakes world of Filecoin proof generation, where each Groth16 proof consumes hundreds of gigabytes of memory and minutes of compute time, a seemingly innocuous architectural change can cascade into a memory crisis. Message 3010 of an opencode optimization session captures the exact moment when a developer—the AI assistant—confronts a puzzling memory accumulation problem in Phase 12 of a GPU proving pipeline optimization, traces it to its root cause, and designs a targeted fix. This message is a masterclass in diagnostic reasoning under the constraints of a real-world production system.

The Context: Phase 12 and the Split GPU API

The assistant had been working through a series of optimization phases for the SUPRASEAL_C2 Groth16 proof generation pipeline used in Filecoin's Proof-of-Replication (PoRep). Phase 12 introduced a "split GPU proving API": instead of running the entire GPU proof generation synchronously (including the b_g2_msm multi-scalar multiplication), the API was restructured so that start_groth16_proof kicks off the GPU kernel and returns immediately with a PendingProofHandle, while a background C++ thread (prep_msm_thread) handles the b_g2_msm computation. A finalizer task later joins the background thread and completes the proof. This decoupling was designed to hide the b_g2_msm latency (~1.7 seconds) from the GPU worker's critical path, improving throughput.

But the user had reported OOM (Out of Memory) failures when increasing synthesis parallelism (pw=12), and had repeatedly expressed suspicion that something was leaking: "OOM-ish again" ([msg 2985]), "Maybe we're not freeing b_g2_m mem?" ([msg 2986]), "It's not 400GB tho, something is leaking somewhere" ([msg 2993]). The assistant had been tracing the memory lifecycle through multiple rounds of code reading and analysis, and message 3010 is the culmination of that investigation.

The RSS Trace: A Story in Numbers

The message opens with a clear, data-driven conclusion:

Clear trend. RSS grows monotonically during the run: - Start: 70 GiB - Peak: 367 GiB at 01:44:35 - Stabilizes around 300-350 GiB in the later proofs - Returns to 71 GiB after all work completes (no permanent leak)

This is the smoking gun. The assistant had set up an RSS monitor that sampled the daemon's memory usage every 5 seconds during a 20-proof benchmark run. The trace showed a monotonic climb from a 70 GiB baseline (SRS tables, PCE runtime, and other fixed overhead) to a peak of 367 GiB, followed by a plateau in the 300-350 GiB range, and finally a return to 71 GiB after all work completed. The return to baseline was critical: it ruled out a true memory leak. Memory was not being permanently lost—it was accumulating during the run and then being released when the workload finished.

The assistant immediately reframed the problem: "So there's no leak per se, but memory is accumulating during the run." This distinction—between a leak (permanent loss) and accumulation (temporary buildup due to throughput mismatch)—is the conceptual breakthrough that drives the rest of the analysis.

Connecting Accumulation to Architecture

The assistant's reasoning proceeds in a tight causal chain. With 15 concurrent jobs (j=15), each proof consisting of 10 partitions (pw=10), the system has up to 150 partition synthesis tasks in flight at any given moment. Each partition synthesis allocates roughly 13 GiB of ProvingAssignment data (the a, b, c NTT evaluation vectors plus input/auxiliary assignments).

In the pre-Phase 12 code, the synthesis data was freed as soon as gpu_prove returned synchronously. The GPU worker thread would synthesize a partition, call gpu_prove (which included b_g2_msm), get the result, and free the synthesis data—all within a single synchronous call. The synthesis data's lifetime was tightly bounded.

Phase 12 changed this. Now start_groth16_proof returns a PendingProofHandle that holds the Rust-side synthesis data (provers, input_assignments, aux_assignments) alive until the finalizer task runs finish_pending_proof, which joins the b_g2_msm background thread. The GPU worker thread picks up the next partition immediately after start_groth16_proof returns, but the synthesis data from the previous partition is still alive in the PendingProofHandle, waiting for the ~1.7 second b_g2_msm to complete. With gw=2 (two GPU workers), up to two PendingProofHandles can be outstanding simultaneously, each holding ~13 GiB of synthesis data. That's ~26 GiB of extra memory that was previously freed immediately.

The assistant calculates the impact precisely:

With pw=10 and j=15 this peaks at 367 GiB, leaving ~388 GiB headroom. With pw=12 (two more concurrent syntheses, each ~13 GiB), that would add ~26 GiB, pushing to ~393 GiB. The pw=15 OOM is explained by the even higher synthesis concurrency.

This explains why pw=10 worked (367 GiB peak, within the 755 GiB system limit) but pw=12 OOM'd (projected ~393 GiB peak, plus the extra synthesis concurrency pushing beyond capacity). The Phase 12 split API had inadvertently extended the lifetime of synthesis data, creating a memory accumulation problem that was invisible in the pre-Phase 12 code.

The Proposed Fix: Early Deallocation

Having identified the root cause, the assistant proposes a surgical fix:

The fix: Free the Rust synthesis data immediately after start_groth16_proof returns — don't hold it in PendingProofHandle. The question is whether the C++ b_g2_msm thread still needs that data.

This is the critical design question. The PendingProofHandle was designed to keep synthesis data alive precisely because the developer assumed the background b_g2_msm thread might reference Rust-owned memory. But if the C++ thread only accesses data that was copied into C++-owned memory during start_groth16_proof, then the Rust data can be freed immediately, restoring the pre-Phase 12 memory behavior while keeping the throughput benefit of the split API.

The assistant immediately starts investigating by reading the C++ code:

[read] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu

The quoted snippet shows the prep_msm_thread lambda capturing provers by reference ([&, num_circuits]) and accessing prover.inp_assignment_data and prover.aux_assignment_data—raw pointers into Rust-owned Vec<Arc<Vec<Fr>>>. This confirms that the background thread does read Rust-owned memory, meaning the data cannot be freed until the thread completes.

But the assistant doesn't stop there. The investigation continues beyond this message (in subsequent chunks) to find a way to either copy the data into C++-owned memory or restructure the ownership so that Rust can free its vectors earlier. The key insight established in this message is the direction of the fix: shorten the lifetime of synthesis data, not by changing the finalization path, but by freeing it as soon as the GPU kernel region completes.

Assumptions and Their Validity

The assistant's analysis rests on several assumptions that deserve scrutiny. First, it assumes that the 367 GiB peak is entirely explained by the extended lifetime of synthesis data in PendingProofHandles. This is a reasonable first-order model, but it doesn't account for potential second-order effects like memory fragmentation, glibc arena expansion, or the interaction between the Rust and C++ dealloc mutexes that the assistant had been investigating in earlier messages (<msg id=3001-3004>). The RSS trace shows the peak but not its composition.

Second, the assistant assumes that freeing the Rust synthesis data immediately after start_groth16_proof returns would restore the pre-Phase 12 memory behavior. This depends on whether the C++ prep_msm_thread can be made to not reference Rust-owned memory—either by copying the data into C++-owned buffers during start_groth16_proof, or by restructuring the data flow so that the Rust vectors are no longer needed. The assistant acknowledges this uncertainty explicitly: "The question is whether the C++ b_g2_msm thread still needs that data."

Third, the assistant assumes that the 70 GiB baseline and 367 GiB peak are representative of steady-state behavior. The RSS trace shows a monotonic climb to 367 GiB, then stabilization around 300-350 GiB. But the "stabilization" might be an artifact of the specific workload mix (20 proofs, 15 concurrent jobs) rather than a true steady state. A longer run with more proofs might show continued growth to a higher plateau, or even eventual OOM.

The Thinking Process: A Window into Diagnostic Reasoning

What makes this message particularly valuable is the visible reasoning process. The assistant doesn't just present conclusions—it walks through the logic step by step, showing how raw data (RSS samples) is transformed into insight (memory accumulation from extended data lifetime).

The reasoning chain is:

  1. Observe: RSS grows monotonically from 70 GiB to 367 GiB, then returns to baseline.
  2. Classify: This is accumulation, not a leak—memory is released when work stops.
  3. Quantify: Each partition is ~13 GiB, each PendingProofHandle holds one partition's data for ~1.7s extra.
  4. Model: With gw=2, two handles outstanding → ~26 GiB extra. With pw=12, two more syntheses → ~26 GiB more.
  5. Predict: pw=10 works (367 GiB < 755 GiB), pw=12 OOMs (~393 GiB projected, plus synthesis concurrency).
  6. Prescribe: Free synthesis data earlier—immediately after start_groth16_proof returns.
  7. Verify: Check if C++ thread still needs Rust data (it does—reads inp_assignment_data and aux_assignment_data). This is textbook diagnostic methodology: measure, classify, quantify, model, predict, prescribe, verify. The assistant's ability to move fluidly between Rust code, C++ CUDA code, system-level RSS monitoring, and architectural reasoning is what makes the analysis compelling.

Knowledge Created and Consumed

This message consumes several forms of input knowledge. It requires understanding of the Phase 12 split API architecture (the PendingProofHandle, the prep_msm_thread, the finalizer task). It requires familiarity with the memory characteristics of Groth16 proof generation (partition size ~13 GiB, SRS baseline ~70 GiB). It requires knowledge of Linux memory management (RSS, malloc_trim, glibc arena behavior). And it requires the ability to read and interpret C++ CUDA code to trace data dependencies.

The message creates new knowledge that is immediately actionable. It establishes that the memory problem is accumulation, not a leak—which changes the debugging strategy from "find the missing free" to "shorten the data lifetime." It provides a quantitative model linking the number of outstanding PendingProofHandles to the peak RSS, enabling precise predictions about which configurations will OOM. And it identifies the specific code location (the prep_msm_thread's reference to provers[c]) that must be addressed to enable early deallocation.

Conclusion

Message 3010 is a turning point in the Phase 12 optimization effort. Before this message, the team was chasing a phantom memory leak, suspecting missing deallocations or mutex contention. After this message, the problem is clearly understood as a structural consequence of the split API design—an architectural trade-off between throughput and memory footprint. The path forward is equally clear: either copy the data into C++-owned memory so Rust can free its vectors early, or restructure the ownership to decouple the Rust synthesis data lifetime from the C++ background thread.

The message exemplifies the kind of diagnostic work that distinguishes surface-level debugging from deep system understanding. It doesn't just answer "what is wrong?"—it answers "why is it wrong, how wrong is it, and what do we need to change to make it right?" In a system where a single proof consumes 200+ GiB of memory and takes 37 seconds to generate, that depth of understanding is not a luxury—it is a necessity.