The Moment of Hypothesis: Diagnosing a Memory Leak in Phase 12's Split GPU Proving API

Introduction

In high-performance systems engineering, the most critical insights often arrive not as fully-formed answers but as carefully articulated hypotheses. Message [msg 2987] of this opencode session captures precisely such a moment. The assistant, having just completed the Phase 12 split GPU proving API for the SUPRASEAL_C2 Groth16 proof generation pipeline, faces an unexpected memory pressure crisis. Attempts to increase synthesis parallelism from 10 to 12 or 15 partition workers have resulted in out-of-memory (OOM) failures on a 755 GiB system — a puzzling outcome that defies simple arithmetic. The user's pointed observation — "Maybe we're not freeing b_g2_m mem?" ([msg 2986]) — triggers a chain of reasoning that transforms the team's understanding of the problem. This message is the pivot point between treating OOM as a capacity ceiling and recognizing it as a genuine memory leak.

The Context: What Phase 12 Changed

To understand this message, one must grasp what Phase 12 actually accomplished. The Groth16 proving pipeline for Filecoin's Proof-of-Replication (PoRep) involves multiple computationally intensive stages: circuit synthesis (CPU-bound), NTT evaluation (GPU-bound), and multi-scalar multiplication (MSM) operations. One particular MSM — b_g2_msm — operates on the G2 curve and is relatively fast (~1 second) but occupies the GPU worker thread, blocking it from picking up the next partition's work. Phase 12 introduced a "split API": instead of the GPU worker waiting for b_g2_msm to complete before returning, the worker starts the proof, spawns a background finalizer task that runs b_g2_msm asynchronously, and immediately becomes available for the next partition. This architectural change promised better GPU utilization by decoupling the worker's critical path from a low-latency but blocking operation.

The initial benchmark results were promising: with the baseline configuration of 10 partition workers (pw=10), Phase 12 achieved 37.1 seconds per proof — a ~2.4% improvement over the Phase 11 baseline of 38.0 seconds. But the real test was whether the pipeline could scale to higher synthesis parallelism. When the user suggested trying pw=15 and pw=20 ([msg 2964]), the assistant dutifully ran the experiments. Both configurations OOM'd, with pw=12 surviving only 8 proofs before crashing ([msg 2983]). The user's laconic report — "OOM-ish again" ([msg 2985]) — followed by the sharper hypothesis "Maybe we're not freeing b_g2_m mem?" ([msg 2986]) sets the stage for the assistant's response.

The Message: Articulating the Memory Lifecycle Hypothesis

The assistant's reply in [msg 2987] is a model of disciplined systems thinking. It begins by validating the user's intuition: "Good point — with the split API, the finish_pending_proof runs in a spawned finalizer task, and the Rust-side synthesis data (provers, input_assignments, aux_assignments) is held alive in the PendingProofHandle until finalization completes." This is the core insight: Phase 12 changed the memory lifecycle of synthesis data.

Before Phase 12, the monolithic gpu_prove function was synchronous — it performed all GPU work (NTT, MSM, and b_g2_msm) in one blocking call, and when it returned, all Rust-side synthesis data could be freed. The memory was held for the duration of the function call and released immediately upon return. Phase 12's split API changed this: gpu_prove_start kicks off GPU work, returns a PendingProofHandle containing all the Rust-side data (provers, input_assignments, aux_assignments), and a spawned finalizer task calls finish_pending_proof later. The Rust-side data must remain alive until the finalizer completes because the C++ background thread may still reference it.

The assistant then quantifies the impact: "With gw=2, you could have two pending handles alive simultaneously (one per GPU worker), each holding ~13 GiB of synthesis data. That's ~26 GiB extra memory pressure compared to Phase 11." This calculation is significant because it frames the problem as a bounded, predictable increase — ~26 GiB — which should not cause OOM on a 755 GiB system. The assistant is implicitly acknowledging that the hypothesis of "extra memory from pending handles" is insufficient to explain the observed OOM. Something else must be going on.

The Reasoning Process: What the Message Reveals

The message contains a todowrite call that updates the task list, adding "Investigate memory leak: PendingProofHandle holds Rust-side data until finalization completes" as an in-progress item. This todo update is revealing: it shows the assistant has already accepted the user's framing that this is a leak, not merely a capacity issue. The shift from "we need more memory headroom" to "something is leaking" is the critical intellectual move.

The assistant's reasoning, visible in the structure of the message, follows a clear pattern:

Acknowledge the hypothesis: The user suggested memory isn't being freed. The assistant agrees this is worth investigating.
Trace the lifecycle change: Before Phase 12, data was freed synchronously. After Phase 12, data persists until the background finalizer completes. This is a structural change in memory ownership.
Quantify the expected impact: Two pending handles × ~13 GiB each = ~26 GiB extra. This is a concrete, testable prediction.
Identify the next investigative step: "Let me trace the memory lifecycle" — the assistant commits to examining the actual code paths to determine whether the hypothesis holds. The message is notable for what it doesn't do. It doesn't jump to conclusions. It doesn't propose a fix. It doesn't assert that the hypothesis is correct. Instead, it lays out a framework for investigation: here's what changed, here's the expected memory impact, now let's verify by examining the code.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message, some explicit and some implicit:

Explicit assumption: Each pending handle holds ~13 GiB of synthesis data. This is based on the known per-partition memory footprint from earlier analysis (each partition is ~1/10th of a full proof's synthesis data). The assumption is reasonable but may be incomplete — the PendingProofHandle might hold additional data beyond the raw synthesis vectors.

Explicit assumption: Two handles can be alive simultaneously because there are two GPU workers (gw=2). This assumes the pipeline's concurrency model allows both workers to have pending proofs at the same time, which is architecturally correct but depends on the actual scheduling dynamics.

Implicit assumption: The memory pressure from pending handles is additive and independent of other memory consumers. In reality, memory fragmentation, allocator behavior, and peak-vs-steady-state dynamics could cause OOM at lower total allocations than expected.

Potential mistake: The assistant calculates ~26 GiB extra pressure and implicitly concludes this shouldn't cause OOM on 755 GiB. While the arithmetic is sound, it overlooks the possibility that the leak is cumulative — if handles are never freed (or freed very slowly), the memory could grow unboundedly rather than plateauing at 2 × 13 GiB. The user's follow-up "It's not 400GB tho" ([msg 2990]) and "something is leaking somewhere" ([msg 2993]) push the assistant toward recognizing this.

Input Knowledge Required

To fully understand this message, the reader needs:

The Phase 12 split API architecture: Understanding that gpu_prove_start returns a PendingProofHandle and a background finalizer calls finish_pending_proof later, with the handle holding Rust-side data alive during the interval.
The memory profile of partition synthesis: Each partition's synthesis produces ~13 GiB of data (provers, input_assignments, aux_assignments, NTT evaluation vectors). This was established in earlier analysis (segment 0's deep-dive).
The GPU worker model: gw=2 means two GPU worker threads, each capable of having one pending proof at a time. This creates the possibility of two simultaneous pending handles.
The OOM symptom pattern: pw=10 works fine, pw=12 OOMs after 8 proofs, pw=15 OOMs immediately. The pattern suggests a cumulative leak rather than a fixed overhead.
The pre-Phase 12 behavior: Before the split API, gpu_prove was synchronous and freed all Rust-side data upon return. This is the baseline against which Phase 12's memory behavior must be compared.

Output Knowledge Created

This message creates several important outputs:

A testable hypothesis: The assistant proposes that pending handles holding synthesis data until finalization is the source of extra memory pressure. This hypothesis can be verified by instrumenting the code to track handle lifetimes and memory usage.
A quantitative prediction: ~26 GiB extra pressure from two pending handles. This provides a benchmark for evaluating whether the hypothesis explains the observed OOM.
A direction for investigation: "Let me trace the memory lifecycle" commits to examining the actual code paths — the C++ groth16_pending_proof struct, the PendingProofHandle Rust struct, and the prep_msm_thread background thread — to determine what memory is held and when it's freed.
A shift in framing: The message transforms the problem from "we need more memory headroom" to "we need to verify memory is being freed correctly." This reframing is essential for the subsequent deep investigation that follows in chunk 1.

The Aftermath: Where This Hypothesis Led

The investigation that follows this message ([msg 2988] onward) reveals a more complex picture than the initial hypothesis suggested. The assistant traces the C++ prep_msm_thread and discovers that the massive a, b, c NTT evaluation vectors (~12 GiB per partition) are only needed by the GPU kernel region, which completes before prove_start returns. An early deallocation of these vectors is implemented. But the OOM persists.

The true root cause, uncovered through building a global buffer tracker with atomic counters, is more subtle: the partition semaphore (pw=12) releases immediately after synthesis, allowing tasks to pile up while blocking on the single-slot GPU channel. The provers counter peaks at 28 — meaning 28 synthesized partitions are queued holding their full ~16 GiB datasets. This is a pipeline synchronization issue, not a simple leak from pending handles.

The assistant's initial hypothesis in [msg 2987] was partially correct — the split API did change memory lifetimes — but the actual magnitude of the problem (28 partitions × ~16 GiB = ~448 GiB) far exceeded the predicted ~26 GiB. The hypothesis was valuable not because it was perfectly accurate, but because it opened the door to systematic investigation. By accepting the user's framing that "something is leaking" and committing to trace the memory lifecycle, the assistant set in motion the diagnostic chain that ultimately revealed the true bottleneck.

Conclusion

Message [msg 2987] exemplifies the engineering discipline of hypothesis-driven debugging. In a complex system with ~200 GiB peak memory, multiple GPU workers, asynchronous finalization, and intricate Rust/C++/CUDA interactions, the assistant resists the temptation to guess at solutions. Instead, it articulates a clear, testable hypothesis grounded in the architectural change introduced by Phase 12, quantifies the expected impact, and commits to verification through code tracing. The hypothesis turns out to be incomplete — the real problem is a pipeline synchronization issue causing 28 queued partitions rather than 2 pending handles — but the investigative framework it establishes is what enables the true root cause to be found. This message is a testament to the power of disciplined reasoning in high-performance systems engineering: the best hypotheses are not the ones that are perfectly correct, but the ones that lead to deeper understanding.