Tracing a Phantom Leak: How One Message Pivoted from Memory Pressure to Finalizer Debugging in Phase 12

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, memory is the ultimate constraint. When the Phase 12 split GPU proving API for the SUPRASEAL_C2 Groth16 pipeline delivered a promising 37.1 seconds per proof, the natural next step was to push harder: increase synthesis parallelism from 10 partition workers to 12, then 15, and watch throughput improve. Instead, the system crashed with out-of-memory (OOM) errors. The 755 GiB machine, which had comfortably handled Phase 11's concurrency, was suddenly gasping for RAM. In message [msg 2996], the assistant pivots from a superficial memory accounting exercise to a deep code trace, confirming a critical design constraint and then redirecting the investigation toward a more subtle bug: a potential failure to drop pending proof handles in the finalizer task.

The Context: Phase 12's Split API and the OOM Mystery

The Phase 12 optimization ([msg 2996]) was designed to decouple the GPU worker's critical path from the CPU-bound b_g2_msm multi-scalar multiplication. In the original design, each GPU worker would submit a partition to the GPU, wait for all GPU work including b_g2_msm to complete, then return. Phase 12 split this into start_groth16_proof (which launches GPU kernels and returns immediately, while b_g2_msm runs in a background C++ thread) and finish_pending_proof (which waits for the background thread and completes the proof). This allowed the GPU worker to pick up the next partition without stalling on b_g2_msm, improving pipeline throughput.

The baseline benchmark with partition_workers=10 succeeded at 37.1s/proof. But when the assistant tried pw=15 ([msg 2979]), the daemon crashed immediately with a broken pipe — the OOM killer had struck. Even pw=12 ([msg 2983]) failed after 8 proofs, with RSS spiking beyond the system's capacity. The assistant initially calculated that two extra partitions should add only ~26 GiB (~13 GiB each), which should be trivial on a 755 GiB machine. The user's observation in [msg 2993] — "It's not 400GB tho, something is leaking somewhere" — was the crucial insight that reframed the problem from a capacity calculation to a memory leak investigation.

The Two-Part Investigation

Message [msg 2996] unfolds in two distinct movements, each revealing a different layer of the problem.

Part 1: Confirming the C++ Memory Reference

The assistant begins by examining whether the C++ background thread (prep_msm_thread) references Rust-owned memory after start_groth16_proof returns. This is a critical safety question for the split API design. The assistant quotes the C++ code:

The prep_msm_thread captures provers by reference ([&, num_circuits]) and reads prover.inp_assignment_data and prover.aux_assignment_data — these are raw pointers into the Rust input_assignments and aux_assignments Vec<Arc<Vec<Fr>>>.

The conclusion is emphatic: "So yes, the b_g2_msm thread reads Rust-owned memory. The Rust data MUST stay alive until the prep_msm_thread completes."

This confirmation serves two purposes. First, it validates the Phase 12 design: the PendingProofHandle must hold the Rust-side provers, input_assignments, and aux_assignments alive until finalization. This is not a bug — it is a necessary design constraint. Second, it rules out one class of potential memory leak: the handle is not unnecessarily holding memory; it is holding memory that the C++ thread legitimately needs.

The assistant explicitly states: "That's correct behavior — the PendingProofHandle keeping them alive is necessary." This is an important moment of validation. The split API is not leaking memory through a design flaw; the memory retention is intentional and required for correctness.

Part 2: Pivoting to the Real Question

Having confirmed the design is sound, the assistant pivots sharply: "But the real leak question is different. Let me check if the PendingProofHandle is actually getting dropped/finalized."

This is the key insight. The question is not whether the handle should hold memory — it should. The question is whether the handle is being released after the background thread completes. If the finalizer task that calls finish_pending_proof is not properly dropping the PendingProofHandle, or if the async deallocation path has a bug, the memory would accumulate across partitions and proofs, explaining the OOM at pw=12.

The assistant then issues a [read] tool call to examine engine.rs at the finalizer task flow, specifically around line 1673 where the start_result is matched and the finalizer is spawned. This is the assistant moving from reasoning about the C++ side to tracing the Rust async lifecycle — from "is the memory held correctly?" to "is the memory released correctly?"

Assumptions and Their Refinement

This message reveals several assumptions, some explicit and some implicit:

Assumption 1: The OOM is caused by the extra partition workers. The assistant initially assumed that pw=12 simply required too much memory (12 × ~13 GiB = ~156 GiB for synthesis). The user's correction — that this shouldn't OOM a 755 GiB machine — forced a deeper investigation.

Assumption 2: The PendingProofHandle might be holding memory unnecessarily. The assistant checked whether the C++ thread actually needed the Rust data. It turns out it does. This assumption was wrong, but the investigation to disprove it was valuable because it confirmed the design's correctness.

Assumption 3: The finalizer task properly drops handles. This is the implicit assumption being questioned. The assistant now suspects this assumption might be false, which would explain the accumulating memory.

The refinement of these assumptions is a classic debugging pattern: start with the most obvious explanation (too much memory requested), rule it out, then move to the next hypothesis (memory not being freed). The user's nudge was essential in breaking the assistant out of the first hypothesis.

Input and Output Knowledge

Input knowledge required to understand this message includes: the Phase 12 split API architecture, the C++ groth16_cuda.cu implementation of prep_msm_thread, the Rust PendingProofHandle struct definition, the FFI boundary between Rust and C++, the engine's async task spawning pattern, and the memory characteristics of partition synthesis (~13 GiB per partition).

Output knowledge created by this message is twofold. First, a confirmed design invariant: the C++ b_g2_msm background thread reads Rust-owned memory through raw pointers, so the PendingProofHandle must keep that data alive until finalization. This is a documented constraint that future optimizations must respect. Second, a new investigation direction: the finalizer task's handle dropping behavior must be traced. The assistant has shifted from asking "what holds memory?" to asking "what fails to release it?"

The Thinking Process

The reasoning in this message is a model of structured debugging. The assistant starts with a concrete symptom (OOM at pw=12), formulates a hypothesis (the PendingProofHandle holds memory unnecessarily), tests it by reading the C++ code, finds the hypothesis is wrong (the memory is needed), and then formulates a new hypothesis (the handle isn't being dropped). Each step is grounded in code reading, not speculation.

The language is precise and confident: "So yes" — the bold emphasis reflects the certainty of the finding. The pivot is signaled by the contrastive "But the real leak question is different" — a clear demarcation between the hypothesis that was tested and the new direction.

The tool call to read engine.rs is not just a mechanical action; it is the logical next step in the investigation. The assistant doesn't guess about the finalizer behavior — it goes to read the code. This is the hallmark of a mature debugging approach: trace the data, don't speculate about it.

Conclusion

Message [msg 2996] is a turning point in the Phase 12 memory investigation. It transforms a vague suspicion of a leak into a targeted hypothesis about the finalizer task's handle lifecycle. By first confirming that the C++ thread legitimately needs the Rust data, the assistant rules out a red herring and narrows the search space. The subsequent investigation (documented in the chunk summary) would reveal the actual root cause: the partition semaphore releasing immediately after synthesis, allowing tasks to pile up while blocking on the single-slot GPU channel. But in this message, the groundwork is laid — the assistant has learned where not to look and where to look next. In the collaborative dance of debugging, this is the moment the music changes.