Tracing the Memory Leak: A Diagnostic Deep-Dive into Phase 12's Split GPU Proving API

Introduction

In the high-stakes world of Filecoin Proof-of-Replication (PoRep) generation, every millisecond counts and every gigabyte matters. The SUPRASEAL_C2 pipeline is a beast of engineering—a multi-layered system spanning Go orchestration, Rust FFI boundaries, C++ CUDA kernels, and GPU-accelerated number theory. When the team implemented Phase 12's "split API"—a clever architectural change that decoupled the GPU worker's critical path from the CPU-bound b_g2_msm computation—they achieved a respectable 2.4% throughput improvement, bringing proof time from 38.0s to 37.1s. But then came the mystery: increasing synthesis parallelism from 10 to 12 partition workers caused the 755 GiB system to run out of memory, despite the expected overhead being only ~26 GiB. This is the story of how the assistant diagnosed that memory leak, one grep at a time.

The Scene: A Promising Optimization Hits a Wall

The conversation leading up to message 2995 reveals a team riding the optimization frontier. Phase 12 had just been built and benchmarked successfully at partition_workers=10 (pw=10), delivering 37.1s per proof. The user, eager to push further, asked the assistant to try higher synthesis parallelism: "Try higher synthesis partition_workers in config, maybe 15/20?" ([msg 2964]). The assistant dutifully tried pw=15, which immediately OOM'd. A more conservative pw=12 also OOM'd after just 8 proofs. The user then offered two critical observations: "oom" ([msg 2977]), followed by "Maybe we're not freeing b_g2_m mem?" ([msg 2986]). This was the clue that set the diagnostic hunt in motion.

The assistant's initial response ([msg 2987]) considered the possibility that the PendingProofHandle—the Rust-side struct that holds synthesis data alive until GPU finalization completes—could be causing extra memory pressure. With two GPU workers (gw=2), two pending handles could coexist, each holding ~13 GiB of synthesis data, adding ~26 GiB. But as the user pointed out, "It's not 400GB tho, something is leaking somewhere" ([msg 2993]). The math didn't add up: pw=12 should only add ~26 GiB over pw=10, not enough to OOM a 755 GiB machine. Something more insidious was at play.

The Subject Message: A Methodical Memory Lifecycle Trace

Message 2995 is the assistant's response to this challenge. It is a classic diagnostic message: part reasoning, part hypothesis-testing, part code inspection. The assistant begins by laying out its mental model of the memory lifecycle:

Let me trace the memory lifecycle carefully. The key question: in finish_pending_proof, the Rust-side data dealloc is spawned as a std::thread gated by a static DEALLOC_MTX. If multiple finalizers complete close together, the dealloc threads queue up on that mutex and memory piles up. But that's the same as the pre-Phase 12 code.

This opening paragraph reveals the assistant's systematic approach. It identifies a potential mechanism for memory accumulation—the static mutex that gates deallocation threads. If multiple finalizers finish in rapid succession, their deallocation threads would serialize on this mutex, causing memory to pile up while earlier threads are still running. However, the assistant immediately recognizes that this mechanism existed before Phase 12 as well, so it can't be the root cause of a new memory leak.

The assistant then refines its hypothesis:

The more likely issue: the PendingProofHandle holds provers, input_assignments, aux_assignments alive until the finalizer task runs finish_pending_proof. The finalizer waits for b_g2_msm (~1.7s), then runs the async dealloc. Meanwhile the GPU worker has already picked up the NEXT partition and its synthesis data is allocated.

This is the critical insight. In the pre-Phase 12 code, the GPU worker would block until the entire proof (including b_g2_msm) completed, and then all memory was freed before the next partition was picked up. In Phase 12's split API, the GPU worker launches b_g2_msm in a background thread and immediately picks up the next partition. This means the PendingProofHandle—and its ~13 GiB of synthesis data—stays alive for the ~1.7s duration of the background b_g2_msm computation. During that window, the GPU worker allocates another ~13 GiB for the next partition. With two GPU workers, this overlap could create a temporary doubling of in-flight synthesis memory.

But even this doesn't fully explain the OOM. The assistant then asks the most penetrating question:

But wait — does the C++ prep_msm_thread (b_g2_msm) still reference the Rust-owned prover data? Let me check:

This is the moment of diagnostic pivot. The assistant realizes that if the C++ background thread still holds pointers to Rust-owned memory, then that memory cannot be freed until the thread completes. Worse, if those pointers are dangling—pointing to stack-allocated memory that goes out of scope—the system has a use-after-free bug that could cause corruption or crashes, not just memory pressure.

The assistant then executes a grep command to trace every reference to the Rust-owned data structures (provers, inp_assignment, aux_assignment) in the C++ CUDA source file groth16_cuda.cu. The output reveals that the C++ code does indeed access these structures:

55:    const Scalar* inp_assignment_data;
56:    size_t inp_assignment_size;
58:    const Scalar* aux_assignment_data;
59:    size_t aux_assignment_size;
354:        auto& p = provers[c];
356:        assert(points_l.size() == p.aux_assignment_size);
357:        assert(points_a.size() == p.inp_assignment_size + p.a_aux_popcount);

Lines 55-59 show that the Assignment struct (the C++ representation of a prover's data) contains pointers to scalar data that was originally allocated by Rust. Lines 354-361 show the C++ code dereferencing these pointers through the provers[c] array. The critical question—which the assistant is setting up to answer—is whether the provers array itself is stack-allocated in the generate_groth16_proofs_start_c function, making it a dangling reference once that function returns.

The Reasoning Process: A Window into Diagnostic Thinking

What makes this message fascinating is the visible structure of the assistant's reasoning. It follows a classic diagnostic pattern:

  1. Identify the symptom: OOM at pw=12, not at pw=10.
  2. Quantify the expected delta: ~26 GiB extra, which shouldn't OOM a 755 GiB system.
  3. Formulate hypotheses: (a) dealloc thread queuing, (b) PendingProofHandle holding data during overlap window, (c) C++ background thread referencing Rust-owned data.
  4. Test each hypothesis against known facts: (a) is dismissed because it existed pre-Phase 12, (b) is plausible but quantitatively insufficient, (c) is the most concerning and requires code inspection.
  5. Execute the inspection: grep for references in the C++ code. The assistant is essentially building a decision tree in real-time, pruning branches that don't explain the observed behavior and drilling deeper into the most promising ones. This is not just debugging—it's a form of collaborative reasoning with the user, where each hypothesis is stated explicitly and evaluated transparently.

Assumptions and Their Risks

The message makes several assumptions worth examining. First, the assistant assumes that the dealloc thread mutex queuing behavior is identical between Phase 12 and pre-Phase 12 code. While this is likely true for the mechanism itself, the pattern of finalization may have changed: in Phase 12, finalizers may complete in tighter clusters because the GPU worker no longer blocks on b_g2_msm, potentially causing more queuing. The assistant dismisses this possibility too quickly.

Second, the assistant assumes that the primary memory concern is the Rust-side data held by PendingProofHandle. But the C++ side also allocates GPU memory for the b_g2_msm computation, and if that memory isn't freed promptly, it could contribute to pressure. The grep output shows that the C++ code accesses inp_assignment_data and aux_assignment_data—these are pointers to Rust-allocated host memory, but the C++ code may also allocate GPU-side buffers that persist.

Third, the assistant's question "does the C++ prep_msm_thread still reference the Rust-owned prover data?" implicitly assumes that if the answer is no, the Rust data could be freed earlier. But the Rust side doesn't know when the C++ thread has finished using the data—that's exactly why the PendingProofHandle exists. The handle is the synchronization mechanism; it can't be freed until the C++ side signals completion.

Input Knowledge Required

To fully understand this message, the reader needs substantial background knowledge. One must understand the Groth16 proving pipeline: how a proof is constructed from multiple partitions, each requiring synthesis (circuit evaluation) followed by GPU-accelerated MSM (multi-scalar multiplication) and NTT (number-theoretic transform) operations. One must understand the Rust/C++ FFI boundary: how Rust allocates memory for provers, input_assignments, and aux_assignments, passes pointers to C++, and relies on C++ to signal when it's done. One must understand the Phase 12 split API architecture: how generate_groth16_proofs_start_c launches a background thread for b_g2_msm and returns immediately, while finish_pending_proof waits for that thread and then frees resources. And one must understand the memory footprint of each partition: ~13 GiB for synthesis data, plus SRS (44 GiB), PCE (26 GiB), and other overhead.

Output Knowledge Created

This message produces several important outputs. First, it establishes a clear hypothesis about the memory leak mechanism: the PendingProofHandle holds synthesis data alive during the b_g2_msm overlap window. Second, it identifies the critical question about C++ references to Rust-owned data—a question that will lead, in subsequent messages, to the discovery of a genuine use-after-free bug where the provers array was stack-allocated and became dangling after generate_groth16_proofs_start_c returned. Third, it provides a grep-based trace of the C++ code's data access patterns, confirming that the C++ background thread does access Rust-owned memory through the provers array.

The deeper output is methodological: this message demonstrates how to systematically diagnose a memory leak in a complex, multi-language system. The approach—quantify the expected delta, formulate and prune hypotheses, inspect the code at the FFI boundary—is transferable to any system where memory ownership crosses language boundaries.

Conclusion

Message 2995 is a turning point in the Phase 12 investigation. It marks the transition from "we have a performance improvement but it OOMs at higher parallelism" to "we need to understand the memory lifecycle at the FFI boundary." The assistant's methodical reasoning, transparent hypothesis evaluation, and targeted code inspection set the stage for the discovery of the use-after-free bug and the subsequent fixes that made Phase 12 production-ready. In the broader narrative of the SUPRASEAL_C2 optimization journey, this message exemplifies the kind of careful, collaborative debugging that separates a working optimization from a fragile one.