The Dangling Reference: A Use-After-Free Bug Discovered in Phase 12's Split GPU Proving API

Introduction

In the high-stakes world of Groth16 proof generation for Filecoin's Proof-of-Replication (PoRep), every millisecond counts and every byte of memory is precious. The Phase 12 optimization—a split GPU proving API designed to offload the b_g2_msm computation from the GPU worker's critical path—had just been implemented and benchmarked, showing a modest 2.4% throughput improvement to 37.1 seconds per proof. But something was wrong. When the assistant increased synthesis parallelism from 10 to 12 partition workers, the system ran out of memory. The user's response—"It's not 400GB tho, something is leaking somewhere"—sent the assistant on a forensic deep-dive that would uncover not just a memory accumulation problem, but a critical concurrency bug lurking in the C++ CUDA code.

Message [msg 3011] captures the moment of discovery: the assistant, while tracing whether Rust-owned memory must remain alive during the background b_g2_msm thread, realizes that the C++ provers array itself—a stack-local variable—is captured by reference in a background thread that outlives the function that created it. This is a textbook use-after-free bug, and its discovery reshapes the investigation from a performance-tuning exercise into a correctness-critical fix.

The Investigation That Led to the Discovery

The conversation leading up to message [msg 3011] is a masterclass in systematic debugging under pressure. The Phase 12 split API had been committed to the feat/cuzk branch, and initial benchmarks were promising: 37.1 seconds per proof at pw=10 (partition workers). But when the assistant tried pw=12 and pw=15, the 755 GiB system hit OOM conditions. The user's intuition that "something is leaking" pushed back against the assistant's initial hypothesis that the extra memory pressure was simply from PendingProofHandle holding synthesis data alive during the ~1.7 second b_g2_msm wait.

The assistant's response was methodical. In [msg 2994], it began tracing the memory lifecycle, checking whether PendingProofHandle's Rust-side data was actually being freed after finalization. It examined the async dealloc threads, the malloc_trim calls, and the dual dealloc mutexes (one in C++, one in Rust). In [msg 2995], it ran a grep to check whether the C++ prep_msm_thread still referenced Rust-owned prover data after start_groth16_proof returned. The grep revealed that the thread reads prover[c].inp_assignment_data and prover[c].aux_assignment_data—raw pointers into Rust's input_assignments and aux_assignments vectors.

This led the assistant to a correct but incomplete conclusion in [msg 2996]: "So yes, the b_g2_msm thread reads Rust-owned memory. The Rust data MUST stay alive until the prep_msm_thread completes." It then verified that the PendingProofHandle correctly kept this data alive by tracing the finalizer task flow in engine.rs. Everything appeared structurally sound.

But the user's skepticism was warranted. The assistant ran a live RSS monitoring experiment in [msg 3007]-[msg 3009], starting a daemon, submitting 20 proofs, and sampling RSS every 5 seconds. The results were alarming: RSS grew monotonically from a 70 GiB baseline to a peak of 367 GiB, stabilizing around 300-350 GiB during the run, before returning to 71 GiB after completion. There was no permanent leak, but memory was accumulating dramatically during operation. With pw=10 and 15 concurrent jobs, the system was using nearly half its 755 GiB capacity just for intermediate data.

The Critical Insight

Message [msg 3011] represents the turning point. The assistant, having confirmed the memory accumulation pattern, returns to the C++ CUDA source to examine exactly what the prep_msm_thread captures. The key passage from the message reads:

The prep_msm_thread captures [&, num_circuits] — ALL references, including provers (the C stack array of Assignment<fr_t> pointers). The provers array contains pointers to Rust-owned data: inp_assignment_data, aux_assignment_data, and a_aux_density, b_inp_density, b_aux_density.

The assistant then traces exactly what the thread reads:

But wait — provers is a local variable on the stack in generate_groth16_proofs_start_c. It's a vector<Assignment> built from the Rust-passed pointers. The prep_msm_thread captures &provers by reference. When generate_groth16_proofs_start_c returns, the provers local goes out of scope — the reference is dangling!

This is the critical realization. The assistant had been focused on whether the Rust-owned data (the actual assignment vectors) needed to stay alive. But the more immediate problem was that the C++ wrapper array—the vector<Assignment> constructed locally in generate_groth16_proofs_start_c—was itself a stack-local variable. The prep_msm_thread captured it by reference ([&, num_circuits]), meaning the thread held a reference to a local variable that would be destroyed when the function returned.

The assistant then runs a grep on the Rust FFI side (lib.rs) to trace the full chain of how provers flows from Rust through the FFI boundary into C++, confirming the architecture.

Why This Bug Matters

This use-after-free bug is particularly insidious for several reasons. First, it's a race condition: the background prep_msm_thread runs concurrently with the function's return. Whether the thread accesses the dangling reference before or after the stack memory is reused depends on timing, making the bug intermittent and hard to reproduce. Second, the provers array itself is small (a vector of Assignment structs, each holding pointers to larger data elsewhere), so the memory corruption might not cause an immediate crash—it could silently corrupt the b_g2_msm computation, producing incorrect proofs. Third, the bug was introduced by the Phase 12 split itself: before the split, b_g2_msm ran synchronously within generate_groth16_proofs_start_c, so the provers local was alive for the entire computation.

The message demonstrates a crucial shift in the assistant's mental model. Earlier, the assistant had been thinking about memory accumulation—the idea that PendingProofHandle held synthesis data longer than necessary, causing RSS to grow during sustained operation. But the user's pushback ("It's not 400GB tho") correctly identified that the scale of the problem didn't match a simple accumulation hypothesis. The use-after-free bug is a different class of problem: it's a correctness issue that could produce silently corrupted proofs, and it exists independently of the memory pressure question.

Assumptions and Their Corrections

The assistant made several assumptions during the investigation that were progressively refined. Initially, it assumed the OOM at pw=12 was simply due to the extra ~26 GiB from two additional concurrent syntheses, pushing a system already near its limit. The RSS monitoring disproved this: even at pw=10, RSS peaked at 367 GiB, far above what a simple linear model would predict.

The assistant also initially assumed that the PendingProofHandle's data retention was the primary mechanism for increased memory pressure. While this was partially correct—the handle does hold ~13 GiB of synthesis data per partition during the ~1.7s b_g2_msm wait—the use-after-free bug represents a more fundamental issue. The fact that the prep_msm_thread captures a dangling reference means the entire design of the split API's data ownership model needs re-examination.

The user's role in this investigation was critical. By insisting "something is leaking somewhere" rather than accepting the assistant's initial explanation, the user drove the investigation deeper. The assistant's response demonstrates intellectual honesty: rather than defending its initial hypothesis, it pivoted to empirical measurement (the RSS monitoring experiment) and then to deeper code analysis.

Input and Output Knowledge

To understand this message, the reader needs knowledge of several layers of the system architecture. The Phase 12 split API divides proof generation into start_groth16_proof (which launches GPU kernels and spawns a background prep_msm_thread for b_g2_msm) and finish_pending_proof (which joins the background thread and completes the proof). The C++ generate_groth16_proofs_start_c function receives Assignment<fr_t> pointers from Rust via FFI, constructs a local vector<Assignment> on the stack, and spawns a thread that captures this vector by reference. The Rust PendingProofHandle is designed to keep the underlying data (provers, input_assignments, aux_assignments) alive until finalization, but it has no control over C++ stack lifetimes.

The output knowledge created by this message is the discovery of a critical concurrency bug that must be fixed before Phase 12 can be considered production-ready. The fix requires either: (a) copying the provers array into heap memory that outlives the function call, (b) joining the prep_msm_thread before generate_groth16_proofs_start_c returns (which would defeat the purpose of the split API), or (c) restructuring the data ownership so that the C++ background thread owns its copy of the provers data. The subsequent chunk (Chunk 0 of Segment 30) reveals that the assistant ultimately implemented option (a): copying the provers array into the heap-allocated groth16_pending_proof struct as provers_owned.

The Thinking Process Revealed

The assistant's reasoning in this message reveals a sophisticated debugging methodology. It begins with a hypothesis ("the C++ thread reads Rust-owned memory"), confirms it with evidence from the code, then immediately questions whether that's the only concern. The "But wait" transition is the hallmark of a careful engineer: having confirmed one thing, the assistant immediately checks for a deeper issue.

The structure of the message is notable. It starts with a declarative summary of findings, then presents specific evidence (the capture list [&, num_circuits], the specific fields read by the thread), then delivers the critical insight about stack lifetime, and finally runs a cross-reference grep on the Rust FFI to complete the picture. This alternation between analysis and verification is characteristic of the assistant's approach throughout the conversation.

The message also demonstrates the assistant's willingness to revise its understanding. Earlier messages had focused on whether the Rust data was properly retained; this message reveals that even if the Rust data is perfectly managed, the C++ side has its own independent lifetime bug. The assistant doesn't try to minimize the discovery or rationalize it away—it simply states the facts and lets the evidence speak.

Conclusion

Message [msg 3011] is a pivotal moment in the Phase 12 investigation. It transforms the conversation from a memory-pressure tuning exercise into a correctness-critical debugging session. The discovery of the use-after-free bug—the prep_msm_thread capturing a dangling reference to a stack-local provers array—explains why the system's behavior was unpredictable at higher partition worker counts. More importantly, it demonstrates that in complex systems with multiple language boundaries (Rust↔C++↔CUDA) and concurrent execution, correctness must be verified at every interface, not just within each language's ownership model.

The message also serves as a testament to the value of empirical investigation. The user's insistence that "something is leaking" drove the assistant to run real RSS measurements rather than relying on mental models alone. Those measurements revealed a memory accumulation pattern that couldn't be explained by the assistant's initial hypotheses, forcing a deeper examination of the C++ code that ultimately uncovered the dangling reference. In the high-performance computing world, where every optimization pushes against hardware limits, the most dangerous bugs are often the ones that hide behind plausible explanations.