The Use-After-Free That Almost Broke Phase 12: A Deep Dive Into a Concurrency Bug in a GPU Proving Pipeline

Introduction

In the high-stakes world of Filecoin proof generation, where a single Groth16 proof consumes ~200 GiB of memory and takes dozens of seconds to compute, every optimization is a tightrope walk between performance and correctness. The Phase 12 split GPU proving API was designed to decouple the GPU worker's critical path from CPU post-processing by offloading the b_g2_msm computation into a background thread, allowing the GPU worker to immediately begin the next proof. But in message [msg 3016], the assistant discovered that this optimization harbored a critical use-after-free bug — a dangling reference to a stack-allocated array that could corrupt memory, crash the process, or silently produce incorrect proofs. This article examines that moment of discovery in detail, tracing the reasoning that led from a mysterious memory buildup to the identification of undefined behavior in a C++ lambda capture.

The Context: Memory Pressure After Phase 12

The story begins not with a crash, but with a puzzling memory accumulation. The assistant had just completed the Phase 12 split API implementation, achieving a clean build and a promising 37.1 seconds per proof benchmark. But when it tried to increase synthesis parallelism from partition_workers=10 to partition_workers=12, the process was killed by the OOM killer at a peak RSS of 668 GiB — dangerously close to the 755 GiB system limit. Even at pw=10, a dedicated RSS monitoring run showed memory growing monotonically from a 70 GiB baseline to a peak of 367 GiB, before eventually returning to baseline after all work completed.

This was not a memory leak in the traditional sense — the memory was eventually freed. But the accumulation during the run was severe enough to limit the system's throughput. The assistant hypothesized that the Phase 12 split API was the culprit: by holding the PendingProofHandle alive until the background b_g2_msm thread completed (~1.7 seconds), the GPU worker released synthesis data later than in the synchronous code, causing more partition allocations to be outstanding simultaneously.

The assistant considered several explanations. Perhaps the malloc_trim call was missing from the finalizer path, leaving freed pages unreclaimed. Perhaps the Rust DEALLOC_MTX was contended, causing deallocation threads to pile up. Perhaps the two independent deallocation mutexes (one in C++, one in Rust) were serializing memory reclamation in unexpected ways. Each hypothesis was methodically checked against the code, but none fully explained the magnitude of the accumulation.

The Turning Point: Tracing the C++ prep_msm_thread

The breakthrough came when the assistant decided to trace exactly what data the C++ prep_msm_thread accessed after generate_groth16_proofs_start_c returned. The thread was spawned inside the C++ function with a lambda capture [&, num_circuits] — a capture-by-reference of all local variables. The thread read extensively from the provers array: provers[c].inp_assignment_data, provers[c].aux_assignment_data, density bit-vectors, and the a, b, c NTT evaluation vectors. The provers parameter was a C-style array parameter — const Assignment<fr_t> provers[] — which in C++ decays to a pointer.

In the old synchronous path (generate_groth16_proofs_c), this was safe: the function called generate_groth16_proofs_start_c which spawned the thread, then immediately called finalize_groth16_proof_c which joined the thread. The function did not return until the thread was joined, so the provers pointer remained valid throughout. But in the Phase 12 split path, generate_groth16_proofs_start_c was designed to return immediately with a pending handle, leaving the prep_msm_thread running in the background. The function's stack frame — including the provers parameter — would be destroyed upon return.

The assistant initially focused on whether the Rust start_groth16_proof function's local provers: Vec<Assignment> was kept alive. It was — the PendingProofHandle held the Vec, and the Assignment structs' raw pointers (inp_assignment_data, aux_assignment_data) pointed into Rust-owned input_assignments and aux_assignments vectors that were also kept alive. So the underlying data was safe. The problem was subtler.

The Discovery: A Reference to a Stack Variable

In message [msg 3016], the assistant connects the final pieces. The C++ function takes const Assignment<fr_t> provers[] — a pointer to the Rust Vec's buffer. The prep_msm_thread lambda captures [&, num_circuits], which captures provers by reference. In C++, a parameter declared as const Assignment<fr_t> provers[] is adjusted to const Assignment<fr_t>* provers — a pointer. The lambda's capture &provers is therefore a reference to the pointer variable itself, which lives on the stack. When generate_groth16_proofs_start_c returns, that stack slot is reclaimed. The thread continues to dereference through this dangling reference, reading from memory that is no longer valid.

The assistant walks through the logic carefully, almost as if thinking aloud:

"Wait, actually #1 is the concern. The C++ function takes const Assignment<fr_t> provers[] which is a pointer to the Rust Vec's data. The prep_msm_thread captures &provers — a reference to the PARAMETER. In C++, const Assignment<fr_t> provers[] decays to const Assignment<fr_t>* provers, so &provers is const Assignment<fr_t>**. This is a reference to the local pointer variable on the stack. When the function returns, that stack slot is gone."

Then the assistant considers a subtle nuance: does the lambda actually need the reference, or would a value capture suffice? The thread reads provers[c] which dereferences the pointer. If the pointer value were captured by value (i.e., the thread held a copy of the pointer), the underlying buffer would still need to be valid — but the Rust Vec is kept alive by the PendingProofHandle, so that would be safe. The problem is the double indirection: the thread holds a reference to the stack variable that holds the pointer, not a copy of the pointer itself. After the function returns, that stack variable is garbage.

The assistant concludes: "This is definitely undefined behavior. The fix: we need to either copy the provers array into the groth16_pending_proof struct, or ensure the prep_msm_thread completes before the function returns. But the whole point of Phase 12 is that prep_msm_thread continues after the function returns! So we need to copy the Assignment array into the pending proof handle."

Why This Bug Was Invisible

The use-after-free had likely been present since the first Phase 12 implementation, yet the benchmarks ran successfully, producing correct proofs. This is the insidious nature of undefined behavior: it can appear to work for a long time before manifesting as a crash or corruption. Several factors contributed to its invisibility:

Stack memory persistence: The function's stack frame, though logically destroyed, may persist physically in memory for some time before being overwritten by subsequent function calls. The prep_msm_thread reads the pointer value from this location, and if the value hasn't been overwritten, the read succeeds.
The Rust Vec buffer: The actual Assignment data (the array of structs) is owned by the Rust Vec, which is kept alive by the PendingProofHandle. So even after the C++ stack variable is gone, the data it pointed to remains valid. The dangling reference only affects the pointer value, not the data itself.
Short window: The prep_msm_thread runs for approximately 1.7 seconds. If the stack memory isn't reused within that window (which is likely, given that the function returns to Rust code that then returns the pending handle up the call chain), the pointer value remains intact.
No memory safety checks: C and C++ do not provide runtime bounds checking or use-after-free detection by default. The bug operates silently within the rules of undefined behavior.

The Broader Significance

This discovery is a textbook example of the challenges inherent in Rust/C++ FFI boundaries, especially when concurrency is involved. The Rust side correctly managed lifetimes — the PendingProofHandle kept all necessary data alive. But the C++ side introduced a subtle lifetime dependency on a stack variable that Rust had no visibility into. The Rust borrow checker, which would have caught this bug immediately in pure Rust code, was powerless because the data crossed the FFI boundary into raw pointers and C++ references.

The bug also illustrates a common pattern in high-performance systems: the tension between optimization and correctness. Phase 12's entire premise was that the b_g2_msm computation could be safely deferred to a background thread, allowing the GPU worker to proceed. But this optimization introduced a lifetime constraint that was not properly propagated across the language boundary. The synchronous version of the code was safe precisely because it was less optimized — the thread was joined before the function returned, keeping the stack frame alive.

The fix — copying the provers array into the heap-allocated groth16_pending_proof struct — is straightforward but carries a cost: an extra allocation and copy of the Assignment array (which contains pointers, not the actual assignment data, so it is relatively small). More importantly, it establishes a pattern for future split APIs: any data that a background thread accesses must be owned by the handle, not borrowed from the caller's stack.

Conclusion

Message [msg 3016] captures the moment when a subtle concurrency bug was uncovered through careful reasoning about memory lifetimes across a Rust/C++ FFI boundary. The assistant's methodical approach — starting from a symptom (memory accumulation), ruling out simpler explanations, tracing the C++ thread's data dependencies, and finally identifying the dangling reference — demonstrates the depth of systems thinking required to debug high-performance GPU proving pipelines. The use-after-free bug, invisible in benchmarks and unlikely to cause immediate failure, could have led to sporadic corruption or crashes in production. Its discovery and fix were essential to making Phase 12 not just fast, but correct.