The Use-After-Free That Nearly Broke Phase 12: A Case Study in Asynchronous GPU Proving

Introduction

In the high-stakes world of Filecoin proof generation, every millisecond counts and every byte of memory is precious. The SUPRASEAL_C2 Groth16 proving pipeline, responsible for generating zk-SNARK proofs for Filecoin's Proof-of-Replication (PoRep), operates at an extraordinary scale: peak memory footprints approaching 200 GiB, multi-threaded CPU synthesis across dozens of partitions, and GPU kernels that must coordinate access to a shared device. When the assistant set out to implement Phase 12 of the cuzk optimization project — a split GPU proving API designed to offload the b_g2_msm computation from the GPU worker's critical path — the goal was a modest but meaningful throughput improvement. What emerged instead was a cautionary tale about the subtle dangers of asynchronous programming across language boundaries, culminating in the discovery of a critical use-after-free bug that could have silently corrupted proofs or crashed the daemon in production.

The Subject Message

The message under analysis is brief and seemingly mundane:

I need to add the provers copy right after allocating pp (line 376), and then create a local reference alias that shadows the function parameter. All subsequent code using provers[c] will use the heap-allocated copy: [edit] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu Edit applied successfully.

This single message, [msg 3020], represents the culmination of a lengthy and painstaking debugging journey. It is the moment when the assistant, having traced a subtle concurrency bug through layers of Rust FFI and C++ CUDA code, applies the definitive fix. The message itself is terse — a statement of intent followed by a confirmation that an edit was applied — but the reasoning compressed into these few lines draws on deep knowledge of memory lifetimes, C++ lambda capture semantics, Rust ownership models, and the specific architecture of the Groth16 proving pipeline.

The Bug: A Race Against Memory

The story begins with Phase 12's core innovation. In the original synchronous proving pipeline, the function generate_groth16_proofs_c would call generate_groth16_proofs_start_c, which spawned a background thread (prep_msm_thread) to perform the b_g2_msm computation, and then immediately called finalize_groth16_proof_c, which joined the thread. The function did not return until the thread completed, so all memory references remained valid throughout.

Phase 12 split this into two phases: start and finalize. The start function would spawn the prep_msm_thread and return immediately with a groth16_pending_proof handle, allowing the GPU worker to move on to the next partition while the b_g2_msm computation continued in the background. This promised a ~2.4% throughput improvement by hiding the ~1.7 seconds of b_g2_msm latency behind the GPU worker's next task.

But this architectural change introduced a subtle violation of memory safety. The prep_msm_thread was created with a lambda capture [&, num_circuits] — capturing all local variables by reference. Among those variables was provers, the function parameter of type const Assignment<fr_t> provers[]. In C++, array parameters decay to pointers, so provers was a local pointer variable on the stack. The lambda captured a reference to this stack variable, not a copy of the pointer value. When generate_groth16_proofs_start_c returned, the stack frame was destroyed, and the background thread was left holding a dangling reference to a now-invalid memory location.

The Discovery: From Memory Pressure to Undefined Behavior

The bug was discovered not through a crash or a corrupted proof, but through a memory pressure investigation. The assistant had been benchmarking Phase 12 and noticed alarming RSS growth: starting at 70 GiB, peaking at 367 GiB during a 20-proof run, and only returning to baseline after all work completed. The initial hypothesis was that the PendingProofHandle was holding synthesis data alive longer than necessary — the Rust input_assignments and aux_assignments vectors, each containing the massive NTT evaluation vectors (~12 GiB per partition), were being kept alive until the finalizer ran, rather than being freed immediately after start_groth16_proof returned.

The assistant traced this through multiple messages ([msg 3010] through [msg 3017]), systematically narrowing the scope of the investigation. The first hypothesis — that the Rust-side data could be freed early — was quickly disproven by examining the C++ code. The prep_msm_thread actively read from provers[c].inp_assignment_data, provers[c].aux_assignment_data, and the density bit vectors. The Rust data had to stay alive until the background thread finished.

But then the assistant noticed something more alarming. In [msg 3011], the realization hit:

But wait — provers is a local variable on the stack in generate_groth16_proofs_start_c. It's a vector<Assignment> built from the Rust-passed pointers. The prep_msm_thread captures &provers by reference. When generate_groth16_proofs_start_c returns, the provers local goes out of scope — the reference is dangling!

This was not merely a memory pressure issue. This was a use-after-free bug — undefined behavior that could manifest as corrupted proofs, segmentation faults, or silent data corruption. The fact that it had not crashed during testing was likely due to the stack memory not being immediately overwritten, a classic instance of "working by coincidence."

The Reasoning: Tracing the Lifetime Chain

The assistant's thinking process in the preceding messages reveals a meticulous forensic analysis. The chain of reasoning went through several stages:

Stage 1: Memory pressure diagnosis. The assistant benchmarked Phase 12 and observed RSS growing to 367 GiB ([msg 3009]). This was higher than expected, and the assistant correctly identified that the PendingProofHandle was keeping synthesis data alive during the b_g2_msm wait.

Stage 2: The "free early" hypothesis. The assistant considered whether the Rust data could be freed immediately after start_groth16_proof returned ([msg 3010]). This would have been the ideal fix — reducing peak memory without changing the C++ code.

Stage 3: Disproof via code reading. By reading the C++ CUDA source ([msg 3011]), the assistant confirmed that the prep_msm_thread actively reads from the provers array. The Rust data could not be freed early.

Stage 4: The use-after-free discovery. While examining the capture semantics of the C++ lambda, the assistant realized the deeper problem: not only was the Rust data potentially freed, but the C++ provers parameter itself was a stack variable whose lifetime ended when the function returned ([msg 3013]). The lambda captured &provers — a reference to the local pointer, not a copy of the pointer value.

Stage 5: Verification through grep. The assistant confirmed the extent of provers usage throughout the function body ([msg 3014]–[msg 3015]), finding that provers was accessed not just in the background thread but also in the main GPU kernel region (lines 825–1177) and in the cudaHostUnregister calls (lines 1175–1177). Any code path that touched provers after the function returned was potentially reading from freed memory.

Stage 6: Design of the fix. The assistant examined the groth16_pending_proof struct ([msg 3017]) and the Assignment struct ([msg 3018]) to determine the correct fix: copy the provers array into the heap-allocated pending proof struct, creating a stable copy whose lifetime matched the background thread's execution.

The Fix: Copying Provers into Stable Memory

The fix, applied in [msg 3018] and [msg 3020], was conceptually simple but required careful attention to detail. The assistant added a std::vector<Assignment<fr_t>> provers_owned field to the groth16_pending_proof struct. Early in generate_groth16_proofs_start_c, immediately after allocating the pending proof handle, the provers array would be copied into this vector. A local reference alias (auto& provers_local = pp->provers_owned;) would shadow the function parameter, ensuring that all subsequent code — including the background thread, the GPU kernel region, and the cudaHostUnregister calls — accessed the heap-allocated copy rather than the stack-local parameter.

The key insight was that Assignment<fr_t> is a trivially copyable POD struct containing only pointers and sizes. Copying the array was cheap — it was the pointer values that needed to be stable, not the underlying data they pointed to. The Rust-side data (input_assignments, aux_assignments, density bit vectors) was still owned by the PendingProofHandle and would remain valid until the finalizer ran. The fix merely ensured that the C++ code's reference to the Assignment structs themselves was stable.

Assumptions and Their Validity

The assistant made several assumptions during this investigation, most of which were validated by careful code reading:

Assumption 1: The provers parameter is a stack-local pointer. This is correct. In C++, const Assignment<fr_t> provers[] decays to const Assignment<fr_t>* provers, a local pointer variable on the function's stack frame. When the function returns, this stack slot is invalid.

Assumption 2: The lambda capture [&] captures provers by reference. This is correct. The C++ lambda capture-by-reference semantics mean the lambda holds a reference to the local variable, not a copy of its value. After the function returns, dereferencing this reference is undefined behavior.

Assumption 3: The Rust provers Vec is also freed when start_groth16_proof returns. This requires more careful analysis. The Rust start_groth16_proof function builds a local Vec<Assignment<S>> and passes a pointer to its buffer to the C++ function. When start_groth16_proof returns, the Vec is dropped and its buffer is freed. However, the PendingProofHandle returned by start_groth16_proof holds the input_assignments and aux_assignments vectors — but not the provers Vec itself. So the C++ code's provers pointer (pointing into the Rust Vec's buffer) does become dangling when start_groth16_proof returns. This compounds the bug: even if the C++ stack variable were somehow valid, the memory it pointed to would be freed.

Assumption 4: Assignment<fr_t> is trivially copyable. The assistant verified this by reading the struct definition ([msg 3017]), confirming it contains only pointers (const uint64_t*, const fr_t*, etc.) and size fields (size_t). Shallow copying is safe because the underlying data is owned by the Rust side and has a longer lifetime.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning multiple domains:

C++ lambda capture semantics. The distinction between capture-by-reference ([&]) and capture-by-value ([=]) is critical. The bug exists precisely because [&] captures a reference to the stack variable, not a copy of the pointer.

Rust FFI and ownership model. The Rust side builds a Vec<Assignment<S>> and passes a pointer to its buffer to C++. The Rust ownership model means this buffer is freed when the Vec is dropped, which happens when start_groth16_proof returns. The C++ code has no way to extend the lifetime of this buffer.

CUDA GPU proving pipeline architecture. Understanding that provers is accessed in multiple contexts — the background prep_msm_thread, the GPU kernel region, and the cudaHostUnregister calls — is essential to ensuring the fix covers all access paths.

The Groth16 proof structure. The Assignment struct contains density bit vectors and scalar pointers that represent the circuit's input and auxiliary assignments. These are the raw material from which the proof's A, B, and C points are computed.

Output Knowledge Created

This message and its surrounding investigation produced several forms of output knowledge:

A documented use-after-free bug in the Phase 12 split API. The bug was not hypothetical — it was a real undefined behavior that could have caused crashes or corruption in production. The investigation produced a clear understanding of the root cause and the fix.

A generalizable pattern for async FFI safety. The lesson — that stack-allocated data passed across a language boundary must be copied into stable memory before spawning background threads — applies to any multi-language asynchronous system.

A validated fix with minimal performance impact. Copying the Assignment array is cheap (a shallow copy of POD structs), and the fix does not affect the performance characteristics of the split API. The throughput improvement of Phase 12 (~2.4%) is preserved.

A revised understanding of the memory pressure problem. The use-after-free fix does not address the memory accumulation issue (RSS peaking at 367 GiB). That remains a separate concern, to be addressed by reducing per-partition memory footprint or adjusting the synthesis/GPU channel capacity.

Conclusion

The subject message [msg 3020] appears, on its surface, to be a routine edit confirmation. But it represents the resolution of a complex debugging journey that spanned memory pressure analysis, Rust FFI semantics, C++ lambda capture rules, and CUDA GPU programming. The assistant's systematic approach — starting from a high-level performance observation, drilling down through code reading, identifying a subtle lifetime violation, and designing a minimal fix — exemplifies the kind of cross-disciplinary systems thinking required to build reliable high-performance proving infrastructure. The fix itself is elegant: a single std::vector copy that transforms a dangling reference into a stable, owned allocation, ensuring that the background thread always accesses valid memory. In doing so, the assistant not only fixed a critical bug but also produced a documented case study in the dangers of asynchronous programming across language boundaries — knowledge that will inform future optimization work on the cuzk proving engine.