The Use-After-Free That Almost Wasn't: Tracing a Concurrency Bug in Phase 12's Split GPU Proving API

Introduction

In the high-stakes world of Filecoin proof generation, where a single Groth16 proof consumes ~200 GiB of memory and spans CPU synthesis, GPU kernels, and multi-threaded MSM computations, the difference between a correct optimization and a silent data corruption bug can be measured in milliseconds—or in corrupted proofs that silently invalidate a storage provider's entire batch. Message 3021 captures a pivotal moment in the development of Phase 12 of the cuzk SNARK proving engine: the discovery and analysis of a use-after-free bug introduced by the split GPU proving API, and the careful reasoning required to fix it correctly.

This message is the intellectual hinge point of a debugging session that began with memory pressure diagnosis and escalated into a concurrency correctness audit. The assistant, having just edited the C++ CUDA source to copy the provers array into a heap-allocated struct, pauses to verify that the fix is sound—and in doing so, reveals a deep understanding of C++ reference semantics, lambda capture mechanics, and the subtle ways that seemingly correct code can hide undefined behavior.

The Message

The assistant writes:

Now the provers name is shadowed — all code below will use pp->provers_owned via the reference. The prep_msm_thread's [&, num_circuits] capture will now capture a reference to this local reference, which resolves to pp->provers_owned — a heap-allocated vector that outlives the function.

Wait, actually there's a subtlety. The const auto& provers = pp->provers_owned; creates a local reference on the stack. The lambda captures [&] which captures provers by reference — that's a reference to the local reference variable, which is on the stack. When the function returns, the stack reference variable is gone.

But in C++, capturing a reference variable by [&] actually captures the referent, not the reference itself. So the lambda sees the same pp->provers_owned vector. Actually no — [&] captures the variable's address. For a reference variable, &provers is the address of the referent (&pp->provers_owned). So the lambda effectively holds a pointer to pp->provers_owned, which lives on the heap. This is safe.

Actually, let me be more precise. In C++14+, when a reference variable is captured by [&], the lambda uses the original reference directly. Since provers is a reference to pp->provers_owned, and pp is heap-allocated and lives until delete pp in the finalizer, this is safe. The thread accesses provers[c] which resolves to pp->provers_owned[c].

But there's another subtlety — provers is now a std::vector<Assignment<fr_t>> instead of const Assignment<fr_t>*. Most usages like provers[c] work the same, but provers[0].a etc. should work fine too since vector::operator[] returns a reference.

Let me now also check if there are similar issues with other captured references from the function parameters. The r_s and s_s are already copied into pp->r_s_owned/pp->s_s_owned (line 1185-1186). What about points_h, points_l, etc.? Those come from srs which is passed by reference — the SRS is long-lived (managed by SrsManager).

Now rebuild. First clear the supraseal-c2 build cache: ``bash rm -rf /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-* ``

Context: The Phase 12 Split API

To understand why this message exists, we must trace the architecture of Phase 12. The cuzk proving engine processes Groth16 proofs through a pipeline: CPU synthesis generates partition data (the a, b, c NTT evaluation vectors, ~12 GiB per partition), then a GPU worker runs the core proof computation. Within that GPU computation, the b_g2_msm (multi-scalar multiplication on the G2 curve) is a significant cost—approximately 1.7 seconds per proof.

Before Phase 12, the GPU worker held the partition's synthesis data alive for the entire duration of gpu_prove, including the b_g2_msm. Phase 12 split the API into start_groth16_proof (which returns a PendingProofHandle) and finish_pending_proof (which joins the background b_g2_msm thread and assembles the final proof). This allowed the GPU worker to release the synthesis data earlier and move on to the next partition, improving throughput by ~2.4%.

But this split introduced a critical concurrency bug. The C++ function generate_groth16_proofs_start_c spawns a prep_msm_thread that runs b_g2_msm in the background. This thread captures all local variables by reference ([&, num_circuits]), including the provers array—a C-style array parameter (const Assignment<fr_t> provers[]) that is actually a pointer to the Rust-owned Assignment structs. In the original synchronous API, the function did not return until the thread was joined, so the stack remained valid. In the split API, generate_groth16_proofs_start_c returns immediately, the C++ stack frame is destroyed, and the provers pointer (a stack-local variable) becomes a dangling reference.

The assistant discovered this bug in the preceding messages ([msg 3013] through [msg 3018]) while investigating why memory was accumulating during Phase 12 benchmarks. The memory pressure investigation—which showed RSS peaking at 367 GiB with pw=10 and 15 concurrent jobs—led the assistant to trace the lifetime of the provers data. What started as a memory optimization question ("can we free the synthesis data earlier?") turned into a correctness audit when the assistant realized that the prep_msm_thread was reading from memory that might already be freed.

The Reasoning Process

The message opens with the assistant confirming that the edit just applied will work: "Now the provers name is shadowed — all code below will use pp->provers_owned via the reference." This is the confident, surface-level analysis. But then the assistant catches itself: "Wait, actually there's a subtlety."

This self-interruption is the hallmark of expert-level reasoning. The assistant has just realized that the fix might not be as straightforward as it first appeared. The const auto& provers = pp->provers_owned; declaration creates a reference variable on the stack. The prep_msm_thread lambda captures [&]—by reference. If the lambda captures the reference variable provers by reference, then when the function returns and the stack frame is destroyed, the reference variable is gone. The lambda would be holding a dangling reference to a stack-local reference.

The assistant then walks through three layers of analysis:

First intuition: "capturing a reference variable by [&] actually captures the referent, not the reference itself." This is a common mental model: references are aliases, so capturing a reference by reference should resolve to the underlying object.
Second thought: "Actually no — [&] captures the variable's address. For a reference variable, &provers is the address of the referent (&pp->provers_owned). So the lambda effectively holds a pointer to pp->provers_owned, which lives on the heap." This is a more precise model: [&] captures the address of each variable, and for a reference variable, taking its address yields the address of the referent.
Third, definitive analysis: "In C++14+, when a reference variable is captured by [&], the lambda uses the original reference directly. Since provers is a reference to pp->provers_owned, and pp is heap-allocated and lives until delete pp in the finalizer, this is safe." The assistant is essentially performing a formal verification of the fix, considering three different models of C++ lambda capture semantics and concluding that all three yield the same result: the background thread accesses pp->provers_owned, which is heap-allocated and outlives the function.

Assumptions and Potential Mistakes

The assistant makes several assumptions that deserve scrutiny:

Assumption 1: pp outlives the thread. The assistant states that pp is "heap-allocated and lives until delete pp in the finalizer." This is correct by design—the groth16_pending_proof struct is allocated with new in generate_groth16_proofs_start_c and deleted in finalize_groth16_proof_c, which joins the thread first. So the struct's lifetime is guaranteed to exceed the thread's.

Assumption 2: vector::operator[] is equivalent to array indexing. The assistant notes that provers is now a std::vector<Assignment<fr_t>> instead of const Assignment<fr_t>*, and that provers[c] and provers[0].a should work the same. This is correct—std::vector::operator[] returns a reference to the element at the given index, and vector elements are stored contiguously, so pointer arithmetic on &provers[0] would also work. However, there is a subtle difference: the old code passed provers as a raw pointer, and some usages might rely on pointer arithmetic (e.g., provers + c). The assistant doesn't check for this, but since all usages in the code use array subscript notation (provers[c]), the vector behaves identically.

Assumption 3: The SRS is long-lived. The assistant checks whether points_h, points_l, etc. (which come from the SRS) are safe, concluding that "the SRS is long-lived (managed by SrsManager)." This is a reasonable assumption given the architecture, but it's not verified in this message. A thorough audit would confirm that the SRS object's lifetime exceeds all pending proof handles.

Potential mistake: The r_s and s_s arrays. The assistant notes that these are "already copied into pp->r_s_owned/pp->s_s_owned." But this copy happens at lines 1185-1186, which is after the prep_msm_thread is spawned. If the thread reads r_s or s_s before the copy completes... but actually, the thread doesn't read r_s/s_s—those are used in the GPU kernel region, which completes before the function returns. The thread only reads provers. So this is safe.

Input Knowledge Required

To fully understand this message, the reader needs:

C++ lambda capture semantics: The difference between [&] (capture by reference) and [=] (capture by value), and how reference variables interact with lambda captures. The assistant's analysis hinges on the precise behavior of [&] when applied to a reference variable.
C++ reference semantics: That a reference is an alias, and &ref_var yields the address of the referent, not the reference itself.
The architecture of Phase 12: The split API design, the groth16_pending_proof struct, the prep_msm_thread, and the lifetime relationships between the Rust PendingProofHandle, the C++ heap-allocated groth16_pending_proof, and the background thread.
The memory pressure investigation: The preceding messages established that RSS was growing to 367 GiB, that the provers data (the Assignment structs with pointers into Rust-owned vectors) was being held alive longer than necessary, and that the root cause was the split API's deferred finalization.
The use-after-free bug: That the prep_msm_thread captured &provers where provers was a C function parameter (a stack pointer), and that this pointer became dangling when the function returned.

Output Knowledge Created

This message creates several pieces of knowledge:

Verification of the fix: The assistant confirms that the provers_owned copy in the heap-allocated struct is safe, and that all three possible interpretations of C++ lambda capture semantics lead to the same conclusion.
A checklist for similar issues: The assistant systematically checks other captured references (r_s, s_s, points_h, points_l) to ensure they are not similarly vulnerable. This establishes a pattern for auditing concurrency fixes.
A build command: The rm -rf of the supraseal-c2 build cache ensures a clean rebuild, preventing stale object files from masking compilation errors.
Documentation of the reasoning process: The message itself serves as documentation of why the fix is correct, which is valuable for future maintainers who might wonder why the provers_owned field exists.

The Broader Significance

This message is notable not just for the bug it fixes, but for the reasoning methodology it demonstrates. The assistant does not simply apply the fix and move on. It pauses to verify the fix's correctness through multiple mental models, catches a potential subtlety, resolves it through careful analysis, and then extends the audit to other parameters.

This is the difference between a superficial fix and a robust one. A less careful developer might have made the edit, observed that the code compiles and the benchmarks pass, and declared victory. The assistant instead asks: "Is this actually safe? Let me trace through the C++ semantics carefully." This is the kind of reasoning that prevents subtle, intermittent, production-only crashes that are nearly impossible to debug.

The message also illustrates the challenges of mixed-language systems programming. The bug originated at the Rust/C++ FFI boundary, where Rust-owned memory is passed as raw pointers to C++ code. The C++ code then spawns a thread that reads from those pointers. The Rust side has no way to know that the C++ thread is still running after the function returns—the FFI function's return is the signal that Rust considers the operation complete. This mismatch in lifetime expectations is a classic source of bugs in systems that span multiple languages.

Conclusion

Message 3021 is a masterclass in concurrent systems debugging. It captures the moment when a developer, having just applied a fix for a use-after-free bug, pauses to verify that the fix is sound through multiple layers of analysis. The assistant's self-correction—from confident assertion to "wait, there's a subtlety" to definitive resolution—reveals the iterative, self-critical thinking that characterizes expert-level work in systems programming.

The fix itself is elegant: copy the stack-local array into a heap-allocated struct that outlives the background thread. But the reasoning behind the fix—the careful tracing of C++ lambda capture semantics, the audit of other captured references, the verification that std::vector::operator[] behaves identically to raw pointer arithmetic—is what makes this message worth studying. It demonstrates that in high-performance concurrent systems, correctness is not a property of the code alone, but of the reasoning that produced it.