The Dangling Pointer: A Use-After-Free Bug Discovered in a Split GPU Proving Pipeline

Introduction

In the high-stakes world of zero-knowledge proof generation for Filecoin's Proof-of-Replication (PoRep), every microsecond counts and every byte of memory is precious. The opencode session captured in message 3017 represents a pivotal moment in a deep optimization campaign: the discovery of a subtle use-after-free bug in a multi-threaded C++/CUDA/Rust FFI system. This message, written by the AI assistant during a Phase 12 optimization of the SUPRASEAL_C2 Groth16 proving pipeline, documents the moment when a seemingly working optimization was revealed to contain undefined behavior—a dangling reference that could corrupt memory at any moment.

The Context: Phase 12 Split API

To understand the significance of message 3017, we must first understand what Phase 12 aimed to achieve. The SUPRASEAL_C2 pipeline generates Groth16 proofs for Filecoin storage proofs. A critical bottleneck was the b_g2_msm computation (a multi-scalar multiplication on the G2 curve), which occupied the GPU worker's critical path. Phase 12 introduced a "split API": instead of performing the b_g2_msm synchronously within the GPU worker's lock-holding region, the computation was offloaded to a background thread (prep_msm_thread). This allowed the GPU worker to release its lock earlier and begin processing the next partition, theoretically improving throughput.

The split API worked by having generate_groth16_proofs_start_c spawn a background thread and return immediately with a groth16_pending_proof handle. Later, finalize_groth16_proof_c would join the thread and complete the proof. This design required careful management of memory lifetimes across the C++/Rust boundary.

The Bug Discovery

Message 3017 is the culmination of a long debugging session. Earlier messages (2999–3016) show the assistant investigating memory growth during benchmarking. The RSS was peaking at 367 GiB during a 20-proof run with 15 concurrent jobs, far higher than expected. The assistant initially suspected a memory leak or that the PendingProofHandle was holding synthesis data unnecessarily.

The breakthrough came when the assistant traced the C++ prep_msm_thread code and realized something alarming. Let us quote the exact reasoning from the message:

"Wait — provers is a local variable on the stack in generate_groth16_proofs_start_c. It's a vector<Assignment> built from the Rust-passed pointers. The prep_msm_thread captures &provers by reference. When generate_groth16_proofs_start_c returns, the provers local goes out of scope — the reference is dangling!"

This is the core insight. The C++ function generate_groth16_proofs_start_c receives a const Assignment<fr_t> provers[] parameter—a pointer to an array of Assignment structs. The background thread captures this parameter by reference ([&, num_circuits]), meaning it holds a reference to the stack-allocated pointer variable. When the function returns, that stack frame is destroyed, and the thread is left reading from freed memory.

Deep Dive: The Use-After-Free Mechanism

The assistant's reasoning in message 3017 demonstrates a systematic approach to understanding undefined behavior. Let us trace the logic step by step.

First, the assistant examines the groth16_pending_proof struct to see what it currently holds:

struct groth16_pending_proof {
    std::thread prep_msm_thread;  // b_g2_msm still running
    msm_results results;
    ...
};

The struct holds the thread and results, but critically, it does not hold a copy of the provers array. The thread captures provers by reference from the function parameter.

Second, the assistant checks how extensively provers is used in the function body. A grep reveals that provers is referenced not just in the prep_msm_thread but also in the main GPU kernel region (lines 825–1177), including for cudaHostUnregister calls that read provers[0].a, provers[0].b, and provers[0].c. This means the entire function body depends on provers being alive.

Third, the assistant traces the function's return path. The function:

  1. Releases the GPU lock
  2. Unregisters host pages (reading provers[0].a/b/c)
  3. Moves the prep_msm_thread into the pending handle After step 3, the function returns. The provers parameter (a pointer on the stack) is gone. The background thread continues running, reading from provers[c]—a dangling reference. The assistant then makes a crucial observation about the capture semantics:
"In C++, const Assignment<fr_t> provers[] decays to const Assignment<fr_t>* provers, so &provers is const Assignment<fr_t>**. This is a reference to the local pointer variable on the stack. When the function returns, that stack slot is gone."

But the assistant also considers a more subtle point:

"But actually, the pointer VALUE is what matters, not its stack location. The thread reads provers[c] which reads *(provers + c) — the pointer value was captured when the lambda was created. Wait, no — [&] captures by reference, not by value. So the thread has a reference to the stack variable holding the pointer, and dereferences it through that reference."

This is a critical distinction. If the lambda had captured provers by value (using [=]), the pointer value would have been copied, and the thread would still have a valid address to the Rust-owned buffer. But because [&] captures by reference, the thread holds a reference to the stack slot that held the pointer. After the function returns, that stack slot is invalid. Even if the memory hasn't been overwritten yet, the behavior is undefined.

The Fix Design

Having identified the bug, the assistant immediately designs the fix:

"I need to add a std::vector<Assignment<fr_t>> provers_owned field to the struct, copy the provers array into it at the start of the function, and have the prep_msm_thread reference pp->provers_owned instead of the function parameter."

The fix involves three steps:

  1. Add a provers_owned field to the groth16_pending_proof struct. This is a std::vector<Assignment<fr_t>> that will hold a deep copy of the Assignment array.
  2. Copy the provers array at the start of generate_groth16_proofs_start_c, before spawning the background thread. This ensures the data is stable in heap memory before any concurrent access begins.
  3. Update all code paths that reference the stack provers parameter to use pp->provers_owned instead. This includes both the prep_msm_thread and the GPU kernel region code that reads provers[0].a/b/c for cudaHostUnregister. The assistant then checks the Assignment struct to understand its layout and confirm that copying is safe:
template<class Scalar>
struct Assignment {
    const uint64_t* a_aux_density;
    size_t a_aux_bit_len;
    size_t a_aux_popcount;
    const uint64_t* b_inp_density;
    size_t b_inp_bit_len;
    size_t b_inp_popcount;
    const uint64_t* b_aux_density...
};

The Assignment struct contains raw pointers (to Rust-owned memory) and size/popcount fields. Copying the struct itself is cheap—it's just copying pointer values and integers. The underlying data that the pointers reference remains owned by the Rust PendingProofHandle, which is kept alive by the Rust side until finalize_groth16_proof_c completes. This is a critical point: the Rust PendingProofHandle holds the input_assignments and aux_assignments vectors that the Assignment pointers reference. As long as the Rust handle is alive, the data is valid. The fix only needs to ensure that the C++ side has a stable copy of the Assignment struct array itself.

Assumptions and Their Validity

The assistant makes several assumptions in this message, most of which are correct:

Assumption 1: The Rust PendingProofHandle keeps the underlying data alive. This is correct. The Rust start_groth16_proof function builds a PendingProofHandle that owns the input_assignments, aux_assignments, and the provers Vec. This handle is returned to the engine and kept alive until finish_pending_proof is called, which joins the background thread and then frees the data.

Assumption 2: Copying Assignment structs is cheap. This is correct. The struct contains pointers and integers, no heap-allocated data. A memcpy-level copy suffices.

Assumption 3: The prep_msm_thread only reads from provers and does not modify it. This is correct based on the code—the thread reads density bit-vectors and assignment data to prepare the MSM, but does not modify the Assignment structs.

One subtle assumption that the assistant does not explicitly state but relies on: the Rust provers Vec and the C++ provers[] pointer point to the same memory. This is true because the Rust side passes provers.as_ptr() as *const _ to the C++ function. The C++ function receives this as const Assignment&lt;fr_t&gt; provers[]. Both sides share the same buffer.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

  1. C++ reference capture semantics: How [&amp;] in a lambda captures local variables by reference, and why capturing a function parameter by reference creates a dangling reference after the function returns.
  2. Rust FFI memory model: How Rust passes data to C++ via raw pointers, and how ownership is maintained on the Rust side while C++ holds borrowed pointers.
  3. CUDA programming patterns: The concept of cudaHostRegister/cudaHostUnregister for pinning host memory, and why the GPU kernel region needs to access provers[0].a/b/c even after the kernel has launched.
  4. The Phase 12 split API design: Understanding that the whole point of the split is to allow the background thread to outlive the function call that spawned it.
  5. Groth16 proof structure: The roles of a, b, c vectors and the b_g2_msm computation in the proof generation pipeline.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. A confirmed use-after-free bug in the Phase 12 implementation. The bug is not theoretical—the assistant has traced the exact mechanism by which undefined behavior occurs.
  2. A concrete fix plan with three specific code changes: add provers_owned to the struct, copy the array, and update all references.
  3. A deeper understanding of memory ownership across the Rust/C++ boundary. The Assignment structs are "thin" wrappers containing pointers to Rust-owned data, and their lifetime must be managed carefully.
  4. A diagnostic methodology: The assistant demonstrates how to trace undefined behavior by examining lambda capture semantics, stack lifetimes, and pointer ownership. This methodology is reusable for similar FFI bugs.
  5. A validation of the Phase 12 approach: The bug is in the implementation details, not the architectural design. The split API concept remains sound—it just needs proper memory management.

The Thinking Process

The assistant's reasoning in message 3017 is a masterclass in systematic debugging. The thought process follows a clear arc:

  1. Hypothesis formation: "I need to add a provers_owned field..." — the assistant immediately identifies the fix based on the bug analysis from previous messages.
  2. Verification through code reading: The assistant reads the Assignment struct definition to confirm that copying is safe and to understand what data needs to be preserved.
  3. Scope analysis: The assistant considers not just the prep_msm_thread but also the GPU kernel region code that reads provers[0].a/b/c. This shows an understanding that the fix must be comprehensive.
  4. Ownership reasoning: The assistant traces the chain of ownership: Rust PendingProofHandle → Rust provers Vec → C++ provers[] parameter → background thread. Each link in this chain must be valid for the data to be safe.
  5. Semantic precision: The assistant carefully distinguishes between capturing a pointer by value vs. by reference, and understands the subtle difference between the pointer value being valid vs. the stack slot holding the pointer being valid.

Broader Implications

This bug is a classic example of a class of errors that plague FFI systems: lifetime mismatches across language boundaries. In Rust, the borrow checker would prevent this bug at compile time—you cannot spawn a thread that holds a reference to a stack-local variable and then let that variable go out of scope. But in C++, there is no such protection, and the bug manifests as undefined behavior that may appear to "work" for years before causing a crash.

The fact that the bug was discovered through memory pressure analysis (the 367 GiB RSS peak) rather than a crash is telling. Use-after-free bugs often go undetected because the freed memory is not immediately reused. In this case, the memory pressure investigation led the assistant to trace the lifetime of every buffer, which revealed the dangling reference.

The fix—copying the Assignment array into the pending proof struct—is a textbook solution. It converts a borrowed pointer into an owned copy, ensuring that the background thread always accesses stable memory regardless of when the parent function returns.

Conclusion

Message 3017 captures a critical moment in a complex optimization effort. The discovery of the use-after-free bug in the Phase 12 split API could have derailed the entire optimization, but the assistant's systematic analysis and clear fix plan turned it into a manageable engineering problem. The message demonstrates the importance of careful lifetime management in FFI systems, the value of tracing memory ownership across language boundaries, and the power of systematic debugging.

The fix described in this message—adding provers_owned to the groth16_pending_proof struct—would eventually be implemented and committed, allowing Phase 12 to deliver its intended performance improvement. But the real value of this message lies in the reasoning process it documents: a careful, methodical investigation that turned a mysterious memory growth problem into a concrete, fixable bug.