The Edit That Fixed a Phantom: Tracing a Use-After-Free Through C++ Reference Semantics

[assistant] [edit] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu
Edit applied successfully.

At first glance, this message is unremarkable — a routine confirmation that a file edit was applied. But in the context of the opencode session, this single line represents the resolution of one of the most subtle and dangerous bugs in concurrent systems programming: a use-after-free error introduced by restructuring a synchronous GPU proving API into an asynchronous one. The message is the final commit of a fix that required the assistant to trace C++ reference semantics through lambda captures, stack lifetimes, and the interaction between Rust's memory management and CUDA kernel execution. To understand why this edit was written, we must reconstruct the chain of reasoning that led to it.

The Context: Phase 12's Split API

The session was in the midst of implementing Phase 12 of a GPU proving optimization for Filecoin's Groth16 proof generation pipeline. The core idea of Phase 12 was to split the synchronous generate_groth16_proofs function into two asynchronous phases: generate_groth16_proofs_start_c (which returns immediately with a pending handle) and finalize_groth16_proof_c (which completes the proof later). This split allowed the GPU worker to offload the b_g2_msm computation to a background thread, hiding its latency from the critical path and improving throughput.

In the old synchronous design, generate_groth16_proofs_c called generate_groth16_proofs_start_c, which spawned a prep_msm_thread, then immediately called finalize_groth16_proof_c, which joined the thread. The function did not return until the thread completed, so all stack-allocated data remained valid throughout the thread's lifetime. This was safe, if suboptimal.

Phase 12 changed this contract fundamentally. Now generate_groth16_proofs_start_c returns immediately with a groth16_pending_proof handle, while the prep_msm_thread continues running in the background. The C++ function's stack frame is destroyed upon return, but the background thread continues reading from data that was captured by reference — data that no longer exists.

The Bug Revealed

The assistant discovered the bug through careful code inspection in the messages leading up to the subject edit. The prep_msm_thread lambda captured variables with [&, num_circuits] — capturing everything by reference. Among the captured variables was provers, the function parameter declared as const Assignment<fr_t> provers[]. In C++, array parameters decay to pointers, so provers was a local pointer variable on the stack. The lambda captured a reference to this stack pointer variable. After the function returned, that stack slot was reclaimed. When the background thread later dereferenced provers[c], it read through a dangling reference to a stack location that might already hold arbitrary data.

The assistant initially considered whether the bug might have been "working by chance" — the stack memory might not have been overwritten yet. But undefined behavior is undefined behavior, and on a heavily loaded system with multiple threads, the window of "working by chance" was narrow and unreliable.

The Reasoning Process

What makes this debugging chain remarkable is the assistant's systematic analysis of C++ reference semantics. The assistant considered multiple approaches:

First attempt: shadow the parameter. The assistant added std::vector<Assignment<fr_t>> provers_owned to the groth16_pending_proof struct and attempted to create a local reference const auto& provers = pp->provers_owned; that would shadow the function parameter. The idea was that all subsequent code using provers[c] would transparently use the heap-allocated copy. But this failed at compile time — C++ does not allow shadowing a function parameter with a local variable in the same scope.

Second attempt: analyze which references are actually dangerous. The assistant carefully categorized every use of provers in the function:

Lines 429 and 577: inside the prep_msm_thread lambda — dangerous, runs after function return.
Lines 838–1115+: inside the GPU thread lambda — safe, the GPU thread is joined before the function returns.
Lines 1188–1190: cudaHostUnregister calls — safe, execute before the function returns. This analysis revealed that only two lines needed fixing, not the entire function body. Third attempt: understand reference-to-reference collapsing. The assistant realized that pp (the pointer to the pending proof struct) was also captured by reference in the lambda. If the lambda used pp->provers_owned[c] directly, would that be safe? The pp pointer itself is a stack variable — after the function returns, the stack location holding the pointer value is gone. However, the assistant reasoned that in C++14+, when a lambda captures a reference variable by [&], it captures the original referent, not the reference itself. So auto& provers_safe = pp->provers_owned; followed by a lambda that captures [&] would cause the lambda to directly reference pp->provers_owned — which lives on the heap and outlives the function. This is the fix that was ultimately applied: adding a local reference alias provers_safe that points to the heap-allocated vector, and replacing provers[c] with provers_safe[c] inside the prep_msm_thread lambda.

Assumptions and Mistakes

The assistant made several assumptions during this debugging process:

The initial assumption that shadowing would work. The assistant assumed that const auto& provers = pp->provers_owned; would compile and shadow the function parameter. This was incorrect — C++ does not allow re-declaring a name in the same scope, even with a different type.
The assumption about pp being safe. The assistant initially worried that pp itself (a stack pointer) would be dangling after function return. But upon deeper analysis, the assistant realized that pp is only used at function scope, not inside the thread — the thread only uses provers and other alias references that were set up from pp-> fields.
The assumption about reference capture semantics. The assistant had to work through the subtlety of whether [&] captures a reference variable by binding to the reference itself or to the referent. The correct understanding is that in C++, reference variables are transparent — capturing a reference by reference captures the original object. This is safe when the original object is heap-allocated.

Input Knowledge Required

To understand this message and the reasoning behind it, one needs:

C++ lambda capture semantics: understanding the difference between [&] and [=], and how reference variables interact with lambda captures.
C++ reference collapsing: understanding that auto& ref = expr; followed by capturing ref by reference in a lambda resolves to the original object.
CUDA programming patterns: understanding the GPU lock mechanism, the separation between GPU kernel execution and CPU-side preparation work, and the cudaHostRegister/cudaHostUnregister memory management.
The Phase 12 architecture: knowing that the split API was designed to offload b_g2_msm to a background thread, and that this restructuring invalidated the previous safety guarantees.
Rust FFI memory management: understanding that the Assignment structs contain raw pointers into Rust-owned memory (Vecs), and that the Rust side keeps these allocations alive through the PendingProofHandle.

Output Knowledge Created

This edit produced a corrected groth16_cuda.cu file where the prep_msm_thread accesses pp->provers_owned[c] (via the provers_safe alias) instead of the stack-allocated provers parameter. This ensures that the background thread always accesses stable, heap-allocated memory that outlives the function call. The fix was minimal — only two lines changed — but it resolved a critical concurrency bug that could have caused data corruption, crashes, or incorrect proof generation under load.

The broader output knowledge is a template for how to safely restructure synchronous C++ APIs into asynchronous ones when background threads capture stack data. The pattern is: identify all data that outlives the function, copy it into a heap-allocated struct, and have background threads reference the heap copy rather than stack parameters. This is a general lesson for any system that transitions from synchronous to asynchronous execution models.

The Significance

This edit, though brief in its confirmation message, represents the culmination of a deep debugging session that touched on the most treacherous corners of C++ concurrency. The assistant navigated undefined behavior, reference semantics, lambda capture rules, and the interaction between two language ecosystems (Rust and C++) to find and fix a bug that could have silently corrupted proofs or crashed the daemon under production load. The fix itself is elegant — two lines changed, leveraging C++ reference-to-reference collapsing to transparently redirect the background thread to stable memory — but the reasoning required to arrive at those two lines was anything but simple.