The Commit That Nearly Wasn't: Phase 12's Split GPU Proving API and the Use-After-Free That Almost Broke It

On the surface, message [msg 3051] appears to be a routine git commit — a developer running git commit -m "feat(cuzk): Phase 12 — split (async) GPU proving API" with a detailed changelog. But this message is the culmination of a much deeper story. It represents the moment when a complex, multi-day optimization effort was finally sealed into the repository's history, after surviving a use-after-free bug, a memory capacity wall, and a cascade of Rust/C++ FFI compilation errors. Understanding this single commit message requires understanding the intricate debugging journey that preceded it — a journey that reveals how high-performance GPU proving systems are built, broken, and rebuilt at the intersection of C++ CUDA kernels, Rust async runtimes, and the unforgiving physics of 755 GiB of system memory.

The Context: Why Phase 12 Existed

The Phase 12 split GPU proving API was born from a specific bottleneck identified in earlier optimization phases. In the SUPRASEAL_C2 Groth16 proof generation pipeline — used for Filecoin Proof-of-Replication (PoRep) — the GPU worker loop had a structural inefficiency. After each partition's GPU work completed, the worker thread would block waiting for a CPU-side computation called b_g2_msm (a multi-scalar multiplication on the G2 curve) to finish before it could pick up the next job. This b_g2_msm step took approximately 1.7 seconds per partition, and during that time the GPU sat idle. The Phase 12 design decoupled these two operations: the GPU worker would launch b_g2_msm as a background task and immediately loop back to fetch the next synthesized partition, while a separate tokio finalizer task would later join the b_g2_msm thread and complete the proof.

The commit message at [msg 3051] summarizes this architectural shift succinctly: "Decouple b_g2_msm CPU computation from the GPU worker loop so the GPU worker can pick up the next synthesized partition ~1.7s faster." But the path to this commit was anything but straightforward.

The Use-After-Free That Nearly Escaped

The most critical bug uncovered during Phase 12 was a use-after-free in the C++ CUDA code, and the commit message's laconic entry — "Fix use-after-free: prep_msm_thread now reads provers_owned (heap copy) instead of the stack parameter that goes out of scope" — barely hints at the subtlety involved.

The generate_groth16_proofs_start_c function received a const Assignment<fr_t> provers[] parameter — a C array parameter that decays to a stack pointer. Inside the function, a background thread (prep_msm_thread) was launched with a lambda capture [&, num_circuits]. This lambda captured provers by reference — meaning it held a reference to the stack variable containing the pointer, not a copy of the data. After generate_groth16_proofs_start_c returned, that stack variable evaporated. The background thread, still running b_g2_msm, would dereference a dangling reference to read the pointer value, then use that pointer to access heap data that might or might not still be valid.

The fix required careful C++ semantics analysis. The assistant traced through the lambda capture chains across multiple messages ([msg 3024] through [msg 3034]), distinguishing between:

• References to pp-> fields (heap-allocated, safe because the heap outlives the function)
• References to the provers function parameter (stack-allocated, dangling after return)
• References used inside the GPU thread (safe because the GPU thread is joined before function return)
• References used inside the prep_msm_thread (unsafe because it runs asynchronously) The solution was to copy the provers array into pp->provers_owned (a std::vector on the heap) and have the background thread reference that instead. The assistant initially attempted to add a const auto& provers_safe = pp->provers_owned; alias inside the lambda body, but discovered that provers was already declared as a function parameter, causing a shadowing compilation error. The final approach used a differently-named alias (provers_safe) and replaced the two provers[c] references inside the prep_msm_thread lambda with provers_safe[c].

The Compilation Gauntlet

Before the UB fix could even be tested, the code had to compile — and Phase 12's cross-language FFI boundary between Rust and C++ presented numerous obstacles. The commit message lists several: a missing SynthesisCapacityHint struct, an unused generic parameter in the start_groth16_proof FFI, the PendingGpuProof type alias needed in pipeline.rs, and helper functions (process_partition_result, process_monolithic_result) that had to be extracted from inline result-processing code in the engine.

One particularly tricky issue involved trait bounds. The prove_start method's parameter had a trait bound mismatch that was resolved by changing it to take params: P by value instead of by reference. Another issue involved continue statements inside async blocks — in Rust, continue inside an async block or closure doesn't work the same way as in a loop body, and the assistant had to replace them with return to properly exit the spawned finalizer task.

These were not superficial compilation errors. They reflected deep design tensions between the synchronous C++ API (which had been a single monolithic function) and the new split API (which required Rust's async runtime to manage the lifecycle of a pending proof handle across two separate calls). The Rust FFI had to expose three new functions: start_groth16_proof, finish_groth16_proof, and drop_pending_proof, each with its own memory safety invariants.

The Memory Capacity Wall

After achieving a clean build, the assistant benchmarked the Phase 12 implementation. The results were promising: with configuration gw=2 pw=10 gt=32 j=15, throughput reached 37.1 seconds per proof, a ~2.4% improvement over the Phase 11 baseline of 38.0 seconds. But the real test — increasing partition workers from 10 to 12 to see if more parallelism would yield further gains — hit a hard wall.

With pw=12, the daemon was killed by the OOM (Out of Memory) killer. RSS monitoring revealed a peak of 668 GiB on a system with 755 GiB total RAM — leaving only 87 GiB for the kernel, page tables, and other processes. The commit message doesn't dwell on this, but the preceding messages show the assistant's analysis: "pw=10 peaked at 367 GiB, pw=12 peaked at 668 GiB. That's 301 GiB difference for just 2 extra workers. That's way more than 2 × 13 GiB = 26 GiB. Something else is going on."

This observation — that adding 2 partition workers consumed 301 GiB instead of the expected 26 GiB — revealed a nonlinearity in the system's memory behavior. The assistant hypothesized that faster partition drain (more workers) caused more proofs to be parsed and queued simultaneously, amplifying memory pressure far beyond the per-partition cost. This was a systemic capacity limitation, not a memory leak, and it set a hard bound on the achievable parallelism for this hardware configuration.

What the Commit Represents

The commit message at [msg 3051] is more than a changelog — it's a boundary marker. It demarcates the point where the Phase 12 optimization was considered complete and correct enough to preserve in version control. The 1,275 lines added and 439 deleted across 6 files represent not just code, but the accumulated understanding of:

• How to safely split a synchronous GPU operation across an async boundary
• How C++ lambda capture semantics interact with stack lifetime in a multi-threaded context
• How Rust's type system enforces memory safety at the FFI boundary
• How memory pressure scales nonlinearly with partition parallelism
• Where the current hardware's capacity limits lie The commit also implicitly documents a design decision that would shape future work: the split API pattern. By creating PendingProofHandle<E> in Bellperson and PendingGpuProof in the pipeline, the architecture now supports asynchronous GPU proving as a first-class concept. Future optimizations can build on this foundation — for instance, by further reducing per-partition memory footprint to push past the pw=10 ceiling.

The Thinking Process Visible in the Commit

The commit message itself reveals the assistant's analytical priorities. The first paragraph explains the why (decouple b_g2_msm to save 1.7s). The second paragraph enumerates the what (the key changes). The final line reports the result (37.1s/proof, 2.4% improvement). But the structure also reveals what the assistant chose not to document in the commit message: the failed attempt at pw=12, the nonlinear memory scaling, and the decision to accept pw=10 as the optimal configuration. These are captured in the surrounding conversation but deliberately omitted from the commit, which focuses on the positive result.

This selective framing is itself a design decision. The commit message presents Phase 12 as a clean success — a ~2.4% throughput improvement with a critical bug fix. The memory capacity limitation is acknowledged only implicitly, in the choice of benchmark parameters (pw=10). The assistant could have documented the OOM failure in the commit message, but chose instead to treat it as a separate concern, to be addressed in a future Phase 13.

Conclusion

Message [msg 3051] is a git commit, but it is also a thesis statement. It asserts that the split GPU proving API is correct, benchmarked, and ready for production use. It encodes the lessons of a multi-day debugging session — lessons about C++ lambda lifetimes, Rust async FFI patterns, and the nonlinear memory dynamics of GPU proving pipelines. And it marks a transition: from optimizing individual operations within a monolithic GPU call to architecting a continuous, asynchronous pipeline where the GPU never waits for CPU work. The 2.4% improvement is modest, but the architectural shift it represents — the decoupling of GPU and CPU critical paths — opens the door to more aggressive optimizations in the future. The commit is not the end of the story; it is the foundation for the next chapter.