From Memory Bandwidth to Latency Hiding: The Phase 12 Split API Refactoring

Introduction

In the high-stakes world of Filecoin proof-of-replication (PoRep) proving, every millisecond counts. The SUPRASEAL_C2 Groth16 proving pipeline, a GPU-accelerated system for generating SNARK proofs, had already been through multiple optimization phases. Phase 9 delivered a 14.2% throughput improvement through PCIe transfer optimization. Phase 11's memory-bandwidth interventions — serializing async deallocation, reducing GPU thread counts, and adding a global throttle flag — squeezed out another 3.4%. But the assistant and user both sensed that a deeper structural bottleneck remained.

This chunk of the opencode session captures the transition from Phase 11's memory-bandwidth analysis to Phase 12's architectural refactoring: a split API designed to hide the latency of b_g2_msm, a CPU-bound multi-scalar multiplication that blocks the GPU worker from picking up the next job even after the GPU lock has been released. What follows is a detailed examination of how this refactoring unfolded across C++ CUDA kernels, Rust FFI boundaries, and application-level orchestration — a case study in cross-language performance engineering.

Phase 11: The Memory Bandwidth Investigation

The chunk opens with the culmination of Phase 11's three memory-bandwidth interventions. The assistant had implemented and benchmarked each intervention against the Phase 9 baseline of 38.0 seconds per proof. Intervention 1 serialized the async deallocation of GPU buffers using a static mutex in both C++ and Rust, preventing TLB shootdowns from multiple threads freeing memory concurrently. Intervention 2 reduced the groth16_pool thread count from 192 to 32 via gpu_threads = 32, cutting down on L3 cache thrashing and DDR5 bandwidth contention. Intervention 3 added a global atomic throttle flag set by C++ around b_g2_msm and checked by Rust's SpMV with yield_now, attempting to coordinate memory-intensive phases.

The benchmark sweep at c=20 j=15 told a clear story: Intervention 2 alone delivered the best result — 36.7 seconds per proof, a 3.4% improvement over baseline. Interventions 1 and 3 had negligible additional impact, and increasing GPU workers to 3 or 4 made throughput worse due to CPU contention. The analysis confirmed that DDR5 memory bandwidth was the primary bottleneck, and that the b_g2_msm computation — approximately 1.7 seconds with 32 threads — was a structural problem: it ran after the GPU lock was released but still blocked the GPU worker from picking up the next synthesis job.

The Design of the Split API

The user's pivotal question — whether b_g2_msm could be shipped to a separate thread to unblock the GPU worker — launched the Phase 12 split API design. The assistant analyzed the dependency chain and confirmed that b_g2_msm runs after the GPU lock is released and does not depend on any GPU state that would be invalidated by the next job. The insight was classic latency hiding: if the GPU worker could hand off the b_g2_msm and subsequent epilogue work to a separate thread or task, it could immediately loop back to acquire the next synthesis job and begin the next GPU kernel launch.

The design was elegant in concept. Instead of a monolithic generate_groth16_proofs_c function that runs the entire proof generation pipeline (GPU kernels, CPU post-processing, epilogue) in one blocking call, the new design would split it into two entry points. generate_groth16_proofs_start_c would handle everything up to and including the GPU unlock, spawn a thread for b_g2_msm, and return an opaque handle. finalize_groth16_proof would join that background thread, run the epilogue, and write the final proof. The GPU worker's critical path would shrink from "GPU execution + CPU post-processing" to just "GPU execution," with post-processing happening in parallel.

But implementing this design required a fundamental restructuring of the C++ CUDA code. The existing monolithic function declared several key data structures — msm_results, batch_add_results, split vectors, tail MSM bases, split flags, and a caught_exception atomic — as local variables on the stack. These locals were captured by reference in multiple threads: the prep_msm_thread (which runs b_g2_msm) and the per-GPU threads (which launch CUDA kernels). In the original design, this was safe because all threads completed before the function returned. In the split design, the function would return before prep_msm_thread finished, leaving dangling references.

The C++ CUDA Refactoring: Aliasing Everything into the Handle

The solution was to allocate a groth16_pending_proof struct on the heap at the very beginning of the function, and to alias all shared state into its fields. The handle would outlive the function call because ownership was transferred to the Rust caller via the opaque pointer return. The assistant worked through this refactoring methodically, one variable at a time.

In messages spanning [msg 2866] through [msg 2878], the assistant moved results into the handle, then split_vectors_l/a/b and tail_msm_*_bases, then batch_add_res, then the split MSM flags (l_split_msm, a_split_msm, b_split_msm), and finally the caught_exception atomic. Each step required reading the file to find the declaration site, replacing the local declaration with a C++ reference alias (auto& x = pp->x), and ensuring the alias was declared before any thread captured it.

This systematic audit of shared state is a case study in concurrent systems programming. The assistant was not writing code by pattern-matching; it was reasoning about the lifetime of every piece of state in a multi-threaded function, tracking which threads accessed which variables, and ensuring that the refactored design preserved all the safety guarantees of the original. As one article in this chunk notes, "the assistant's thoroughness — catching caught_exception as an afterthought while reading the file — is the difference between a correct optimization and a buggy one" [2].

Compilation Errors: The Compiler as Oracle

The refactoring did not compile on the first attempt. Message [msg 2879] captured a build failure that revealed two distinct errors. The first was an ordering issue: pp (the handle pointer) was referenced at line 367 in the split MSM flag aliases before it was allocated. The assistant had added the alias code at lines 364-372 without checking that the allocation preceded it. This was a straightforward ordering error that the compiler caught immediately.

The second error was more subtle. A mult_pippenger template instantiation failed because of a const/non-const pointer ambiguity in a ternary expression. The call site used a ternary to choose between tail_msm_b_g2_bases.data() (which returned non-const affine_fp2_t* after the refactoring) and points_b_g2.data() (which returned const affine_fp2_t* from the SRS slice). C++ template argument deduction does not perform implicit conversions on deduced parameters, so the compiler could not choose between const T* and T*. The fix — casting the ternary result — was simple, but the root cause was revealing: aliasing a local vector to a handle field had subtly changed the const-qualification of .data() at call sites far from the actual changes.

These errors illustrate a core challenge of cross-language FFI development. A single logical change (splitting a function) requires coordinated modifications across C++ structs, Rust wrappers, and application-level orchestration. Each layer has its own type system, lifetime semantics, and compilation model. The compiler is the oracle that surfaces inconsistencies, and the assistant's methodology — edit, build, diagnose, fix, rebuild — is the only reliable way to navigate this complexity.

The Rust FFI Boundary

With the C++ side compiling successfully (message [msg 2886], the "quiet checkpoint"), the assistant turned to the Rust FFI boundary. The work proceeded in three layers, each building on the previous one.

First, in supraseal-c2/src/lib.rs, the assistant added extern "C" declarations for generate_groth16_proofs_start_c and finalize_groth16_proof. These raw FFI functions use C-compatible types: opaque pointers (*mut c_void) for the pending handle, and RustError return types for error handling.

Second, in bellperson/src/groth16/prover/supraseal.rs, the assistant created safe Rust wrappers: PendingProofHandle (a struct wrapping the opaque pointer with Drop semantics), prove_start (which calls the C++ start function and packages the handle), and prove_finish (which calls the C++ finalize function and extracts the proof). These wrappers handle the memory ownership semantics — the handle is allocated by C++, returned to Rust, and freed by C++ when finalized or by Rust's Drop if never finalized.

Third, in bellperson/src/groth16/mod.rs, the assistant added PendingProof, prove_start, and prove_finish to the module's public re-exports. This step, captured in message [msg 2894], was the "final Rust plumb" — the moment when the split API became accessible to the rest of the codebase. Without this export, the higher-level pipeline and engine code would have no way to call the split API, and the entire refactoring would remain stranded at the FFI boundary.

Pipeline Integration

The next layer was the proving pipeline in cuzk-core/src/pipeline.rs. The assistant added gpu_prove_start and gpu_prove_finish wrapper functions that bridge between the engine's job-queue types and the bellperson API. These wrappers handle the conversion of Rust types (scalars, circuit assignments) to the FFI-compatible representations expected by the C++ code.

The pipeline integration was relatively straightforward — the wrappers are thin layers that translate between the engine's data model and the bellperson API. But they serve a critical architectural function: they insulate the engine from the details of the FFI boundary. If the C++ API changes in the future, only the pipeline wrappers need to be updated, not the engine itself.

The Hardest Part: Restructuring the GPU Worker Loop

The final and most complex integration was restructuring the GPU worker loop in engine.rs. The existing code used a monolithic spawn_blocking call wrapping gpu_prove, followed by .await and then hundreds of lines of result-processing logic: tracker updates, partition assembly, error handling, and job status notifications. The assistant's plan was to replace this with a three-step sequence: call gpu_prove_start (fast, returns after GPU unlock), spawn a tokio task for finalization (gpu_prove_finish plus result processing), and immediately loop back to pick up the next synthesis job.

But the assistant quickly discovered that the result-processing code was far more entangled with the GPU worker's local state than anticipated. Variables like tracker, worker_id, job_id, partition_index, is_batched, and batch_requests were all captured by the existing async block. Moving the result processing into a spawned task required either cloning all these variables or extracting them into helper functions.

Message [msg 2908] captured a moment of architectural reflection where the assistant worked through five distinct approaches:

Helper functions: Extract the result-processing logic into process_partition_result and process_monolithic_result. The obstacle was understanding the full Tracker API and the dozen-plus parameters that would need to be passed.
Channel-based finalization: Have the finalizer task send results through a channel, and let the existing result-processing code read from that channel. Architecturally elegant but required new infrastructure.
Same-worker deferred processing: Process the pending proof in the same GPU worker on the next iteration. The assistant immediately rejected this as self-defeating.
Dedicated finalizer task pool: A pool of finalizer tasks reading from a channel. The most complete solution but "a significant architectural change."
Inline the entire result processing into a spawned tokio task: The simplest to implement but would create "massive duplication" between the monolithic path and the split path. The assistant's preference was for the simplest approach that maintained the current code structure. The decision to extract helper functions (process_partition_result, process_monolithic_result) emerged as the leading candidate, but the assistant recognized it required deep understanding of the Tracker API and the type ecosystem surrounding it.

The Broader Significance

This chunk of the opencode session is a microcosm of the entire optimization campaign. It demonstrates the iterative, error-driven methodology that characterizes high-performance systems development: benchmark to identify the bottleneck, design a targeted intervention, implement across multiple layers, diagnose compilation errors, fix, and repeat. The assistant's systematic audit of shared state, its careful management of memory lifetimes across the FFI boundary, and its architectural reflection on the trade-offs between simplicity and maintainability all reflect a disciplined engineering approach.

The Phase 12 split API is not a silver bullet. It introduces architectural complexity — a new handle type, a two-phase API, a channel-based finalization mechanism — in exchange for hiding 1.7 seconds of latency per proof. Whether this trade-off pays off depends on the benchmark results that will follow this implementation. But the work itself is a testament to the kind of careful, multi-layered reasoning required to optimize a production GPU proving pipeline at the intersection of C++, CUDA, and Rust.

Conclusion

From the memory bandwidth analysis of Phase 11 to the architectural refactoring of Phase 12, this chunk captures a pivotal transition in the SUPRASEAL_C2 optimization campaign. The split API design — born from a detailed understanding of the GPU worker's critical path and the b_g2_msm bottleneck — required coordinated changes across C++ CUDA kernels, Rust FFI bindings, pipeline wrappers, and engine orchestration. The assistant's methodical approach, cycling through editing, building, diagnosing, and fixing, demonstrates the discipline required for cross-language performance engineering. The final integration into the engine worker loop remains the last major hurdle before benchmarking can validate whether the latency-hiding strategy delivers the throughput improvement that the analysis promises.