From Memory Bandwidth to Latency Hiding: The Phase 11–12 Transition in the cuzk Proving Engine

Introduction

The optimization of high-performance computing systems rarely follows a straight line. More often, it traces a jagged path: parametric tuning yields diminishing returns, a deeper analysis reveals a structural bottleneck, and the engineering effort pivots from tweaking knobs to redesigning architecture. This article examines exactly such a transition in the context of the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep). Over the course of Segment 29 of an opencode coding session, the assistant completed the implementation and benchmarking of Phase 11's three memory-bandwidth interventions, analyzed the results to identify a deeper bottleneck, designed a split-API architecture to address it, and began implementing Phase 12 — all while maintaining the discipline of systematic measurement, careful reasoning, and cross-language coordination that characterizes the entire optimization campaign.

The story of this segment is the story of how a 3.4% throughput improvement became the catalyst for a fundamental architectural change. It is a case study in the interplay between measurement and design, between understanding what the system is doing and deciding how to change what it does. And it is a masterclass in cross-language performance engineering, demonstrating how a single insight about a CPU-bound computation blocking a GPU worker's critical path ripples through C++ CUDA kernels, Rust FFI bindings, pipeline wrappers, and engine orchestration.

Phase 11: Three Interventions Against Memory Bandwidth Contention

The Phase 11 optimization campaign was born from the ashes of Phase 10, a failed two-lock GPU interlock design that was abandoned after discovering fundamental CUDA device-global synchronization conflicts. The post-mortem analysis had identified the true bottleneck: DDR5 memory bandwidth contention at high concurrency. When the system ran 20 concurrent proofs with 15 parallel jobs, throughput degraded from 32.1 seconds per proof in isolation to 38.0 seconds per proof — an ~18% penalty caused by CPU-side interference rather than GPU utilization.

The root cause analysis traced this degradation to three distinct sources of memory subsystem pressure. First, asynchronous deallocation threads calling munmap() triggered Translation Lookaside Buffer (TLB) shootdown Inter-Processor Interrupts (IPIs) across all CPU cores, stalling execution. Second, the b_g2_msm computation's thread pool of 192 threads allocated ~1.1 GiB of Pippenger bucket arrays, thrashing the L3 cache and evicting useful data from concurrently running synthesis threads. Third, the aggregate TLB pressure from multiple memory-intensive operations — PCE MatVec, prep_msm, b_g2_msm, and async deallocation — overwhelmed the memory subsystem.

Phase 11 proposed three targeted interventions, each addressing one of these pressure sources. Intervention 1 serialized async deallocation with a static mutex in both C++ and Rust, ensuring that only one thread at a time could call munmap(), thereby eliminating the TLB shootdown storm. Intervention 2 reduced the groth16_pool thread count from 192 to 32 via a gpu_threads = 32 configuration parameter, cutting the L3 cache footprint of b_g2_msm by approximately 6×. Intervention 3 added a global atomic throttle flag — a #[no_mangle] pub extern "C" function in the cuzk-pce crate that C++ would set around b_g2_msm and Rust's SpMV would check with yield_now — attempting to coordinate memory-intensive phases across the language boundary.

The Benchmarking Campaign: Systematic Measurement

The assistant implemented each intervention with careful attention to correctness and cross-language coordination. Intervention 1 required adding a static std::mutex dealloc_mtx in groth16_cuda.cu and a corresponding static Mutex in supraseal.rs. Intervention 2 was a single configuration change — setting gpu_threads = 32 in the daemon's TOML config — but its effects were profound, reducing the thread pool from 192 to 32 and cutting L3 cache pressure by ~6×. Intervention 3 was the most architecturally creative: a global atomic flag that C++ set around the b_g2_msm computation, with Rust's SpMV loop checking the flag and yielding periodically to reduce contention.

The benchmark sweep at c=20 j=15 (20 proofs, 15 concurrent jobs) produced a clear ranking. Intervention 1 alone showed 37.9 seconds per proof — essentially identical to the 38.0 second Phase 9 baseline. The TLB shootdown reduction from serializing deallocation was marginal, suggesting that munmap interference was not the dominant bottleneck. Intervention 2, however, delivered a measurable improvement: 36.7 seconds per proof, a 3.4% gain. The reduction in thread count cut L3 cache thrashing enough to benefit the concurrently running synthesis threads, even though b_g2_msm itself slowed from ~0.5 seconds to ~1.7 seconds.

Intervention 3 added nothing measurable: 36.8 seconds per proof with all three interventions, essentially identical to Intervention 2 alone. The throttle flag was harmless but redundant — with only 32 threads, the L3 contention during b_g2_msm was already low enough that voluntary yielding of SpMV had no effect.

The assistant also explored the user's suggestion of increasing GPU workers to 3 or 4 per device. The results were counterintuitive but instructive: 3 workers produced 37.2 seconds per proof with prove times ballooning to 83.0 seconds, and 4 workers were even worse at 37.4 seconds with 104.9 second prove times. The single GPU mutex, introduced in Phase 8 to prevent concurrent kernel launches, created a serialization point that turned additional workers into a liability. The workers spent most of their time waiting for the GPU lock rather than doing useful work.

The winning configuration was clear: gw=2, gpu_threads=32 at 36.7 seconds per proof. The assistant committed the code, updated the project document, and prepared the Phase 11 summary.

The Discovery That Changed Direction

The Phase 11 benchmarks had produced a modest but real improvement. But the assistant's analysis of the b_g2_msm timing data revealed something troubling: with only 32 threads, b_g2_msm had slowed from ~0.5 seconds to ~1.7 seconds — a 3–4× slowdown. This was far more than the design spec had predicted. The Pippenger algorithm, which scales with thread count for a fixed problem size, simply couldn't saturate the CPU's vector units and memory bandwidth with only 32 threads.

The critical question was whether this 1.7 seconds was on the GPU worker's critical path. The worker's cycle was: acquire GPU lock → GPU kernels (~1.8s) → release lock → b_g2_msm (~1.7s) → epilogue → loop back for next job. The GPU lock was released after ~1.8 seconds, but the worker didn't loop back until b_g2_msm and the epilogue completed. In dual-worker mode, Worker A's b_g2_msm could partially overlap with Worker B's GPU kernels, but the overlap was imperfect — the assistant's timeline analysis showed that b_g2_msm frequently outlasted GPU kernel execution, creating idle gaps of 173–310 milliseconds per partition.

The user, reading the Phase 11 summary, asked a question that would reshape the entire optimization trajectory: "can we ship b_g2_msm easily to a separate thread/worker to unblock the gpu worker more quickly?"

This question was a masterclass in systems-level thinking. The user recognized that b_g2_msm had no dependency on the GPU mutex or GPU kernel results — it operated on data already in host memory and produced output consumed only in the epilogue. The GPU worker was being blocked by CPU work that could run independently. The solution was not to make b_g2_msm faster (which would require more threads and more L3 contention) but to move it off the critical path entirely.

Designing the Split API

The assistant's response to the user's question was a multi-pass design exploration that systematically evaluated alternatives before converging on a concrete plan. The first idea — splitting the prep_msm_thread inside C++ into two threads — was rejected because the function would still block before returning. The second idea — returning intermediate MSM results to Rust and doing final assembly there — was acknowledged as "the cleaner approach" but required a significant API change and reimplementation of cryptographic logic.

The third approach became the foundation of Phase 12: a two-phase split API. The monolithic generate_groth16_proofs_c function would be divided into generate_groth16_proofs_start_c, which does everything through GPU unlock and returns an opaque handle, and finalize_groth16_proof, which joins the b_g2_msm thread, runs the epilogue, and writes the final proof. The GPU worker would call the first function, hand the handle to a finalizer thread, and immediately loop back to pick up the next synthesis job.

The key design artifact was the groth16_pending_proof struct, a C++ object that would hold all the state needed for finalization: the b_g2_msm thread, GPU MSM results, batch addition results, the verifying key pointer, split MSM flags, randomness scalars, and deallocation data. This struct would be allocated on the heap and returned as an opaque pointer across the FFI boundary.

The Critical Realization: Allocating the Handle Early

The assistant began implementing the split API by adding the groth16_pending_proof struct definition, the finalize and destroy FFI functions, and the start function skeleton. But in the process of restructuring the existing code, the assistant discovered a fundamental flaw in the initial design.

The prep_msm_thread — the background thread that runs b_g2_msm — captured local variables by reference via the lambda capture [&, num_circuits]. The GPU threads also captured results and batch_add_res by reference. The initial plan was to move these locals into the handle after the GPU threads had joined but before returning. But the whole point of the split API was that b_g2_msm was still running when we wanted to return. Moving results while prep_msm_thread was writing to results.b_g2[c] would create a use-after-move bug — a classic concurrency error that would manifest as intermittent memory corruption.

The solution was elegant and decisive: allocate the groth16_pending_proof handle on the heap before any threads are spawned. Use its fields — results, batch_add_res, the split vectors, the tail MSM bases — as the actual storage locations. Pass references to these fields to both the GPU threads and prep_msm_thread. The handle itself is the storage; it never moves. When the function returns, it simply returns a pointer to the handle. The threads are already referencing its fields by address, and those addresses remain valid for the lifetime of the handle.

This was a "bigger refactor but it's the clean way," as the assistant noted. Every local variable that needed to survive the early return had to be replaced with a field access on the handle. The assistant systematically worked through the code, replacing declarations and adjusting captures, ensuring that every piece of shared state lived at a stable memory address from the very beginning.

Compilation Errors: The Compiler as Oracle

The refactoring did not compile on the first attempt. A build failure revealed two distinct errors. The first was an ordering issue: pp (the handle pointer) was referenced at line 367 in the split MSM flag aliases before it was allocated. The assistant had added the alias code at lines 364-372 without checking that the allocation preceded it. This was a straightforward ordering error that the compiler caught immediately.

The second error was more subtle. A mult_pippenger template instantiation failed because of a const/non-const pointer ambiguity in a ternary expression. The call site used a ternary to choose between tail_msm_b_g2_bases.data() (which returned non-const affine_fp2_t* after the refactoring) and points_b_g2.data() (which returned const affine_fp2_t* from the SRS slice). C++ template argument deduction does not perform implicit conversions on deduced parameters, so the compiler could not choose between const T* and T*. The fix — casting the ternary result — was simple, but the root cause was revealing: aliasing a local vector to a handle field had subtly changed the const-qualification of .data() at call sites far from the actual changes.

These errors illustrate a core challenge of cross-language FFI development. A single logical change (splitting a function) requires coordinated modifications across C++ structs, Rust wrappers, and application-level orchestration. Each layer has its own type system, lifetime semantics, and compilation model. The compiler is the oracle that surfaces inconsistencies, and the assistant's methodology — edit, build, diagnose, fix, rebuild — is the only reliable way to navigate this complexity.

The Grep That Saved a Proof

One of the most instructive moments in the implementation was a seemingly trivial grep command. The assistant searched for three boolean flags — l_split_msm, a_split_msm, b_split_msm — that controlled how multi-scalar multiplication results were assembled across GPUs. These flags were set during prep_msm and read during the epilogue. In the original monolithic function, they were simple local booleans. But in the split API, they needed to live in the handle.

The grep revealed that the flags were already declared in the groth16_pending_proof struct (added in an earlier edit), initialized in the constructor, and read in the finalize function. But the main function still had local declarations that needed to be replaced with aliases pointing into the heap-allocated handle. Without this verification, the finalize function would read uninitialized or dangling stack memory, producing incorrect MSM assembly and ultimately an invalid Groth16 proof.

This single grep command exemplifies the discipline required for safe cross-language refactoring. The assistant could have assumed the flags were handled and moved on. Instead, it paused, verified, and ensured that every piece of state that needed to survive the split was properly accounted for.

The Rust FFI Boundary

With the C++ side compiling successfully, the assistant turned to the Rust FFI boundary. The work proceeded in three layers, each building on the previous one.

First, in supraseal-c2/src/lib.rs, the assistant added extern "C" declarations for generate_groth16_proofs_start_c and finalize_groth16_proof. These raw FFI functions use C-compatible types: opaque pointers (*mut c_void) for the pending handle, and RustError return types for error handling.

Second, in bellperson/src/groth16/prover/supraseal.rs, the assistant created safe Rust wrappers: PendingProofHandle (a struct wrapping the opaque pointer with Drop semantics), prove_start (which calls the C++ start function and packages the handle), and prove_finish (which calls the C++ finalize function and extracts the proof). These wrappers handle the memory ownership semantics — the handle is allocated by C++, returned to Rust, and freed by C++ when finalized or by Rust's Drop if never finalized.

Third, in bellperson/src/groth16/mod.rs, the assistant added PendingProof, prove_start, and prove_finish to the module's public re-exports. This step was the "final Rust plumb" — the moment when the split API became accessible to the rest of the codebase. Without this export, the higher-level pipeline and engine code would have no way to call the split API, and the entire refactoring would remain stranded at the FFI boundary.

Pipeline Integration

The next layer was the proving pipeline in cuzk-core/src/pipeline.rs. The assistant added gpu_prove_start and gpu_prove_finish wrapper functions that bridge between the engine's job-queue types and the bellperson API. These wrappers handle the conversion of Rust types (scalars, circuit assignments) to the FFI-compatible representations expected by the C++ code.

The pipeline integration was relatively straightforward — the wrappers are thin layers that translate between the engine's data model and the bellperson API. But they serve a critical architectural function: they insulate the engine from the details of the FFI boundary. If the C++ API changes in the future, only the pipeline wrappers need to be updated, not the engine itself.

The Hardest Part: Restructuring the GPU Worker Loop

The final and most complex integration was restructuring the GPU worker loop in engine.rs. The existing code used a monolithic spawn_blocking call wrapping gpu_prove, followed by .await and then hundreds of lines of result-processing logic: tracker updates, partition assembly, error handling, and job status notifications. The assistant's plan was to replace this with a three-step sequence: call gpu_prove_start (fast, returns after GPU unlock), spawn a tokio task for finalization (gpu_prove_finish plus result processing), and immediately loop back to pick up the next synthesis job.

But the assistant quickly discovered that the result-processing code was far more entangled with the GPU worker's local state than anticipated. Variables like tracker, worker_id, job_id, partition_index, is_batched, and batch_requests were all captured by the existing async block. Moving the result processing into a spawned task required either cloning all these variables or extracting them into helper functions.

The assistant worked through five distinct approaches:

Helper functions: Extract the result-processing logic into process_partition_result and process_monolithic_result. The obstacle was understanding the full Tracker API and the dozen-plus parameters that would need to be passed.
Channel-based finalization: Have the finalizer task send results through a channel, and let the existing result-processing code read from that channel. Architecturally elegant but required new infrastructure.
Same-worker deferred processing: Process the pending proof in the same GPU worker on the next iteration. The assistant immediately rejected this as self-defeating — it would defeat the entire purpose of the split.
Dedicated finalizer task pool: A pool of finalizer tasks reading from a channel. The most complete solution but "a significant architectural change."
Inline the entire result processing into a spawned tokio task: The simplest to implement but would create "massive duplication" between the monolithic path and the split path. The assistant's preference was for the simplest approach that maintained the current code structure. The decision to extract helper functions (process_partition_result, process_monolithic_result) emerged as the leading candidate, but the assistant recognized it required deep understanding of the Tracker API and the type ecosystem surrounding it. This architectural reflection — weighing five alternatives against criteria of simplicity, maintainability, and correctness — is characteristic of the engineering discipline that permeates the entire optimization campaign.

The Broader Significance

This segment of the opencode session is a microcosm of the entire optimization campaign. It demonstrates the iterative, measurement-driven methodology that characterizes high-performance systems development: benchmark to identify the bottleneck, design a targeted intervention, implement across multiple layers, diagnose compilation errors, fix, and repeat. The assistant's systematic audit of shared state, its careful management of memory lifetimes across the FFI boundary, and its architectural reflection on the trade-offs between simplicity and maintainability all reflect a disciplined engineering approach.

The transition from Phase 11 to Phase 12 represents a fundamental shift in optimization strategy. Phase 11 was parametric: it tuned thread counts, serialized operations, and added flags to reduce contention within the existing architecture. Phase 12 is architectural: it restructures the GPU worker pipeline to decouple GPU kernel execution from CPU post-processing, changing how work flows through the system. This shift was enabled by the systematic measurement discipline of Phase 11. Without the benchmark data showing that b_g2_msm had slowed to 1.7 seconds, without the timeline analysis revealing the idle gaps, and without the user's sharp question about shipping b_g2_msm to a separate thread, the architectural insight would not have emerged. The 3.4% improvement was modest, but the understanding it produced was transformative.

The segment also demonstrates the importance of concurrency-aware design in cross-language systems. The use-after-move bug that the assistant discovered and fixed is a class of error that is notoriously difficult to debug because it manifests as intermittent corruption rather than a clean crash. The assistant's careful analysis of reference lifetimes — tracing which threads capture which variables by reference, and ensuring that all shared state lives at stable heap addresses — prevented a bug that could have wasted hours of debugging time in production.

Conclusion

This segment of the opencode session captures a pivotal transition in a high-stakes optimization campaign. Phase 11's three memory-bandwidth interventions delivered a 3.4% throughput improvement, but more importantly, they revealed the structural bottleneck that would drive Phase 12: the b_g2_msm computation blocking the GPU worker's critical path. The split API design that emerged from this insight — with its early-allocated pending handle, its two-phase FFI boundary, and its careful attention to memory lifetimes — represents a fundamental architectural improvement to the proving pipeline.

The work in this segment exemplifies the iterative, measurement-driven approach that characterizes the entire optimization campaign. Each phase tests hypotheses, measures results, and generates insights that inform the next phase. The gains are incremental — 3.4% here, a structural improvement there — but they accumulate into a system that is faster, more efficient, and better understood. The discipline of systematic benchmarking, the willingness to abandon flawed approaches, and the careful attention to concurrency semantics are the practices that make this accumulation possible. The Phase 12 split API is not yet benchmarked — the final integration into the engine worker loop remains the last major hurdle — but the analysis, design, and implementation work completed in this segment provides a solid foundation for what promises to be a significant throughput improvement.