From Memory Bandwidth to Architectural Restructuring: The Phase 11–12 Transition in a GPU Proving Pipeline

Introduction

The optimization of high-performance computing systems is rarely a linear progression. More often, it follows a jagged path: parametric tuning yields diminishing returns, a deeper analysis reveals a structural bottleneck, and the engineering effort pivots from tweaking knobs to redesigning architecture. This article examines exactly such a transition in the context of the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep). Over the course of a single chunk of an opencode coding session, the assistant completed the implementation and benchmarking of Phase 11's three memory-bandwidth interventions, analyzed the results to identify a deeper bottleneck, designed a split-API architecture to address it, and began implementing Phase 12 — all while maintaining the discipline of systematic measurement, careful reasoning, and cross-language coordination that characterizes the entire optimization campaign.

The story of this chunk is the story of how a 3.4% throughput improvement became the catalyst for a fundamental architectural change. It is a case study in the interplay between measurement and design, between understanding what the system is doing and deciding how to change what it does.

Phase 11: Three Interventions Against Memory Bandwidth Contention

The Phase 11 optimization campaign was born from the ashes of Phase 10, a failed two-lock GPU interlock design that was abandoned after discovering fundamental CUDA device-global synchronization conflicts [1]. The post-mortem analysis had identified the true bottleneck: DDR5 memory bandwidth contention at high concurrency. When the system ran 20 concurrent proofs with 15 parallel jobs, throughput degraded from 32.1 seconds per proof in isolation to 38.0 seconds per proof — a ~18% penalty caused by CPU-side interference rather than GPU utilization.

The root cause analysis traced this degradation to three distinct sources of memory subsystem pressure. First, asynchronous deallocation threads calling munmap() triggered Translation Lookaside Buffer (TLB) shootdown Inter-Processor Interrupts (IPIs) across all CPU cores, stalling execution. Second, the b_g2_msm computation's thread pool of 192 threads allocated ~1.1 GiB of Pippenger bucket arrays, thrashing the L3 cache and evicting useful data from concurrently running synthesis threads. Third, the aggregate TLB pressure from multiple memory-intensive operations — PCE MatVec, prep_msm, b_g2_msm, and async deallocation — overwhelmed the memory subsystem [1].

Phase 11 proposed three targeted interventions: serializing async deallocation with a static mutex (Intervention 1), reducing the groth16_pool thread count from 192 to 32 (Intervention 2), and adding a global atomic throttle flag that C++ would set around b_g2_msm and Rust's SpMV would check with yield_now (Intervention 3) [1].

The Benchmarking Campaign: Systematic Measurement

The assistant implemented each intervention with careful attention to correctness and cross-language coordination. Intervention 1 required adding a static std::mutex dealloc_mtx in groth16_cuda.cu and a corresponding static Mutex in supraseal.rs [10][14]. Intervention 2 was a single configuration change — setting gpu_threads = 32 in the daemon's TOML config — but its effects were profound, reducing the thread pool from 192 to 32 and cutting L3 cache pressure by ~6× [28]. Intervention 3 was the most architecturally creative: a #[no_mangle] pub extern "C" function in the cuzk-pce crate that C++ called around the b_g2_msm computation, with Rust's SpMV loop checking an AtomicI32 flag and yielding periodically [49][50][51].

The benchmark sweep at c=20 j=15 (20 proofs, 15 concurrent jobs) produced a clear ranking. Intervention 1 alone showed 37.9 seconds per proof — essentially identical to the 38.0 second Phase 9 baseline [25][26]. The TLB shootdown reduction from serializing deallocation was marginal, suggesting that munmap interference was not the dominant bottleneck. Intervention 2, however, delivered a measurable improvement: 36.7 seconds per proof, a 3.4% gain [34]. The reduction in thread count cut L3 cache thrashing enough to benefit the concurrently running synthesis threads, even though b_g2_msm itself slowed from ~0.5 seconds to ~1.7 seconds.

Intervention 3 added nothing measurable: 36.8 seconds per proof with all three interventions, essentially identical to Intervention 2 alone [66][82]. The throttle flag was harmless but redundant — with only 32 threads, the L3 contention during b_g2_msm was already low enough that voluntary yielding of SpMV had no effect.

The assistant also explored the user's suggestion of increasing GPU workers to 3 or 4 per device [67][68]. The results were counterintuitive but instructive: 3 workers produced 37.2 seconds per proof with prove times ballooning to 83.0 seconds, and 4 workers were even worse at 37.4 seconds with 104.9 second prove times [82]. The single GPU mutex, introduced in Phase 8 to prevent concurrent kernel launches, created a serialization point that turned additional workers into a liability. The workers spent most of their time waiting for the GPU lock rather than doing useful work.

The winning configuration was clear: gw=2, gpu_threads=32 at 36.7 seconds per proof. The assistant committed the code, updated the project document, and prepared the Phase 11 summary [84][85][89][90][91].

The Discovery That Changed Direction

The Phase 11 benchmarks had produced a modest but real improvement. But the assistant's analysis of the b_g2_msm timing data revealed something troubling: with only 32 threads, b_g2_msm had slowed from ~0.5 seconds to ~1.7 seconds — a 3–4× slowdown [34][35]. This was far more than the design spec had predicted. The Pippenger algorithm, which scales with thread count for a fixed problem size, simply couldn't saturate the CPU's vector units and memory bandwidth with only 32 threads.

The critical question was whether this 1.7 seconds was on the GPU worker's critical path. The worker's cycle was: acquire GPU lock → GPU kernels (~1.8s) → release lock → b_g2_msm (~1.7s) → epilogue → loop back for next job. The GPU lock was released after ~1.8 seconds, but the worker didn't loop back until b_g2_msm and the epilogue completed. In dual-worker mode, Worker A's b_g2_msm could partially overlap with Worker B's GPU kernels, but the overlap was imperfect — the assistant's timeline analysis showed that b_g2_msm frequently outlasted GPU kernel execution, creating idle gaps of 173–310 milliseconds per partition [97][98].

The user, reading the Phase 11 summary, asked a question that would reshape the entire optimization trajectory: "can we ship b_g2_msm easily to a separate thread/worker to unblock the gpu worker more quickly?" [92]

This question was a masterclass in systems-level thinking. The user recognized that b_g2_msm had no dependency on the GPU mutex or GPU kernel results — it operated on data already in host memory and produced output consumed only in the epilogue. The GPU worker was being blocked by CPU work that could run independently. The solution was not to make b_g2_msm faster (which would require more threads and more L3 contention) but to move it off the critical path entirely.

Designing the Split API

The assistant's response to the user's question was a multi-pass design exploration that systematically evaluated alternatives before converging on a concrete plan [98][112][114][117]. The first idea — splitting the prep_msm_thread inside C++ into two threads — was rejected because the function would still block before returning. The second idea — returning intermediate MSM results to Rust and doing final assembly there — was acknowledged as "the cleaner approach" but required a significant API change and reimplementation of cryptographic logic.

The third approach became the foundation of Phase 12: a two-phase split API. The monolithic generate_groth16_proofs_c function would be divided into generate_groth16_proofs_start_c, which does everything through GPU unlock and returns an opaque handle, and finalize_groth16_proof, which joins the b_g2_msm thread, runs the epilogue, and writes the final proof. The GPU worker would call the first function, hand the handle to a finalizer thread, and immediately loop back to pick up the next synthesis job.

The key design artifact was the groth16_pending_proof struct, a C++ object that would hold all the state needed for finalization: the b_g2_msm thread, GPU MSM results, batch addition results, the verifying key pointer, split MSM flags, randomness scalars, and deallocation data [112][117]. This struct would be allocated on the heap and returned as an opaque pointer across the FFI boundary.

The Critical Realization: Allocating the Handle Early

The assistant began implementing the split API by adding the groth16_pending_proof struct definition, the finalize and destroy FFI functions, and the start function skeleton [117][118][119][120][121][122][123]. But in the process of restructuring the existing code, the assistant discovered a fundamental flaw in the initial design.

The prep_msm_thread — the background thread that runs b_g2_msm — captured local variables by reference via the lambda capture [&, num_circuits]. The GPU threads also captured results and batch_add_res by reference. The initial plan was to move these locals into the handle after the GPU threads had joined but before returning. But the whole point of the split API was that b_g2_msm was still running when we wanted to return. Moving results while prep_msm_thread was writing to results.b_g2[c] would create a use-after-move bug — a classic concurrency error that would manifest as intermittent memory corruption [126][127][128].

The solution was elegant and decisive: allocate the groth16_pending_proof handle on the heap before any threads are spawned. Use its fields — results, batch_add_res, the split vectors, the tail MSM bases — as the actual storage locations. Pass references to these fields to both the GPU threads and prep_msm_thread. The handle itself is the storage; it never moves. When the function returns, it simply returns a pointer to the handle. The threads are already referencing its fields by address, and those addresses remain valid for the lifetime of the handle [126].

This was a "bigger refactor but it's the clean way," as the assistant noted. Every local variable that needed to survive the early return had to be replaced with a field access on the handle. The assistant systematically worked through the code, replacing declarations and adjusting captures, ensuring that every piece of shared state lived at a stable memory address from the very beginning [126][127][128][129][130][131].

The Grep That Saved a Proof

One of the most instructive moments in the implementation was a seemingly trivial grep command [132]. The assistant searched for three boolean flags — l_split_msm, a_split_msm, b_split_msm — that controlled how multi-scalar multiplication results were assembled across GPUs. These flags were set during prep_msm (lines 537, 542, 549) and read during the epilogue. In the original monolithic function, they were simple local booleans declared at line 364. But in the split API, they needed to live in the handle.

The grep revealed that the flags were already declared in the groth16_pending_proof struct (added in an earlier edit), initialized in the constructor, and read in the finalize function. But the main function still had local declarations that needed to be replaced with aliases pointing into the heap-allocated handle. Without this verification, the finalize function would read uninitialized or dangling stack memory, producing incorrect MSM assembly and ultimately an invalid Groth16 proof [132].

This single grep command exemplifies the discipline required for safe cross-language refactoring. The assistant could have assumed the flags were handled and moved on. Instead, it paused, verified, and ensured that every piece of state that needed to survive the split was properly accounted for.

The Broader Significance

The transition from Phase 11 to Phase 12 in this chunk represents a fundamental shift in optimization strategy. Phase 11 was parametric: it tuned thread counts, serialized operations, and added flags to reduce contention within the existing architecture. Phase 12 is architectural: it restructures the GPU worker pipeline to decouple GPU kernel execution from CPU post-processing, changing how work flows through the system.

This shift was enabled by the systematic measurement discipline of Phase 11. Without the benchmark data showing that b_g2_msm had slowed to 1.7 seconds, without the timeline analysis revealing the idle gaps, and without the user's sharp question about shipping b_g2_msm to a separate thread, the architectural insight would not have emerged. The 3.4% improvement was modest, but the understanding it produced was transformative.

The chunk also demonstrates the importance of concurrency-aware design in cross-language systems. The use-after-move bug that the assistant discovered and fixed is a class of error that is notoriously difficult to debug because it manifests as intermittent corruption rather than a clean crash. The assistant's careful analysis of reference lifetimes — tracing which threads capture which variables by reference, and ensuring that all shared state lives at stable heap addresses — prevented a bug that could have wasted hours of debugging time in production.

Conclusion

This chunk of the opencode session captures a pivotal transition in a high-stakes optimization campaign. Phase 11's three memory-bandwidth interventions delivered a 3.4% throughput improvement, but more importantly, they revealed the structural bottleneck that would drive Phase 12: the b_g2_msm computation blocking the GPU worker's critical path. The split API design that emerged from this insight — with its early-allocated pending handle, its two-phase FFI boundary, and its careful attention to memory lifetimes — represents a fundamental architectural improvement to the proving pipeline.

The work in this chunk exemplifies the iterative, measurement-driven approach that characterizes the entire optimization campaign. Each phase tests hypotheses, measures results, and generates insights that inform the next phase. The gains are incremental — 3.4% here, a structural improvement there — but they accumulate into a system that is faster, more efficient, and better understood. The discipline of systematic benchmarking, the willingness to abandon flawed approaches, and the careful attention to concurrency semantics are the practices that make this accumulation possible.