A Milestone of Threes: The Phase 11 Intervention Benchmark

"All 3 interventions compile. Now let me benchmark with all 3 active (Int1 dealloc mutex + Int2 gpu_threads=32 + Int3 membw_throttle)"

This single sentence, issued by the AI assistant at message index 2803 in a months-long optimization campaign for the Filecoin Groth16 proving pipeline, is deceptively compact. It marks the culmination of a sustained, multi-day investigation into DDR5 memory bandwidth contention — the root cause of GPU underutilization that had survived nine prior optimization phases. Behind those twenty words lies a story of iterative diagnosis, cross-language FFI engineering, and a pragmatic triage of three interventions designed to squeeze every last drop of throughput from a system already pushed to its limits.

The Context: A Bottleneck Hunt Across Nine Phases

To understand why this message matters, one must understand the journey that preceded it. The optimization campaign for cuzk — a GPU-accelerated SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) — had already executed eight distinct optimization phases before arriving at Phase 11. Phase 8 introduced a dual-worker GPU interlock that improved throughput by 13–17%. Phase 9 optimized PCIe transfers for another 14.2% gain. Phase 10 attempted a two-lock overlap architecture but was abandoned after discovering fundamental CUDA device-global synchronization conflicts that caused both out-of-memory errors and performance regressions.

The post-mortem of Phase 10 was critical: it revealed that the real bottleneck was not GPU synchronization at all, but DDR5 memory bandwidth contention between CPU threads. The GPU workers were stalling not because of GPU-side conflicts, but because the CPU-side synthesis pipeline (specifically the SpMV sparse matrix-vector multiplication in evaluate_pce) was thrashing the L3 cache and saturating memory bandwidth, starving the CPU threads that were trying to prepare data for the GPU.

Phase 11 was designed around three targeted interventions to address this root cause. The message at index 2803 announces that all three have been implemented and are ready for benchmarking.

The Three Interventions: A Study in Targeted Optimization

Intervention 1 — Dealloc Mutex Serialization: The first intervention addressed a subtle but costly pattern in the C++ CUDA code. Asynchronous deallocation of GPU resources (async_dealloc) was racing with allocation requests from other threads, causing contention on CUDA's internal memory management locks. By serializing these deallocations behind a static std::mutex — mirrored on the Rust side with a corresponding DEALLOC_MTX — the intervention eliminated the contention without introducing significant serialization delay, since deallocations are infrequent relative to allocations.

Intervention 2 — GPU Thread Pool Reduction: The second intervention was the most impactful. The groth16_pool (the CPU thread pool used for b_g2_msm and other pre-computation) had been using 192 threads — a setting that maximized throughput for the Pippenger multi-scalar multiplication but also maximized L3 cache pressure. By reducing gpu_threads to 32, the assistant deliberately traded raw b_g2_msm speed for reduced memory subsystem contention. The trade was asymmetric: b_g2_msm slowed from ~0.5s to ~1.7s (a 3–4x slowdown), but the overall proof time improved by 3.4% (from 38.0s to 36.7s) because the synthesis pipeline — which dominates total runtime — benefited disproportionately from the freed memory bandwidth.

Intervention 3 — Memory Bandwidth Throttle: The third intervention was the most architecturally interesting. It introduced a global atomic throttle flag that the C++ code sets before entering the b_g2_msm computation and clears after exiting. On the Rust side, the spmv_parallel function (the heart of the constraint evaluation) checks this flag during its inner loop and calls std::thread::yield_now() when the flag is set, voluntarily yielding the CPU to reduce memory bandwidth contention during b_g2_msm's window. This required careful cross-language FFI design: a Rust-side static AtomicI32 in the cuzk-pce crate, exposed as an extern "C" function callable from C++.

The Implementation Journey: FFI and Link-Time Resolution

The implementation of Intervention 3 reveals the complexity of cross-language optimization. The assistant initially considered having Rust alias a C++ atomic variable through unsafe pointers, but rejected this approach as fragile. Instead, a cleaner design emerged: a Rust-side static atomic with a #[no_mangle] pub extern "C" function, called from C++ via an extern "C" declaration.

This design required careful attention to linker visibility. The dependency chain runs: cuzk-daemon → cuzk-core → cuzk-pce + bellperson → supraseal-c2. The C++ code lives in supraseal-c2's CUDA sources, while the throttle flag lives in cuzk-pce. For the symbol to resolve at link time, both crates must be linked into the same binary — which they are, through cuzk-daemon. But the assistant had to verify this chain explicitly (see [msg 2793]–[msg 2796]) before proceeding.

A subtle compilation error emerged during implementation: the extern "C" declaration for set_membw_throttle was initially placed inside a C++ lambda body, which is not valid in CUDA C++. The assistant had to move it to file scope, adjacent to the existing create_gpu_mutex and destroy_gpu_mutex declarations (see [msg 2799]–[msg 2800]). This kind of error is characteristic of cross-language FFI work — the syntax is valid in one context but not another, and the error messages from the NVCC compiler can be opaque.

The Benchmarking That Followed

After the subject message, the assistant launched the daemon and ran a batch benchmark at concurrency 20 with 15 jobs (c=20 j=15). The result: 36.8 seconds per proof with all three interventions active — essentially identical to the 36.7 seconds achieved by Intervention 2 alone. Intervention 3 had negligible additional impact.

This outcome, while disappointing for Intervention 3, was logically consistent: with gpu_threads=32, the b_g2_msm computation already uses fewer threads, so its memory bandwidth pressure during its execution window is already reduced. The throttle mechanism added overhead (atomic checks, yield calls) without providing additional benefit because the contention was already mitigated by Intervention 2.

The user then asked the assistant to try 3 and 4 GPU workers per device ([msg 2807]), which led to further experimentation. But the core insight from the Phase 11 benchmark was clear: Intervention 2 alone captured nearly all the available gain, and the other two interventions were either neutral or redundant.

Assumptions, Knowledge, and the Thinking Process

The message at index 2803 embodies several assumptions:

That all three interventions are independent and composable. The assistant assumed that the benefits of Interventions 1 and 3 would stack additively on top of Intervention 2's 3.4% gain. In practice, they did not — the benchmark showed no measurable improvement beyond Intervention 2 alone. This is a common pitfall in optimization: interventions that target the same bottleneck often have overlapping effects, and the first intervention to address the root cause captures most of the available gain.
That the throttle mechanism would not introduce overhead that negates its benefit. The atomic check in spmv_parallel's inner loop adds a memory load and conditional branch on every iteration. When the throttle flag is not set (the common case), this is pure overhead. The assistant implicitly assumed this overhead was negligible relative to the SpMV computation — an assumption that held, but only because the throttle was never actually triggered in a meaningful way.
That the link-time symbol resolution would work across crate boundaries. This assumption was validated by the successful build, but it required careful verification of the dependency chain. The input knowledge required to understand this message includes: the architecture of the Groth16 proving pipeline (synthesis, SpMV, b_g2_msm, GPU kernels), the CUDA memory model (host registration, pinned memory, device-global synchronization), the Rust FFI model (extern "C", #[no_mangle], link-time resolution), and the specific history of Phases 1–10. The output knowledge created by this message and its surrounding context is: a validated understanding that DDR5 memory bandwidth contention is the dominant bottleneck in the current pipeline, that reducing the GPU thread pool from 192 to 32 threads captures most of the available gain (3.4%), and that further interventions targeting the same root cause are unlikely to yield additional improvements without a more fundamental architectural change.

The Deeper Significance

What makes this message worth studying is what it reveals about the optimization process itself. The assistant did not simply implement a spec and move on. It engaged in a continuous cycle of measurement, hypothesis formation, implementation, and validation. When Intervention 3 showed no benefit, the assistant did not discard the approach — it understood why it didn't help, and that understanding informed the next phase of the work.

The message also reveals the tension between theoretical optimization and practical measurement. Intervention 3 was theoretically sound: if b_g2_msm and SpMV compete for memory bandwidth, then throttling SpMV during b_g2_msm's window should reduce contention. But in practice, Intervention 2 had already reduced b_g2_msm's thread count so dramatically that the contention window was no longer the bottleneck. The theory was correct; the assumptions about the magnitude of the effect were not.

This is the essence of systems optimization: you can reason about bottlenecks all day, but ultimately you must measure. The message at index 2803 is the pivot point between implementation and measurement — the moment when theory meets reality.