The Commit That Closed Phase 11: How Three Memory-Bandwidth Interventions Yielded a 3.4% Victory

In any serious systems optimization project, there comes a moment when the experimentation stops, the data is collected, and the results are codified. Message <msg id=2829> in this opencode session is that moment for Phase 11 of the cuzk SNARK proving engine optimization campaign. It is a single git commit — but that commit encapsulates days of deep memory-subsystem analysis, three carefully designed interventions, a systematic benchmark sweep across multiple configurations, and the hard-won knowledge of what actually moves the needle when the bottleneck shifts from GPU compute to CPU memory bandwidth.

The Message: A Git Commit as a Technical Report

The message is deceptively simple on its surface. It executes a git add of four modified files and a git commit with a structured commit message:

[assistant] [bash] git add cuzk-project.md extern/bellperson/src/groth16/prover/supraseal.rs extern/cuzk/cuzk-pce/src/eval.rs extern/supraseal-c2/cuda/groth16_cuda.cu && git commit -m "feat(cuzk): Phase 11 — memory-bandwidth-aware pipeline scheduling

Three interventions to reduce CPU memory subsystem contention at high
concurrency (c=20 j=15):

1. Serialize async_dealloc threads (static mutex in C++ and Rust) to
   prevent concurrent munmap() TLB shootdown storms. Alone: negligible.

2. Reduce groth16_pool from 192 to 32 threads (gpu_threads=32 config).
   Cuts b_g2_msm L3 cache footprint from ~1.1 GiB to ~192 MB. b_g2_msm
   slows from 0.5s to 1.7s but runs outside GPU lock. Best result:
   36.7s/proof (3.4% improvement over Phase 9 baseline of 38.0s).

3. Memory-bandwidth throttle: global AtomicI32 flag set by C++ around
   b_g2_msm, checked by Rust SpMV every 64 chunks with yield_now().
   No additional gain over Intervention 2 alone.

Also tested gw=3 (37.2s) and gw=4 (37.4s) — both worse due to CPU
contention from additional GPU workers.

Optimal config: gw=2, pw=10, gpu_threads=32 → 36.7s/proof."

But beneath this concise summary lies an entire investigative arc. The commit is not merely recording a code change — it is documenting the conclusion of a root-cause analysis that traced performance degradation at high concurrency all the way down to the DDR5 memory bus and the L3 cache hierarchy.

The Reasoning: Why Phase 11 Existed

To understand why this message was written, one must understand what preceded it. The cuzk project is a CUDA-accelerated Groth16 proving engine for Filecoin's Proof-of-Replication (PoRep). By Phase 9, the system had achieved a baseline throughput of 38.0 seconds per proof at concurrency level c=20 with 15 concurrent jobs (j=15). But the team had identified a troubling pattern: at high concurrency, the GPU utilization would dip, and the bottleneck had shifted from GPU compute to the CPU memory subsystem.

Phase 10 had attempted a two-lock GPU interlock architecture to increase overlap between GPU workers, but that design was abandoned after discovering fundamental CUDA device-global synchronization conflicts that caused Out Of Memory errors and performance regression. The code was reverted to the Phase 9 single-lock design, and a comprehensive waterfall timing analysis identified the true culprit: DDR5 memory bandwidth contention between the CPU-side synthesis (SpMV) and the post-GPU-lock b_g2_msm computation.

Phase 11 was born from that diagnosis. The commit message tells the story of three targeted interventions, each designed to reduce pressure on a specific part of the memory subsystem.

The Three Interventions: A Study in Targeted Optimization

Intervention 1 — Serializing async_dealloc targeted TLB shootdown storms. The Rust async deallocation of GPU buffers was spawning concurrent munmap() calls, each of which triggers a TLB shootdown IPI (Inter-Processor Interrupt) across all CPU cores. At high concurrency, these shootdowns were causing measurable latency spikes. The fix was a static mutex in both C++ and Rust that serialized the deallocations. The result: "Alone: negligible." This is a classic optimization outcome — the intervention addressed a real phenomenon, but it was not the dominant bottleneck.

Intervention 2 — Reducing groth16_pool from 192 to 32 threads was the winner. The groth16_pool is a thread pool used by the C++ CUDA code for the Pippenger multi-scalar multiplication (MSM) that computes the G2 element b_g2_msm. With 192 threads, each Pippenger bucket required ~1.1 GiB of RAM, causing massive L3 cache thrashing that interfered with the Rust synthesis SpMV running concurrently on the same CPU. By reducing to 32 threads (gpu_threads=32), the bucket RAM dropped to ~192 MB, dramatically reducing L3 cache interference. The trade-off was that b_g2_msm slowed from 0.5 seconds to 1.7 seconds — but crucially, this computation runs after the GPU lock is released, so it does not block the GPU pipeline. The net effect was a 3.4% throughput improvement, from 38.0 to 36.7 seconds per proof.

Intervention 3 — Memory-bandwidth throttle was the most architecturally creative. It introduced a global AtomicI32 flag that the C++ code would set to true around the b_g2_msm computation, and the Rust SpMV code would check every 64 chunks, calling yield_now() to voluntarily yield the CPU if the flag was set. The idea was to reduce memory bandwidth contention during the b_g2_msm window by having the synthesis threads temporarily back off. In practice, "No additional gain over Intervention 2 alone." Once the thread count was reduced, the bandwidth contention was no longer severe enough for the throttle to matter.

The Decision-Making: What Was Tried and Rejected

The commit message also records two negative results: testing with 3 and 4 GPU workers per device (gw=3 and gw=4). These were motivated by the user's suggestion to try more GPU workers for better pipeline overlap. The results were 37.2 and 37.4 seconds per proof respectively — both worse than the 36.7 second baseline with 2 workers. The commit correctly diagnoses why: "both worse due to CPU contention from additional GPU workers." Each additional worker competes for CPU time during the prep_msm and synthesis phases, inflating per-proof latency without improving GPU utilization enough to compensate.

This is a textbook example of Amdahl's Law in action: adding more parallelism to a system whose bottleneck is elsewhere (memory bandwidth, not GPU compute) does not help and can hurt.

The Assumptions and Their Validation

The Phase 11 investigation rested on several assumptions, all of which were tested and either validated or refuted:

  1. The bottleneck is CPU memory bandwidth, not GPU compute. This was validated by the waterfall timing analysis that preceded Phase 11, and confirmed by the fact that reducing CPU-side memory pressure (Intervention 2) improved throughput.
  2. TLB shootdowns from concurrent munmap() are a significant contributor. This was tested by Intervention 1 and found to be negligible — a correct assumption that turned out to be wrong about the magnitude.
  3. Voluntary yielding during b_g2_msm can reduce contention. This was tested by Intervention 3 and found to provide no additional benefit once the thread count was already reduced — the assumption was correct in theory but the effect was already addressed.
  4. More GPU workers can hide latency. This was tested with gw=3 and gw=4 and found to be false at this stage — the CPU contention from additional workers outweighed any GPU pipeline benefits.

Input and Output Knowledge

To fully understand this message, one needs input knowledge of: the cuzk project architecture (GPU worker model, partition synthesis, the Groth16 proof pipeline), the concept of TLB shootdowns and their performance impact on multi-core systems, the Pippenger MSM algorithm and its memory footprint characteristics, the L3 cache hierarchy and how thread count affects per-thread cache allocation, and the benchmark methodology (c=20 j=15, concurrency control, the difference between throughput and per-proof latency).

The message creates output knowledge that is both specific and generalizable. Specifically, it establishes that the optimal configuration for the current hardware is gw=2, pw=10, gpu_threads=32, yielding 36.7s/proof. More generally, it demonstrates that when the bottleneck is CPU memory bandwidth, reducing thread count to shrink cache footprint can be more effective than adding more parallelism — a counterintuitive result that runs against the instinct to throw more threads at a performance problem.

The Thinking Process: What the Commit Doesn't Say

What makes this message fascinating is what it implies about the thinking process that led to it. The commit is the final act of a multi-day investigation that involved: reading waterfall timing logs to identify the b_g2_msm window, analyzing L3 cache miss rates, calculating the memory footprint of Pippenger buckets at different thread counts, implementing cross-language FFI for the throttle flag (C++ setting a Rust-visible atomic), debugging linker errors when the extern "C" declaration was placed inside a lambda body, and running multiple benchmark sweeps with careful isolation.

The commit message's structured format — listing each intervention, its mechanism, and its measured effect — reveals a scientific mindset. The assistant is not just making changes and hoping for improvement; it is formulating hypotheses, implementing precise tests, measuring outcomes, and recording what was learned even from negative results. The inclusion of "Alone: negligible" and "No additional gain over Intervention 2 alone" is particularly telling — these are honest admissions that not every idea worked, which is the hallmark of rigorous engineering.

The Larger Arc

This commit sits at a transition point in the optimization campaign. Phase 11 closed the chapter on memory-bandwidth interventions with a modest but real 3.4% gain. But the commit also reveals the next frontier: the b_g2_msm computation, which now takes 1.7 seconds (up from 0.5 seconds due to the reduced thread count), still blocks the GPU worker from picking up the next job. The commit notes that it "runs outside GPU lock" — meaning it doesn't block the GPU itself — but it still blocks the worker thread from starting the next proof cycle.

This observation would directly motivate Phase 12, which immediately follows this commit in the conversation. Phase 12's split API design — decoupling the GPU worker's critical path from the CPU post-processing by spawning a separate tokio task for finalization — is the natural next step, and it is already visible in the thinking behind this commit. The 1.7-second b_g2_msm window that the commit documents becomes the target for the next optimization.

Conclusion

Message <msg id=2829> is far more than a routine git commit. It is a technical report, a decision record, and a roadmap all in one. It captures the conclusion of a deep-dive into CPU memory subsystem behavior, documents three carefully tested interventions with their measured outcomes, records negative results that are as valuable as positive ones, and identifies the next bottleneck to attack. For anyone studying systems optimization methodology, this single message demonstrates the essential pattern: hypothesize, implement, measure, learn, and commit what you know — even when the answer is "this didn't help."