The Benchmark That Revealed the Bottleneck Shift: Phase 9 PCIe Optimization Under Load
Introduction
In the ongoing effort to optimize the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep), a single benchmark run captured in message 2534 represents a pivotal moment of diagnostic clarity. After weeks of progressively deeper investigation—from mapping the call chain from Curio through Rust FFI into C++/CUDA kernels, to implementing PCIe transfer optimizations, to instrumenting fine-grained timing—the assistant launched a high-concurrency benchmark that would confirm a fundamental shift in the bottleneck landscape. The message is deceptively simple: a bash command invoking cuzk-bench with concurrency 20 and 15 proofs, followed by partial output showing the first five proofs completing. But beneath this surface lies the culmination of a diagnostic journey that had just revealed that the bottleneck had moved from GPU kernel execution and PCIe transfers to something far more elusive: CPU memory bandwidth contention on an 8-channel DDR5 system.
The Preceding Investigation
To understand why this particular benchmark matters, we must trace the investigation that led to it. The Phase 9 PCIe optimization had introduced pre-staging of VRAM allocations and host-to-device uploads, aiming to overlap data transfer with GPU computation. Earlier messages (2507–2525) show the assistant adding fine-grained timing instrumentation to the pre-staging path, expecting to find that the 1.8s gap between C++ kernel time (~1.8s) and TIMELINE wall time (~3.8s) was caused by pre-staging overhead. The results were surprising: the pre-staging setup itself took only 12–17ms per partition—essentially negligible. The cudaDeviceSynchronize calls took 0–1ms, the 12 GiB cudaMalloc took 11–15ms, and the async upload was queued in 0ms.
This forced a re-examination. Where was the missing 1.9s per partition going? Further instrumentation revealed the answer: the CPU-side prep_msm (1.7s typical) and b_g2_msm (~380ms typical) operations were running on a separate thread that the C++ function waited to join after the GPU threads had finished. When GPU kernels completed in ~1.4s but prep_msm took 1.7s, the function blocked for 300ms waiting for the CPU thread. Worse, outliers showed b_g2_msm ballooning to 2.7s and even 4.9s—clear evidence of CPU memory bandwidth contention when synthesis workers were simultaneously hammering the same DDR5 channels.
The user had presciently suggested this root cause in message 2514: "Also maybe the whole synth+gpu pipeline is hitting the 8 chan ddr5 bw limitations." The assistant confirmed this theory in message 2527, concluding that "the GPU is actually idle waiting for the CPU to finish."
The Message: Running the Big Benchmark
With this diagnosis in hand, the assistant needed to establish steady-state performance numbers at higher concurrency to confirm the memory bandwidth contention theory at scale. Earlier runs had used only 3 proofs (c=3, j=3), which didn't saturate the memory channels. The user had suggested testing c=15–30. Message 2534 represents the first of these larger runs:
Now let me run the big benchmark: c=20 j=15.
>
``bash echo "=== Phase 9 gw=1 c=20 j=15 ===" && time /home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch \ --type porep --c1 /data/32gbench/c1.json \ -c 20 -j 15 2>&1 | tee /tmp/cuzk-p9-c20-j15-bench.log ``
>
=== Phase 9 gw=1 c=20 j=15 === ... [1/20] COMPLETED — 75.9s (prove=40361 ms, queue=291 ms) [2/20] COMPLETED — 117.9s (prove=40699 ms, queue=730 ms) [3/20] COMPLETED — 157.6s (prove=37946 ms, queue=1181 ms) [4/20] COMPLETED — 200.8s (prove=38212 ms, queue=1633 ms) [5/20]...
The choice of parameters is deliberate. Concurrency 20 with 15 proofs ensures that at any given moment, up to 20 proofs could be in flight simultaneously, each requiring its own synthesis worker and GPU partition processing. With 10 synthesis workers (the partition_workers setting) all doing heavy memory reads over multi-GiB witness data, and the GPU worker's CPU thread doing prep_msm and b_g2_msm on multi-GiB point tables, the 8-channel DDR5 memory subsystem would be thoroughly saturated.
Reading the Partial Results
Even the incomplete output tells a revealing story. The per-proof wall times are relatively stable at 38–41s of prove time, consistent with earlier single-worker measurements. But the queue times tell a different story: they grow monotonically from 291ms to 730ms to 1181ms to 1633ms. This is the signature of a system approaching saturation—each new proof waits longer in the queue because the previous proofs are still occupying resources. The pipeline is backing up.
The prove time itself (~38–41s) is the critical number. Earlier Phase 8 benchmarks with optimal partition_workers settings had achieved ~36–39s per proof. Phase 9's PCIe optimization was expected to improve this, but the numbers are essentially flat. The reason is now clear: the bottleneck has moved. PCIe transfers and GPU kernel execution are no longer the limiting factors. Instead, the CPU memory bandwidth contention between synthesis workers and the prep_msm/b_g2_msm thread creates a CPU-side bottleneck that the GPU spends significant time idle waiting for.
Why This Matters
This benchmark is not merely another data point—it is a confirmation of a bottleneck shift that fundamentally changes the optimization strategy. The project had been operating under the assumption that GPU utilization was the key metric to optimize. Phase 7 had introduced per-partition dispatch to improve GPU utilization. Phase 8 had added dual-GPU-worker interlock to eliminate CPU-side mutex contention. Phase 9 targeted PCIe transfer overhead. Each phase had moved the bottleneck, but always toward the GPU.
Now, the bottleneck has moved to the CPU memory subsystem—a place where CUDA kernel optimizations and PCIe transfer improvements cannot reach. The GPU is no longer the constraint; the CPU cannot feed it fast enough because all cores are competing for the same memory bandwidth. This is a fundamentally different class of problem that requires architectural changes rather than micro-optimizations.
The assistant's thinking process reveals this understanding. In message 2527, the assistant wrote: "This is a CPU memory bandwidth problem, as you suggested. The prep_msm does heavy memory reads (building split vectors from multi-GiB point tables), and with 10 synthesis workers also doing heavy memory reads, they're all competing for 8-channel DDR5. Not much we can do about that from the CUDA side."
The Broader Implications
This benchmark also demonstrates a crucial principle in systems optimization: the bottleneck always moves. Each optimization phase successfully addressed its target bottleneck, but the system's throughput ceiling was determined by the next constraint in the chain. Phase 7 improved GPU utilization, but revealed CPU-side mutex contention. Phase 8 fixed the mutex, but revealed PCIe transfer overhead. Phase 9 addressed PCIe, but revealed CPU memory bandwidth contention.
The message also highlights the importance of running benchmarks at realistic concurrency levels. Early testing with c=3 gave optimistic numbers that didn't reveal the memory bandwidth contention. Only when the system was pushed to higher concurrency (c=20) did the true bottleneck emerge. This is a lesson in benchmark methodology: single-threaded or low-concurrency measurements can miss systemic bottlenecks that only appear under load.
Conclusion
Message 2534 captures the moment when the optimization team confirmed that the bottleneck had shifted from the GPU to the CPU memory subsystem. The partial benchmark output—five proofs completing with growing queue times and flat prove times—tells a story of a system hitting a new wall. The Phase 9 PCIe optimization had successfully eliminated one bottleneck, only to reveal a deeper constraint in the memory architecture. This diagnosis would drive the next phase of the project: designing a two-lock architecture (Phase 10) to better overlap CPU and GPU work, and eventually recognizing that device-global synchronization conflicts made even that approach infeasible without fundamental redesign. The benchmark in message 2534 is thus a turning point—the moment when the team realized that further GPU-side optimizations would yield diminishing returns, and that the path forward required rethinking the entire CPU-GPU work partitioning strategy.