The Moment the Bottleneck Shifted: Diagnosing CPU Memory Bandwidth Contention in a GPU Proving Pipeline

In the relentless pursuit of throughput for Filecoin's Proof-of-Replication (PoRep) Groth16 proof generation, an optimization journey had reached a curious plateau. The Phase 9 PCIe transfer optimization had successfully reduced GPU-side kernel time to a lean ~1.8 seconds per partition, and pre-staging overhead had been instrumented down to a mere ~15 milliseconds. Yet the steady-state throughput stubbornly refused to improve beyond ~41 seconds per proof. Something was swallowing nearly two seconds per partition that no one could account for. Message 2526 is the precise moment that mystery unravels — a diagnostic breakthrough that reveals the bottleneck has shifted from the GPU and PCIe bus to an entirely unexpected culprit: CPU memory bandwidth contention on an 8-channel DDR5 system.

The Context: A Pipeline Under Investigation

To understand why this message was written, one must appreciate the arc of the optimization work that preceded it. The cuzk SNARK proving engine, built atop the supraseal-c2 CUDA library, generates Groth16 proofs for Filecoin sectors. Each proof requires processing 10 partitions, each partition involving substantial GPU compute (NTTs, MSMs, batch additions) and CPU-side preprocessing (vector splitting, G2 MSM operations). The Phase 9 optimization had introduced VRAM pre-staging — allocating and uploading data to GPU memory before the mutex-locked compute phase — to overlap memory management with kernel execution.

The user had observed something puzzling: actual GPU compute (visible as power draw and memory activity) seemed to be active only about 50% of the time, despite TIMELINE metrics showing 90% GPU utilization. The assistant's investigation in preceding messages had already revealed a critical discrepancy: C++ kernel time averaged 1,834ms per partition, but the TIMELINE gpu_ms (GPU_START to GPU_END wall time) averaged 3,760ms — a gap of nearly two seconds. The initial hypothesis was that pre-staging overhead (cudaDeviceSynchronize, pool trim, cudaMalloc for 12 GiB, async uploads) was consuming that time. But fine-grained instrumentation proved that hypothesis spectacularly wrong: the entire pre-staging setup took only 12–17ms. The gap remained unexplained.

The Discovery: prep_msm and b_g2_msm

Message 2526 opens with the assistant reading the timing logs and delivering the answer:

prep_msm_ms: 1.7-2.3s typical (CPU preprocessing: split vectors + batch add setup). One outlier at 5.3s (first run, cold cache). b_g2_msm_ms: ~340-400ms typical, but two huge outliers: 2763ms and 4894ms — those are CPU memory bandwidth contention with synthesis workers.

These two numbers — prep_msm_ms and b_g2_msm_ms — had been hiding in plain sight. The C++ code had always logged them, but they had been overlooked because the investigative focus had been on GPU-side operations. The prep_msm phase runs on a dedicated CPU thread, performing vector splitting and batch addition setup for the multi-scalar multiplication. The b_g2_msm phase runs the G2 MSM on the CPU, also on a separate thread. Together, these CPU operations consume 2.1 seconds per partition — and crucially, they run after the GPU threads have already finished their work.

The Architectural Flaw: Join Ordering

The message then pivots to the source code, reading the critical lines around the thread join point:

955:     // then release the GPU lock so another worker can start its kernels.
956:     // prep_msm_thread's b_g2_msm (CPU-only) may still be running — that's
957:     // fine, it doesn't need the GPU.
958:     for (auto& tid : per_gpu)
959:         tid.join();

The comment on line 956-957 reveals the original design intent: the developers knew that b_g2_msm might still be running after GPU threads finish, and they considered this acceptable because "it doesn't need the GPU." But what they missed — and what the assistant's analysis now exposes — is that the function blocks on prep_msm_thread.join() at line 984, which cannot complete until b_g2_msm finishes. If the CPU work (2.1s) exceeds the GPU work (1.4s), the entire function stalls waiting for the CPU thread, even though the GPU is idle and could be processing the next partition.

The assistant summarizes the issue with devastating clarity:

The issue: prep_msm_thread.join() happens at line 984, after the GPU threads finish. If prep_msm + b_g2_msm (1.7s + 0.4s = 2.1s) takes longer than the GPU work (1.4s), the function blocks waiting for the CPU thread. And those two outliers (2.7s, 4.9s b_g2_msm) confirm DDR5 bandwidth contention is real — when synthesis workers are hammering memory, the CPU MSM slows down dramatically.

This is the moment the bottleneck shifts. The Phase 9 PCIe optimization had been so effective at reducing GPU kernel time that the GPU now finishes its work in ~1.4s, leaving it idle for ~600ms while waiting for the CPU thread to complete. The bottleneck is no longer PCIe transfers or GPU kernel execution — it is CPU memory bandwidth contention between the synthesis workers (10 concurrent threads evaluating constraints over multi-GiB witness data) and the CPU MSM operations.

The Timeline Breakdown

The assistant constructs a detailed per-partition timeline table that crystallizes the entire pipeline:

| Phase | Time | Where | |---|---|---| | Pre-staging setup | ~15ms | In mutex, before kernels | | NTT+H MSM | ~690ms | GPU kernels | | Batch additions | ~650ms | GPU kernels | | Tail MSMs | ~83ms | GPU kernels | | prep_msm | ~1.7s | CPU thread, overlaps with GPU | | b_g2_msm | ~380ms | CPU thread, after prep | | Epilogue (joins, result accum, Rust serialization) | ~??? | CPU |

This table is the key output knowledge created by this message. It partitions the per-partition wall time into its constituent phases and identifies exactly where the waste is. The GPU phases (NTT+H MSM, batch additions, tail MSMs) total ~1.4s, but the CPU phases (prep_msm + b_g2_msm) total ~2.1s — and because of the join ordering, the GPU sits idle for the difference.

Assumptions and Their Revision

Several assumptions were challenged and revised in this message:

The pre-staging hypothesis: The assistant had assumed that the 1.8s gap between kernel time and wall time was caused by pre-staging overhead (cudaDeviceSynchronize, pool trim, cudaMalloc, async uploads). Fine-grained timing proved this wrong — pre-staging took only ~15ms. This forced a search for the real culprit.

The GPU-boundedness assumption: Earlier phases had operated under the assumption that the pipeline was GPU-bounded — that GPU kernel execution was the critical path. The Phase 9 optimization was so effective that it flipped the bottleneck to the CPU side, a shift that went undetected until this message.

The overlap assumption: The code comment on line 956-957 assumed that because b_g2_msm doesn't need the GPU, its continued execution after GPU thread join was harmless. The assistant's analysis reveals that while it doesn't need the GPU, it delays the release of the GPU mutex because the main function blocks on prep_msm_thread.join() before releasing the lock. This means the next worker cannot start its GPU work until the CPU thread finishes.

Input Knowledge Required

To fully grasp this message, the reader needs understanding of several domains:

Groth16 proving pipeline: Knowledge that proof generation involves both GPU-accelerated operations (NTTs, MSMs over G1) and CPU-only operations (G2 MSM, vector preprocessing). The G2 MSM runs on CPU because the G2 curve operations don't benefit from GPU acceleration in this implementation.

CUDA threading model: Understanding that cudaDeviceSynchronize is device-global, that cudaMalloc is synchronous, and that async uploads via streams can overlap with kernel execution. The pre-staging design uses CUDA streams and events to overlap uploads with computation.

CPU memory bandwidth contention: Knowledge that DDR5 memory bandwidth is a shared resource. With 10 synthesis workers and a CPU MSM thread all performing heavy memory reads, the 8-channel DDR5 system (~300 GB/s theoretical) becomes a bottleneck. The outliers at 2.7s and 4.9s for b_g2_msm (vs. ~380ms typical) demonstrate 7–13× slowdowns under contention.

The cuzk architecture: Understanding that the engine uses a partition_workers model where multiple GPU workers compete for a mutex, and synthesis workers run independently. The interaction between these thread pools through shared memory bandwidth is the root cause.

Output Knowledge Created

This message produces several pieces of critical knowledge:

  1. The bottleneck has shifted: The pipeline is no longer GPU-bounded or PCIe-bounded. It is CPU memory bandwidth-bounded, with the CPU MSM operations (prep_msm + b_g2_msm) consuming 2.1s per partition and blocking GPU progress.
  2. The specific timing breakdown: A per-partition accounting that separates GPU kernel time (~1.4s) from CPU thread time (~2.1s) from pre-staging (~15ms). This enables targeted optimization.
  3. The join ordering problem: The architectural flaw where prep_msm_thread.join() blocks after GPU threads complete, preventing the next partition from starting GPU work. This is the lever for Phase 10's two-lock design.
  4. DDR5 bandwidth contention evidence: The outlier data (2.7s, 4.9s b_g2_msm) provides concrete evidence that CPU memory bandwidth is the limiting factor under high concurrency, confirming the user's earlier hypothesis.

The Thinking Process

The reasoning arc visible in this message is a masterclass in diagnostic elimination. The assistant starts with the timing data — the raw prep_msm_ms and b_g2_msm_ms values — and immediately recognizes their significance. The typical values (1.7–2.3s for prep_msm, ~380ms for b_g2_msm) are large enough to explain the 1.8s gap. The outliers (2.7s, 4.9s) provide the causal link to memory bandwidth contention.

The assistant then constructs the timeline table, which serves as a mental model of the pipeline. The table forces a comparison: GPU work totals ~1.4s, CPU work totals ~2.1s. The imbalance is obvious once laid out.

The final step is reading the source code to confirm the join ordering. The assistant reads lines 955-962, finding the comment that reveals the developers' assumption ("prep_msm_thread's b_g2_msm (CPU-only) may still be running — that's fine, it doesn't need the GPU"). The assistant's analysis exposes the flaw in this assumption: while the CPU thread doesn't need the GPU, the main function blocks on joining it, which delays mutex release and prevents the next worker from starting.

This thinking process — measure, model, verify — is the pattern that drives the entire optimization effort. The message ends with the assistant about to read further into the epilogue area, setting up the next phase of investigation that will lead to the Phase 10 two-lock design.

Significance in the Larger Arc

Message 2526 is a turning point in the optimization journey. Before it, the team was optimizing GPU-side operations (pre-staging, PCIe transfers, kernel launch parameters) under the assumption that the GPU was the bottleneck. After it, the focus shifts to CPU-side restructuring: overlapping CPU and GPU work through a two-lock design, increasing gpu_workers_per_device to hide CPU latency, and ultimately redesigning the locking protocol to eliminate the serialization.

The message also demonstrates a crucial principle in systems optimization: bottlenecks are not static. An optimization that successfully eliminates one bottleneck can reveal another, deeper one — and the new bottleneck may be in an entirely different subsystem. The Phase 9 PCIe optimization was so effective that it transformed a GPU-bounded pipeline into a CPU memory bandwidth-bounded one, a shift that required entirely new diagnostic tools and a fundamentally different optimization strategy.

The user's earlier observation — that actual GPU compute looked like ~50% utilization — was the first hint of this shift. Message 2526 provides the quantitative confirmation and the architectural root cause, setting the stage for the Phase 10 redesign that follows.