From Perfect GPU-Boundedness to Hidden PCIe Idle: The Phase 9 Optimization Campaign in the cuzk SNARK Proving Engine

Introduction

The cuzk pipelined SNARK proving engine had reached an apparent plateau. Phase 8's dual-worker GPU interlock had driven GPU utilization to 100%, and the measured throughput of 37.4 seconds per proof exactly matched the serial CUDA kernel time of 10 partitions × 3.75 seconds. By every conventional metric, the system was perfectly GPU-bound — the GPU was never idle, synthesis was fully overlapped with proving, and cross-sector transitions were under 50 milliseconds after warmup. Further CPU-side optimizations like synthesis_concurrency=2 would provide no benefit, and the TIMELINE analysis conclusively showed that the bottleneck had shifted entirely to the CUDA kernel execution itself.

But the user noticed something that the aggregate metrics missed. Despite 100% GPU utilization, there were subtle dips in GPU power consumption and utilization that correlated with bursts of PCIe traffic reaching 50 GB/s. These dips were invisible in the TIMELINE analysis because the GPU was never fully idle — it was just running at reduced throughput during data transfer phases. The GPU's Streaming Multiprocessors (SMs) were stalling while waiting for data to arrive over PCIe, and these micro-stalls were hidden inside the "busy" intervals because the GPU was still executing kernel launches and memory operations even though actual compute was paused.

This discovery launched a deep investigation into the PCIe transfer patterns inside the GPU mutex-protected region, culminating in the design of Phase 9 — a two-tier optimization campaign to eliminate hidden PCIe stalls and push the cuzk engine beyond its current GPU-bound plateau.

The TIMELINE Revelation

The segment began with a systematic analysis of the Phase 8 partition_workers sweep data. The assistant wrote a series of Python analysis scripts to extract every TIMELINE event from the benchmark logs, reconstructing the precise sequence of synthesis and GPU operations across all five sectors of the benchmark run.

The Perfect GPU-Bound Discovery

The TIMELINE analysis of the pw=10 run (commit 2fac031f) revealed a remarkable finding: the system was operating at the theoretical maximum for the given hardware. The key numbers were:

Implications for Further Optimization

The TIMELINE analysis had immediate practical implications. It meant that:

  1. synthesis_concurrency=2 would not help — synthesis was already fully overlapped with GPU work. Running two sectors' synthesis simultaneously would only increase memory pressure without improving throughput.
  2. The control benchmark (gpu_workers_per_device=1) was unnecessary — the TIMELINE already proved that the dual-worker interlock was achieving 100% GPU utilization, so running with a single worker would only confirm that throughput would drop.
  3. The bottleneck was purely the CUDA kernel speed — any further throughput improvement required either faster CUDA kernels or additional GPUs. These conclusions were documented and committed as f5bb819a, updating cuzk-project.md with the Phase 6-8 results, the partition workers sweep table, the Phase 7 vs Phase 8 comparison, and the full TIMELINE analysis.

The Hidden PCIe Bottleneck

Despite the perfect GPU-bound conclusion, the user observed something puzzling: GPU utilization and power consumption showed slight dips that correlated with PCIe traffic bursts reaching 50 GB/s. This was a critical observation because it revealed that the GPU was not truly compute-bound during the entire "busy" period — it was spending a significant fraction of time stalled on data transfers.

The 23.6 GiB Transfer Inventory

The assistant launched a subagent to perform a deep analysis of every PCIe transfer happening inside the GPU mutex-protected region. The result was a comprehensive inventory of 23.6 GiB of host-to-device (HtoD) transfers per partition, broken down by phase:

Phase 1: NTT + H MSM (~2.4 seconds)

Root Cause 1: Non-Pinned Host Memory

The most significant finding was that the a/b/c polynomial transfers (6 GiB total) used non-pinned host memory. These vectors were allocated by Rust's Vec<Fr> and passed through the FFI boundary as raw pointers. When cudaMemcpyAsync is called with a non-pinned source pointer, CUDA cannot initiate a true DMA transfer. Instead, it must first stage the data through a small internal pinned bounce buffer (typically 32 MB), which serializes the transfer and effectively halves the achievable bandwidth.

For the RTX 5070 Ti connected via PCIe Gen4 x16, the theoretical peak bandwidth is approximately 32 GB/s. With non-pinned memory, the effective throughput drops to roughly 16 GB/s or less. The 6 GiB of a/b/c transfers alone would take approximately 375 ms at full bandwidth but nearly 750 ms at half bandwidth — and critically, the GPU cannot begin the NTT computations until the data arrives.

Root Cause 2: Per-Batch Pippenger MSM Sync Stalls

The second root cause was more subtle but equally impactful. The Pippenger multi-scalar multiplication (MSM) algorithm processes points in batches, and each batch ends with a hard sync() call. The sync ensures that the GPU-to-host transfer of bucket results is complete before the CPU can process them. However, this creates a stop-start pattern:

  1. GPU finishes compute kernels for batch N
  2. GPU performs DtoH transfer of bucket results (~small, ~1.5 MiB)
  3. GPU issues sync() — GPU stalls until DtoH completes
  4. CPU processes bucket results (collect()) — GPU is idle
  5. CPU launches next batch's uploads and kernels
  6. GPU finally starts batch N+1 With 8 batches in the H MSM and additional batches in the tail MSMs, this pattern repeats 8+ times per partition. Each sync stall might only be tens of microseconds, but the cumulative effect across 10 partitions and 5 sectors adds up to significant GPU idle time.

The Phase 9 Mitigation Plan

Armed with this detailed understanding of the PCIe transfer patterns, the assistant designed a two-tier mitigation plan documented in c2-optimization-proposal-9.md (committed as 673967f2).

Tier 1: Pre-Stage a/b/c Outside the Mutex

The first and most impactful change addresses the non-pinned a/b/c polynomial uploads. The key insight is that the a/b/c data is available at function entry — it comes from the Rust Assignment struct and doesn't depend on any preprocessing. Currently, the upload happens inside the GPU mutex as the first operation of the per-GPU thread, but it could be moved earlier.

The proposed approach uses three CUDA primitives:

  1. cudaHostRegister — Before acquiring the GPU mutex, pin the Rust-allocated memory pages for DMA. This function registers existing host memory with the CUDA driver, making it page-locked for DMA transfers without requiring a data copy. The overhead is approximately 1-5 ms per 2 GiB region.
  2. Pre-allocate device buffers — Allocate d_a, d_b, and d_c device memory before the mutex. The current code allocates d_a inside the mutex and d_b/d_c inside the NTT function. Moving these allocations earlier allows the uploads to start before the mutex is acquired.
  3. Async upload on a dedicated copy stream — Issue cudaMemcpyAsync on a separate copy stream that can overlap with the other GPU worker's CUDA kernels. The GPU's copy engine is independent of the compute engine, so data transfers can proceed in parallel with kernel execution. The VRAM budget analysis confirmed feasibility. With 16 GiB on the RTX 5070 Ti, pre-allocating d_a (2 GiB), d_b (2 GiB), and d_c (2 GiB) consumes 6 GiB. During the NTT phase, the working set peaks at approximately 8-9 GiB (6 GiB pre-allocated + 2 GiB for NTT working memory + ~1 GiB for MSM buckets), leaving 7 GiB of headroom. The expected impact of Tier 1 alone is a reduction of 200-400 ms per partition by eliminating the non-pinned HtoD bottleneck from the mutex-protected region. Across 10 partitions per sector, this translates to 2-4 seconds per proof — a 5-11% improvement over the current 37.4-second baseline.

Tier 3: Deferred Batch Sync in Pippenger MSM

The second change addresses the per-batch sync stalls in the Pippenger MSM. Rather than synchronizing after every batch, the proposal restructures the batch loop to use double-buffered host result buffers and defer the sync to overlap with the next batch's compute.

The current pattern in pippenger.cuh's invoke() method is:

for each batch i:
    upload points + scalars for batch i
    launch compute kernels for batch i
    if i > 0: collect(prev_batch_results)  // CPU processes previous batch
    DtoH bucket results for batch i
    sync()  // GPU stalls until DtoH complete

The proposed pattern is:

for each batch i:
    upload points + scalars for batch i
    launch compute kernels for batch i
    DtoH bucket results for batch i into double-buffer slot (i&1)
    if i > 0:
        sync() on PREVIOUS batch's DtoH  // already complete by now
        collect(prev_batch_results from slot (i-1)&1)
// sync + collect last batch

This restructuring eliminates the GPU idle gap between batches. The sync for batch N-1 happens while batch N's compute kernels are running, so by the time the CPU needs the results, they're already in host memory. The GPU never waits for the CPU to process results.

The expected impact of Tier 3 is 50-200 ms per partition, or 0.5-2 seconds per proof. Combined with Tier 1, the total improvement could reach 2.5-6 seconds per proof (7-16% over baseline).

Design Considerations

The Phase 9 spec also addressed several design considerations:

GPU device context management: The pre-upload operations must happen on the correct GPU device context. The gpu.Dmalloc calls in the existing code select the GPU via select_gpu(), and the pre-upload code must do the same.

Multi-circuit handling: Phase 7's per-partition pipeline always uses num_circuits=1, but the code path also supports num_circuits>1 for Phase 3 batching. The spec proposes asserting num_circuits==1 for the pre-staged path initially, with a fallback to the original code path for multi-circuit cases.

Sppark dependency: The Pippenger change modifies sppark/msm/pippenger.cuh, which is a third-party dependency. The change is localized to the invoke() method and is backward-compatible — the same API, same numerical results, just different scheduling of sync operations.

Cleanup: After mutex release, cudaHostUnregister unpins the a/b/c memory, and the upload stream and events are destroyed. The spec recommends performing cleanup in the epilogue or async deallocation thread to avoid blocking the caller.

Documentation and Knowledge Capture

A significant portion of the segment was devoted to documentation. The assistant updated cuzk-project.md with:

Implications for the cuzk Architecture

The TIMELINE analysis and Phase 9 design have broader implications for the cuzk proving engine's architecture:

The GPU-bound ceiling is real but not absolute. The system is GPU-bound in the sense that the GPU is never idle, but the GPU is spending a significant fraction of its time on data movement rather than computation. The Phase 9 optimizations target this distinction — they don't make the GPU compute faster, but they reduce the overhead of data movement so that a larger fraction of GPU time is spent on actual computation.

The dual-worker interlock enables further optimization. Phase 8's key contribution was creating a window of opportunity for pre-staging: because one worker can do CPU work while the other runs CUDA kernels, the pre-upload operations can overlap with useful work rather than adding latency. Without the dual-worker interlock, pre-staging would just shift idle time from inside the mutex to outside it.

The PCIe bottleneck will become more important with faster GPUs. As GPU compute throughput increases with each generation (Blackwell, Rubin, etc.), the relative cost of PCIe transfers grows. A GPU that can process a partition in 2 seconds instead of 3.75 seconds will be even more sensitive to the 23.6 GiB of data transfers per partition. The Phase 9 optimizations are future-proofing — they ensure that the cuzk engine can scale with GPU compute improvements without being bottlenecked by PCIe.

Conclusion

Segment 25 of the cuzk development journey represents a shift from macro-level pipeline optimization to micro-level GPU efficiency analysis. The TIMELINE analysis confirmed that the Phase 8 dual-worker interlock had achieved its goal of 100% GPU utilization, but also revealed that "100% utilization" doesn't mean "100% compute." The hidden PCIe stalls caused by non-pinned memory transfers and per-batch MSM syncs were creating micro-idle periods that reduced effective compute throughput.

The Phase 9 mitigation plan addresses these issues with a two-tier approach: pre-staging a/b/c polynomial uploads outside the mutex using cudaHostRegister and async copy streams, and restructuring the Pippenger MSM batch loop to eliminate per-batch sync stalls through double-buffered result buffers. Together, these changes could reduce per-proof proving time by 7-16%, breaking through the GPU-bound plateau that had seemed like a fundamental limit.

The segment also demonstrated the importance of combining quantitative analysis with qualitative observation. The TIMELINE metrics showed perfect GPU utilization, but the user's observation of GPU power dips correlated with PCIe traffic revealed a bottleneck that the aggregate metrics missed. This interplay between data-driven analysis and hands-on observation has been a recurring theme throughout the cuzk project, and it continues to drive the engine toward ever-higher efficiency.