From Perfect GPU-Boundedness to Hidden PCIe Idle: The Phase 9 Optimization Campaign in the cuzk SNARK Proving Engine

Introduction

In the high-stakes world of Filecoin proof-of-replication (PoRep) proving, where every second of proof generation time translates directly into operational cost for storage miners, the cuzk SNARK proving engine had reached an enviable milestone. After eight phases of optimization spanning parallel synthesis, per-partition dispatch, and a dual-worker GPU interlock, the system was delivering Groth16 proofs at 37.4 seconds each — a 2.4× improvement over the baseline. A rigorous TIMELINE analysis had declared the system perfectly GPU-bound: the measured throughput exactly matched the serial CUDA kernel time of 10 partitions × 3.75 seconds per partition. Cross-sector GPU transitions after warmup were under 50 milliseconds. Synthesis was fully overlapped with GPU work. By every conventional metric, the CPU-side optimization was complete.

But the user, watching real-time GPU metrics, noticed something the TIMELINE had not captured: slight dips in GPU utilization and power consumption that correlated with bursts of PCIe traffic reaching approximately 50 GB/s. If the GPU was truly running at 100% efficiency, there should be no dips. Something was causing the GPU's streaming multiprocessors to stall while data shuffled across the PCIe bus.

This observation triggered one of the most detailed investigations in the project's history — a forensic accounting of every byte transferred across the PCIe bus inside the GPU mutex-protected region, revealing 23.6 GiB of host-to-device (HtoD) transfers per partition, two distinct root causes of GPU idle gaps, and a two-tier mitigation plan that would reshape the project's understanding of where its bottlenecks truly lie. This article synthesizes that entire arc, spanning messages 2296 through 2367 of the conversation, tracing the journey from macro-level throughput analysis to micro-level transfer accounting, and ultimately to a concrete implementation plan for Phase 9.

Part I: The TIMELINE Analysis and the Discovery of Perfect GPU-Boundedness

The arc begins with the completion of Phase 8 — the dual-worker GPU interlock — which narrowed a C++ static mutex to cover only the CUDA kernel region, allowing two GPU workers per device to overlap their compute while still serializing access to shared GPU state. The assistant had just completed a systematic sweep of the partition_workers parameter from 10 to 20, running five-proof batches at concurrency 3 for each configuration. The results were striking: throughput was essentially flat between pw=10 and pw=20, hovering around 43.5–44.9 seconds per proof, with pw=10 and pw=12 achieving the lowest numbers at 43.5 seconds.

The assistant then performed a deep TIMELINE analysis of the pw=10 benchmark [8][9][10], extracting the coarse-grained event log to understand where the system spent its time. The analysis revealed a system that was perfectly GPU-bound: the 37.4 seconds per proof throughput exactly matched the serial CUDA kernel time of 10 partitions × 3.75 seconds. Cross-sector GPU transitions after warmup were under 50 milliseconds. Synthesis was fully overlapped with GPU work. The assistant concluded that further CPU-side optimizations — such as increasing synthesis_concurrency or running control benchmarks with gpu_workers_per_device=1 — would yield zero benefit and cancelled those experiments.

This conclusion was documented in the project's cuzk-project.md file, which was updated with Phase 6-8 results, benchmark tables, and the TIMELINE analysis, then committed as commit f5bb819a. The tone was one of completion: the bottleneck was the CUDA kernel speed itself, and further gains would require "faster CUDA kernels or more GPUs."

But the user's observation of GPU utilization dips told a different story. The TIMELINE instrumented coarse-grained events — GPU_START and GPU_END per partition — which captured when the GPU began and ended compute work, but not the micro-idle periods within a partition's GPU execution. A partition's CUDA kernel time of 3.75 seconds might contain hundreds of milliseconds of accumulated idle time — gaps where the GPU was waiting for data transfers to complete, for synchronization barriers to clear, or for kernel launch overhead. The TIMELINE would still show a single GPU_START→GPU_END interval of 3.75 seconds, masking these internal dips.

Part II: The User's Observation and the PCIe Transfer Investigation

The user's message at [msg 2333] was a masterclass in systems-level observation [38]: "One thing I still see is there are slight dips in GPU util/power correlated to large pci traffic (50GB/s rx, also large tx sometimes), can those moves be moved outside the 'gpu compute' semaphore?"

This question reframed the entire optimization problem. Instead of asking "how do we make the GPU faster?" — which requires new hardware — it asked "how do we keep the GPU continuously fed with data so it never has to wait?" This is a software architecture question, and it opened a new front of optimization that the TIMELINE analysis had prematurely closed.

The assistant responded by launching a comprehensive investigation. It spawned a subagent task to analyze PCIe transfers inside the GPU mutex-protected region of generate_groth16_proofs_c, then systematically read the relevant source files: groth16_cuda.cu, groth16_ntt_h.cu, groth16_split_msm.cu, and the Pippenger MSM implementation in sppark/msm/pippenger.cuh. Each file revealed another piece of the puzzle. The assistant traced every HtoD call, examined the gpu_t stream infrastructure, checked VRAM capacity with nvidia-smi, and verified the cudaMemcpyAsync implementation to understand how non-pinned memory is handled.

The result, presented in [msg 2350], was a staggering inventory [55]: 23.6 GiB of host-to-device (HtoD) transfers per partition inside the GPU mutex, broken down by phase:

Part III: Two Root Causes of GPU Utilization Dips

Root Cause 1: Non-Pinned Host Memory for a/b/c Polynomials

The a, b, and c polynomial vectors — each approximately 2 GiB for a total of 6 GiB per partition — were being uploaded from host memory that was not page-locked (not "pinned" in CUDA terminology). These vectors come from Rust Vec<Fr> allocations in the Assignment struct, which are allocated on the heap via Rust's allocator. Heap memory is pageable — the operating system can page it out, and more importantly, it is not registered with the CUDA driver for direct memory access (DMA).

When cudaMemcpyAsync is called with a non-pinned source pointer, CUDA cannot directly issue a DMA transfer. Instead, it must stage the data through a small internal pinned buffer (~32 MB) on the CPU side, copying from the pageable source into the bounce buffer, then issuing the DMA transfer from the bounce buffer to the GPU. This staging process has two consequences: it halves the effective bandwidth because the data must be copied twice, and it serializes the transfer — the GPU cannot start processing the data until the entire transfer completes, but the transfer itself is slower than it could be.

The assistant estimated that this non-pinned overhead costs 200-400 milliseconds per partition for the a/b/c transfers alone. Across 10 partitions, that's 2-4 seconds per proof — a 5-10% throughput penalty that was invisible in the macro-level TIMELINE analysis because it was embedded within the CUDA kernel execution time.

Root Cause 2: Per-Batch Hard Sync Stalls in the Pippenger MSM

The second root cause was more subtle but potentially more impactful. The H MSM phase streams 6 GiB of SRS points through 8 Pippenger batches. Each batch follows a pattern: upload batch N of SRS points to GPU (HtoD), launch compute kernel for batch N, synchronize (gpu[i&1].sync()), download bucket results (DtoH), process results on CPU (collect()), and start batch N+1.

This pattern introduces 8 hard synchronization stalls per partition. During each sync, the GPU finishes its current work and then idles while the CPU processes the bucket results and prepares the next batch. The sync is necessary because the Pippenger algorithm accumulates results into buckets that must be collected between batches — the CPU needs the bucket data to prepare the next batch's inputs. But the problem is that these syncs create a stop-start pattern where the GPU alternates between compute and idle.

The assistant's analysis of the Pippenger code in [msg 2352] [57] and [msg 2353] [58] revealed that the existing implementation already had sophisticated double-buffering for point uploads (d_points[d_off] alternates via (i+1)&1), and scalar digit-sorting ran on a separate stream (gpu[2]) while compute ran on flip-flop streams (gpu[0]/gpu[1]). But the hard sync() at line 556 defeated the purpose — it ensured DtoH of bucket results was complete before the CPU collect() in the next iteration, but it left the GPU idle during the handoff.

Part IV: Designing the Two-Tier Mitigation Plan

With the root causes identified, the assistant designed a two-tier mitigation plan, documented in [msg 2355] [60] and formalized in c2-optimization-proposal-9.md.

Tier 1: Pre-upload a/b/c Polynomials Outside the Mutex

The Tier 1 proposal addresses the first root cause: the 6 GiB of non-pinned polynomial data uploaded inside the mutex. The key insight is that the a/b/c polynomial data is available at function entry — it comes from the Rust Assignment struct and doesn't depend on any preprocessing. Currently, it's uploaded inside the NTT as the first thing in the GPU thread.

The proposed change is to allocate device memory for a/b/c and issue cudaMemcpyAsync before acquiring the GPU mutex. This would allow the transfer to overlap with the other worker's CUDA kernels, since it uses a separate CUDA stream and the copy engine is independent of the compute engine on modern NVIDIA GPUs.

However, there's a complication: the a/b/c memory is not pinned. Non-pinned cudaMemcpyAsync still blocks even on a separate stream. The assistant evaluated three options:

Tier 3: Deferred Batch Sync in Pippenger MSM

The Tier 3 proposal tackles the second root cause: the per-batch hard sync stalls in the Pippenger MSM. The assistant's insight was that the existing double-buffering infrastructure provided the foundation for a fix. The points were already double-buffered, and the scalar processing already ran on a separate stream. What was missing was double-buffered host result buffers — two sets of host-side bucket result arrays so that the CPU can process batch N's results from buffer A while batch N+1's results are being written to buffer B.

The proposed fix restructures the batch loop to defer synchronization:

for batch i = 0..7:
  // GPU compute kernels...
  gpu[i&1].DtoH(ones_buf[i&1], ...);   // DtoH to double-buffered host
  gpu[i&1].DtoH(res_buf[i&1], ...);    // DtoH to double-buffered host
  gpu[i&1].record(ev_batch[i&1]);      // record event instead of sync
  
  if (i > 0):
    gpu[(i-1)&1].sync();               // sync PREVIOUS batch
    collect(p, res_buf[(i-1)&1], ones_buf[(i-1)&1]);
    out.add(p);

This transforms the synchronization pattern from synchronous-per-batch to pipeline-deferred. The GPU never stalls waiting for the CPU to process results, because the sync for batch N happens while batch N+1 is already running on the GPU. The DtoH transfers are tiny (~1.5 MiB per batch), so they complete quickly and the sync is effectively free.

The assistant honestly flagged the risk: this is a change to sppark/msm/pippenger.cuh, which is a third-party dependency from Supranational's sppark library. However, the change is localized to the invoke() method and is backward-compatible — same API, same numerical results, just different scheduling.

The Scope Decision

The assistant presented the plan to the user with a question: which tiers to pursue? The user's answer — "1 + 3-pipeline" — represented a sophisticated strategic choice. Tier 1 addressed the low-hanging fruit (6 GiB of non-pinned transfers trivially moved outside the mutex), while Tier 3 addressed the structural inefficiency in the Pippenger algorithm. Tier 2 (pinning tail MSM bases) was deferred as incremental.

Part V: From Analysis to Documentation to Implementation

With the scope confirmed, the assistant moved through a disciplined workflow. In [msg 2356], it created a structured todo list with the spec-writing task and Tier 1 implementation tasks. In [msg 2357], it marked the first task as "in_progress" and began re-reading the source code to understand the exact memory layout.

This re-reading phase was critical. In [msg 2359], the assistant worked through the memory layout implications of pre-staging, calculating sizes: domain_size = 2^26, sizeof(fr_t) = 32, 3 * domain_size * sizeof(fr_t) = 6 GiB. It then caught a subtlety: "Actually wait — there's a subtlety. I need to reconsider. The d_b memory is freed (goes out of scope at line 115) before the final inverse NTT and MSM." This meant the peak memory during the NTT phase was 6 GiB, but during the MSM phase it dropped to approximately 2.3 GiB — an important validation that the pre-allocation strategy would fit within the 16 GiB VRAM budget.

The assistant also compared the Pippenger pattern against execute_batch_addition, which already used double-buffered point uploads with only a single sync at the end. This comparison validated the Tier 3 approach and provided a template for the refactoring.

The user then issued the directive in [msg 2361]: "write down optimization phase 9 md spec and rationale." The assistant responded by writing the 394-line c2-optimization-proposal-9.md file ([msg 2363]) and committing it as 673967f2 ([msg 2364]). The commit message summarized the two changes and their expected 4-9% throughput improvement.

The assistant then summarized the spec in [msg 2366] [71] as five bullet points covering the problem analysis, Tier 1, Tier 3, verification plan, and full reference inventory. The user's response was an empty message ([msg 2367]) — interpreted by the assistant as tacit approval to proceed with implementation.

Part VI: The Knowledge Architecture of the Investigation

What makes this arc particularly instructive is the layered structure of knowledge it reveals. At the top level, the TIMELINE analysis provided aggregate throughput numbers that declared the system "perfectly GPU-bound." This was a macro-level truth — the GPU was the bottleneck device, and the total time matched the sum of its compute kernels. But it was not the whole truth.

The user's observation of GPU utilization dips introduced a micro-level perspective that the aggregate metrics had averaged out. The GPU was technically busy for the right total duration, but it was not continuously busy. There were micro-idle periods caused by data movement patterns that the TIMELINE instrumentation, with its partition-level granularity, could not capture.

The PCIe transfer inventory bridged these two levels. By tracing every HtoD call inside the mutex, the assistant produced a quantitative accounting of exactly what data was moving, at what size, from what memory type, and with what synchronization pattern. This inventory transformed a qualitative observation ("GPU dips") into a concrete engineering problem: 23.6 GiB of transfers per partition, with 6 GiB suffering from half-bandwidth bounce-buffer overhead and 8+ sync points creating stop-start GPU utilization.

The two-tier mitigation plan then translated this understanding into actionable code changes. Tier 1 used cudaHostRegister to pin existing memory — a localized CUDA API call that required no changes to the Rust allocation strategy. Tier 3 restructured the Pippenger batch loop using double-buffered host result buffers — a scheduling change that preserved the existing algorithm while eliminating the sync-induced idle gaps.

Part VII: Assumptions, Risks, and the Path Forward

The plan rests on several assumptions that will need validation during implementation. The cudaHostRegister approach assumes the Rust-allocated Vec<Fr> memory is page-aligned and can be safely pinned for DMA — a reasonable assumption but one that depends on the specific allocator and system configuration. The overlap between copy engine and compute engine assumes the RTX 5070 Ti's hardware supports concurrent operation, which is true for modern NVIDIA architectures but may have practical limitations under PCIe congestion. The deferred sync pattern in Tier 3 assumes that the collect() time for the previous batch is shorter than the compute time for the current batch — a safe assumption given the relative sizes, but one that could fail if the CPU is under heavy load.

The VRAM budget analysis (8-9 GiB peak within 16 GiB) assumes ideal conditions without fragmentation or unexpected allocations from the CUDA driver. The num_circuits == 1 assumption is valid for the current per-partition dispatch architecture but would need revisiting if multi-circuit batching is re-enabled.

The combined expected improvement of 2.5-6 seconds per proof (7-16%) would break through the GPU-bound plateau that had seemed insurmountable. But even if the actual improvement is at the lower end of this range, the investigation has already produced a valuable outcome: a refined understanding of what "GPU-bound" really means. The system is GPU-bound in the sense that the GPU is the bottleneck device — the overall throughput is limited by GPU compute speed. But it is not GPU-efficient — within the GPU's compute phases, there are idle periods caused by PCIe transfer stalls. This distinction between "bottleneck device" and "device efficiency" is crucial for targeting the right optimizations.

Conclusion

The arc from message 2296 to message 2367 captures a complete cycle of performance engineering: measure, diagnose, design, document, and prepare to implement. It begins with a macro-level TIMELINE analysis that declares victory on CPU-side optimization, pivots through a user observation that reveals hidden GPU inefficiency, conducts a forensic PCIe transfer inventory that identifies two root causes, designs a two-tier mitigation plan with quantified impact estimates, and formalizes the plan into a committed design specification.

The investigation demonstrates that even at the frontier of GPU performance — where a system is running at its theoretical maximum by every conventional metric — there is always another layer of optimization for those willing to look at what the GPU is waiting for, not just what it is doing. The 23.6 GiB of HtoD transfers per partition, the 6 GiB of non-pinned polynomial data suffering through CUDA's bounce buffer, and the 8+ hard synchronization stalls per MSM were all invisible in the aggregate throughput numbers. They were visible only to someone watching the live system — the GPU power dips, the PCIe traffic bursts — and asking the right question: "can those moves be moved outside the semaphore?"

The Phase 9 design specification, now committed as 673967f2, captures this understanding and provides a concrete path forward. Tier 1 will pre-stage a/b/c polynomials outside the mutex via cudaHostRegister and async upload on a dedicated copy stream. Tier 3 will restructure the Pippenger batch loop with double-buffered host result buffers to defer synchronizations and eliminate GPU idle gaps between MSM batches. Together, they aim to close the final gap in what was already a 2.4×-optimized pipeline — proving that even at the limits of hardware utilization, there is always another layer of optimization to uncover.## References

[1] "The Architecture of a Summary: How One Message Captured the State of a GPU-Proving Pipeline at a Critical Juncture" — Analysis of message 2296, the comprehensive Phase 8 summary.

[8] "The Grep That Changed Everything: How a Single TIMELINE Extraction Revealed Perfect GPU Boundedness" — Analysis of the TIMELINE extraction that proved GPU-boundedness.

[9] "The Moment the Bottleneck Crystallized: TIMELINE Analysis Reveals Perfect GPU-Boundedness in a SNARK Proving Pipeline" — Deep analysis of the TIMELINE findings.

[10] "The Moment of Clarity: Diagnosing GPU Boundedness Through TIMELINE Analysis" — The diagnostic reasoning behind the GPU-boundedness conclusion.

[38] "The PCIe Traffic Signal: A User's Observation That Reshaped the Optimization Pipeline" — Analysis of message 2333, the user's observation of GPU utilization dips.

[55] "Uncovering the Hidden PCIe Bottleneck: A Deep Dive into GPU Utilization Dips in a Groth16 Proving Pipeline" — Analysis of message 2350, the comprehensive PCIe transfer inventory.

[57] "The Pippenger Batch Sync Pattern: A Deep Dive Into GPU Idle Stalls" — Analysis of message 2352, reading the Pippenger source code.

[58] "The Critical Sync: Discovering the Pippenger MSM Bottleneck in GPU-Accelerated Proof Generation" — Analysis of message 2353, identifying the hard sync as the bottleneck.

[60] "Breaking Through the GPU-Bound Plateau: Phase 9's PCIe Transfer Optimization Plan for the cuzk SNARK Proving Engine" — Analysis of message 2355, the detailed optimization plan.

[71] "The Phase 9 Handoff: When a 394-Line Spec Becomes a 5-Bullet Summary" — Analysis of message 2366, the summary of the committed spec.