Committing the Blueprint: Phase 9 PCIe Transfer Optimization for Groth16 Proof Generation

In the long arc of optimizing the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep), most milestones are marked by code — a refactored kernel, a new mutex scheme, a benchmark result. But some of the most important milestones are marked by documents. Message [msg 2364] captures one such moment: the commit of c2-optimization-proposal-9.md, a design specification that crystallizes weeks of deep PCIe transfer analysis into a concrete two-tier mitigation plan. On its surface, the message is a simple git operation — add a file, commit it with a message. But the commit message itself tells a story of careful diagnosis, precise measurement, and architectural thinking that reveals how even a perfectly GPU-bound system can still be hiding significant PCIe-related idle time.

The Context: From Perfect GPU-Boundedness to Hidden Idle

To understand why this commit matters, one must understand what preceded it. The optimization journey had progressed through eight phases. Phase 7 introduced per-partition dispatch architecture for the cuzk SNARK proving engine. Phase 8 implemented a dual-worker GPU interlock that narrowed a C++ static mutex, allowing multiple GPU workers per device and achieving a 13–17% throughput improvement. A systematic sweep of the partition_workers parameter identified the optimal setting at 10–12 workers.

Then came the TIMELINE analysis. In the chunk preceding this message, the assistant performed a deep TIMELINE analysis of the Phase 8 benchmark at partition_workers=10. The result was striking: the system was perfectly GPU-bound. The measured 37.4 seconds per proof exactly matched the serial CUDA kernel time of 10 partitions × 3.75 seconds. Cross-sector GPU transitions after warmup were under 50 milliseconds. Synthesis was fully overlapped with GPU work. By every conventional metric, the system was optimal — there was nothing left to optimize on the CPU side.

But the user observed something curious. GPU utilization and power charts showed dips that correlated with approximately 50 GB/s of PCIe traffic. If the GPU was fully occupied with compute kernels, why would power dip during PCIe transfers? The answer, it turned out, was that the GPU was not fully occupied during those transfers — at least, not in the way that matters. The PCIe traffic was triggering brief moments of GPU idleness that accumulated into a measurable throughput gap.

The Diagnosis: Two Root Causes in 23.6 GiB of Transfers

The assistant conducted a meticulous inventory of every Host-to-Device (HtoD) transfer that occurs inside the GPU mutex for a single partition. The total was 23.6 GiB per partition. Within that massive data movement, two root causes of GPU idle time were identified.

Root Cause 1: Non-pinned host memory for a/b/c polynomials. The a, b, and c polynomials — each approximately 2 GiB for a total of 6 GiB — were being uploaded from non-pinned (pageable) host memory. CUDA cannot perform true asynchronous DMA from non-pinned memory. Instead, it must stage the data through a small bounce buffer on the host side, effectively serializing the transfer and achieving only about half of the available PCIe Gen4 bandwidth. This meant the GPU spent longer waiting for data to arrive than necessary, and the CPU thread performing the upload was blocked during the staging process.

Root Cause 2: Per-batch hard sync stalls in the Pippenger MSM. The multi-scalar multiplication (MSM) phase uses Pippenger's algorithm, which processes the computation in batches. In the existing implementation, each batch ended with a hard synchronization point: the GPU would finish computing bucket results, transfer them back to the host via cudaMemcpyDeviceToHost, and then synchronize the stream before the CPU could process the results. During this synchronization, the GPU sat idle — it had completed its work for the current batch but could not begin the next batch's upload or computation because the CPU was still processing the previous batch's results. With 8 or more batches per MSM, these sync points accumulated into a significant idle gap.

The Two-Tier Mitigation Plan

The Phase 9 design specification, now committed as 673967f2, addresses both root causes with a two-tier approach.

Tier 1: Pre-stage a/b/c polynomials outside the mutex. The key insight is that the a/b/c upload does not actually need to happen inside the GPU mutex. The mutex exists to serialize access to GPU compute resources (NTT kernels, MSM kernels), but the copy engine on modern NVIDIA GPUs is an independent hardware unit that can operate concurrently with compute kernels on other streams. The plan calls for:

Calling cudaHostRegister on the host-side a/b/c buffers before entering the mutex, pinning them for true DMA.
Pre-allocating device-side buffers (d_a, d_b, d_c) before the mutex.
Launching cudaMemcpyAsync operations on a dedicated copy stream before acquiring the mutex.
Using CUDA events to signal completion, so the NTT kernels inside the mutex can simply wait on the event rather than performing the upload themselves. This approach overlaps the 6 GiB of a/b/c upload with the other GPU worker's compute kernels. Since the copy engine is independent, the upload consumes no compute cycles on the GPU. The cudaHostRegister call ensures the upload runs at full PCIe Gen4 bandwidth (~25 GB/s effective) rather than being throttled by the bounce buffer. Tier 3: Deferred batch sync in Pippenger MSM. The commit message describes this as "double-buffer host-side bucket results so GPU never waits for CPU to process the previous batch." The implementation restructures the Pippenger batch loop to use two sets of host-side result buffers. Instead of synchronizing after every batch, the GPU launches the DtoH transfer for the current batch's results into one buffer while immediately beginning the next batch's computation. The synchronization is deferred: the CPU processes buffer (i-1)&1 while the GPU fills buffer i&1. This eliminates the 8+ per-batch idle gaps, keeping the GPU continuously fed with work.

The Commit Message as Documentation Artifact

The commit message in [msg 2364] is worth examining as a piece of technical communication. It opens with a one-line summary: "docs: Phase 9 design spec — PCIe transfer optimization." This follows conventional git conventions (the docs: prefix indicating a documentation change) while immediately communicating the scope.

The body then provides a structured explanation:

Two changes to reduce GPU SM idle time caused by PCIe transfers inside the GPU mutex:

1. Pre-stage a/b/c polynomials (6 GiB) outside the mutex via cudaHostRegister + async upload on a dedicated copy stream. Overlaps with the other worker's CUDA kernels.

2. Deferred batch sync in Pippenger MSM: double-buffer host-side bucket results so GPU never waits for CPU to process the previous batch. Eliminates 8+ per-batch idle gaps per MSM.

Includes full PCIe transfer inventory (23.6 GiB HtoD per partition) and expected 4-9% throughput improvement over Phase 8.

This is not merely a log entry — it is a miniature design document that captures the rationale, the mechanism, and the expected impact. The "23.6 GiB HtoD per partition" figure grounds the discussion in concrete measurement. The "4-9% throughput improvement" sets an expectation that can be validated or falsified by future benchmarking. The commit hash 673967f2 on the feat/cuzk branch creates a permanent, citable reference point.

The LSP Error: A Note on Development Environment Awareness

The message opens with a seemingly minor observation: "That LSP error is pre-existing (sppark header resolution, not our file)." This brief acknowledgment reveals something important about the assistant's operating context. The Language Server Protocol (LSP) error — a missing util/thread_pool_t.hpp header in the sppark dependency — could have been a distraction. It could have prompted a debugging detour or raised concerns about the new file's correctness. Instead, the assistant correctly identifies it as a pre-existing issue unrelated to the Phase 9 spec, dismisses it, and proceeds with the commit.

This kind of environmental awareness is critical in large-scale optimization work. The sppark library (Supranational's parallelized polynomial arithmetic kernels) is a dependency that lives outside the main codebase being modified. Its header resolution issues are known and unrelated to the cuzk proving engine. Recognizing what is and is not a signal worth acting on — versus noise to be ignored — is a skill that separates effective engineering from endless debugging.

What the Commit Represents

The creation of c2-optimization-proposal-9.md with 394 lines of content represents a significant analytical achievement. The document did not emerge from thin air — it was the product of:

The TIMELINE analysis that established perfect GPU-boundedness at the macro level.
The user's observation of GPU power dips correlating with PCIe traffic, which prompted deeper investigation.
The meticulous inventory of all 23.6 GiB of HtoD transfers per partition.
The identification of two specific mechanisms (non-pinned memory and Pippenger sync stalls) that explained the micro-level idle gaps.
The design of two targeted mitigations that preserve the existing architecture while eliminating the identified bottlenecks. The commit also marks a transition point. With the design specification committed, the next phase of work — implementation — can begin. The assistant's todo list, visible in preceding messages, already contains items for Tier 1 implementation: cudaHostRegister + pre-alloc device buffers, refactoring execute_ntt_msm_h to accept pre-uploaded buffers, adding cudaHostUnregister in the epilogue, and building and smoke testing.

The Broader Pattern: Document-Driven Optimization

There is a methodological pattern visible in this work that deserves attention. The optimization pipeline does not proceed directly from observation to code change. Instead, it follows a deliberate sequence:

Measure: Run benchmarks, collect TIMELINE data, observe GPU utilization and power.
Diagnose: Identify root causes through detailed analysis and transfer inventory.
Design: Write a specification document that enumerates the changes, their rationale, and their expected impact.
Commit the design: Create a permanent record that can be referenced, reviewed, and revisited.
Implement: Execute the plan described in the specification.
Validate: Benchmark the implementation against the expected improvement. This pattern — document first, implement second — is especially valuable in performance optimization, where the temptation to dive directly into code changes can lead to fragmented, poorly-understood modifications. By committing the design specification before writing any implementation code, the assistant creates an audit trail. Future engineers (or the same engineer returning after a hiatus) can read c2-optimization-proposal-9.md and understand not just what was changed but why it was changed and what problem it was solving.

Conclusion

Message [msg 2364] is, on its face, a simple git commit. But it represents the culmination of a deep diagnostic journey that revealed hidden GPU idle time in a system that appeared perfectly optimized. The Phase 9 design specification, now permanently recorded at commit 673967f2, captures two targeted mitigations — pre-staged polynomial uploads and deferred Pippenger batch syncs — that together are expected to yield 4–9% throughput improvement over Phase 8. More importantly, the commit exemplifies a disciplined approach to performance engineering: measure thoroughly, diagnose precisely, document completely, and only then implement. In the high-stakes world of Filecoin proof generation, where every second of proving time translates directly into operational cost, this discipline is what separates ad-hoc optimization from systematic engineering.