The Architecture of a Plan: Designing Phase 9's PCIe Transfer Optimization for Groth16 Proof Generation

Introduction

In the high-stakes world of Filecoin storage proving, every millisecond counts. The SUPRASEAL_C2 pipeline, responsible for generating Groth16 zero-knowledge proofs for Proof-of-Replication (PoRep), operates at the edge of what modern GPU hardware can deliver. After eight phases of optimization—from sequential partition synthesis through dual-worker GPU interlock—the system had reached a plateau: perfectly GPU-bound at 37.4 seconds per proof, with every CPU cycle already overlapped with GPU computation. Breaking through this barrier required looking not at what the GPU was doing, but at what it was waiting for.

Message 2360 in this conversation represents a pivotal moment: the detailed design of Phase 9, a two-tier optimization targeting the PCIe transfer bottlenecks that were causing mysterious GPU utilization dips and power fluctuations. This article examines that message in depth—its reasoning, its assumptions, its decision-making process, and the deep technical knowledge it both requires and produces.

The Context: A System at the GPU-Bound Plateau

To understand why Message 2360 was written, one must first understand where it sits in the broader conversation. The preceding messages (2351–2359) document a rapid-fire investigation into the GPU utilization patterns observed during Phase 8 benchmarking. The user had noticed something troubling: GPU power and utilization were dipping periodically, and these dips correlated with bursts of PCIe traffic around 50 GB/s. Something was causing the GPU to go idle while data shuffled across the PCIe bus.

The assistant had already formulated a high-level plan in Message 2355, proposing two tiers of optimization:

The Architecture of the Plan

Message 2360 is structured as a formal implementation plan, organized into four steps with clear file-level changes, code snippets, and expected impact analysis. Let us examine each component.

Step 1: The Pre-Staged NTT Variant

The plan begins with groth16_ntt_h.cu, proposing a new method execute_ntts_prestaged() that skips the Host-to-Device (HtoD) memory transfer and instead waits on a CUDA event signaling that the upload is already complete. This is paired with execute_ntt_msm_h_prestaged(), which accepts pre-allocated device buffers for all three vectors (a, b, c) instead of allocating d_b internally.

The key insight here is architectural: by separating the allocation and upload from the computation, the plan enables the upload to happen on a different timeline—specifically, before the GPU mutex is acquired. The CUDA event mechanism provides a lightweight synchronization point: the NTT kernel simply waits for the event, which will have fired long before the mutex is acquired (since the upload takes approximately one second and the mutex wait time is dominated by the other worker's GPU kernel execution).

The proposed method signatures are carefully designed:

static void execute_ntts_prestaged(fr_t* d_inout,
                                   size_t lg_domain_size,
                                   stream_t& stream,
                                   cudaEvent_t pre_staged_event);

static void execute_ntt_msm_h_prestaged(
    const gpu_t& gpu,
    gpu_ptr_t<fr_t> d_a,
    fr_t* d_b,
    fr_t* d_c,
    cudaEvent_t ev_a,
    cudaEvent_t ev_b,
    cudaEvent_t ev_c,
    slice_t<affine_t> points_h,
    point_t& result_h);

Note the three separate events—one per polynomial vector. This allows each upload to proceed independently on the copy stream, and the NTT can begin as soon as its specific input is ready, even if the other vectors are still being transferred.

Step 2: Restructuring the Per-GPU Thread

The most complex change is in groth16_cuda.cu, where the per-GPU thread function generate_groth16_proofs_c() must be restructured to perform pre-staging before acquiring the GPU mutex. The plan specifies a precise sequence:

  1. Pin host memory with cudaHostRegister for all three polynomial vectors across all circuits
  2. Compute domain parameters (domain size, lot_of_memory flag)
  3. Pre-allocate device buffers for d_a, d_b, and d_c
  4. Create a dedicated upload stream and CUDA events
  5. Issue async copies for the first circuit's a/b/c, including zero-padding to domain size
  6. Record completion events on the upload stream
  7. Acquire the GPU mutex
  8. Call the pre-staged NTT variant, which waits on the events (already signaled) and proceeds immediately to computation The plan explicitly acknowledges a complication: the num_circuits &gt; 1 case (Phase 3 batching) is not currently used in Phase 7's per-partition dispatch, so the initial implementation can assert num_circuits == 1 and defer the multi-circuit case. This is a pragmatic engineering decision—solve the current bottleneck first, generalize later. The VRAM budget analysis is particularly thorough:
d_a: 2 GiB, d_b: 2 GiB, d_c: 2 GiB = 6 GiB pre-allocated before mutex During NTT+MSM: d_a reused as MSM scalars (2 GiB), MSM buckets (~205 MiB) batch_addition: ~550 MiB per call Total peak: ~8-9 GiB inside 16 GiB = feasible

This analysis accounts for the fact that d_b and d_c share a single 4 GiB allocation when lot_of_memory is true (which it is on a 16 GiB GPU), and that the MSM working memory is allocated separately. The 8-9 GiB peak leaves comfortable headroom in a 16 GiB VRAM budget.

Step 3: Deferred Batch Sync in Pippenger

The Tier 3 change targets the Pippenger MSM implementation in sppark/msm/pippenger.cuh, a third-party dependency from Supranational's sppark library. The plan identifies the root cause of the GPU idle gaps: a hard sync() call at the end of each batch iteration that ensures bucket results are on the host before the CPU's collect() function processes them.

The current pattern is:

for batch i = 0..7:
  // GPU compute kernels...
  if (i > 0) collect(p, res, ones);     // CPU reduces previous batch
  gpu[i&1].DtoH(ones, ...);            // small DtoH
  gpu[i&1].DtoH(res, ...);             // small DtoH
  gpu[i&1].sync();                     // HARD SYNC — GPU idle

The problem is that sync() blocks the CPU until the GPU has finished all pending work and the DtoH transfers are complete. During this time, the GPU sits idle while the CPU processes results and issues the next batch's commands. The gap is small per iteration, but with 8 batches per MSM and multiple MSMs per partition, it accumulates.

The proposed fix is elegant: double-buffer the host result buffers and defer the sync to the next iteration, so it overlaps with the next batch's GPU computation:

for batch i = 0..7:
  // GPU compute kernels...
  gpu[i&1].DtoH(ones_buf[i&1], ...);   // DtoH to double-buffered host
  gpu[i&1].DtoH(res_buf[i&1], ...);    // DtoH to double-buffered host
  gpu[i&1].record(ev_batch[i&1]);      // record event instead of sync
  
  if (i > 0):
    gpu[(i-1)&1].sync();               // sync PREVIOUS batch
    collect(p, res_buf[(i-1)&1], ones_buf[(i-1)&1]);
    out.add(p);

This transforms the synchronization pattern from synchronous-per-batch to pipeline-deferred. The GPU never stalls waiting for the CPU to process results, because the sync for batch N happens while batch N+1 is already running on the GPU. The DtoH transfers are tiny (~1.5 MiB per batch), so they complete quickly and the sync is effectively free.

The plan acknowledges the risk: this modifies a third-party dependency (sppark). However, the change is localized to the invoke() method and is backward compatible—same API, same numerical results, just different scheduling.

Step 4: Build and Test

The final step is straightforward: rebuild the cuzk-daemon crate and benchmark. The plan specifies the exact build command (rm -rf extern/cuzk/target/release/build/supraseal-c2-* to force a clean rebuild of the C++ CUDA code) and the benchmarking approach (compare CUZK_TIMING output).

Expected Impact and the Plateau Breakthrough

The plan concludes with quantitative estimates:

The Thinking Process: Iterative Refinement Under Constraint

What makes Message 2360 particularly interesting is the thinking process visible in its evolution. The plan is not the first draft—it is the product of several iterations visible in the preceding messages.

In Message 2355, the assistant proposed a high-level plan. In Messages 2357-2359, it read the actual code and discovered complications. The lot_of_memory logic meant that d_b and d_c sometimes share a single allocation, complicating pre-allocation. The execute_batch_addition already used double-buffering for point uploads, suggesting the pippenger pattern could be similarly optimized. The gpu_t class had three flip-flop streams plus a zero stream, providing a model for how to add a dedicated copy stream.

In Message 2359, the assistant briefly considered a simpler approach: just pin the host memory with cudaHostRegister and keep the upload inside the mutex. This would give perhaps half the benefit with much less code change. But then it rejected this approach in favor of the full pre-upload strategy:

But the real win would be to pre-upload AND overlap with the other worker's compute. Let me think about this more carefully...

This moment of reconsideration is crucial. The assistant recognized that the simpler approach would still leave the upload inside the mutex, serialized with GPU compute. The full pre-upload approach, while more complex, allows the upload to overlap with the other worker's GPU kernel execution—effectively hiding the transfer latency behind computation that was already happening.

The constraint of plan mode (no edits allowed) forced the assistant to document this refined plan in exhaustive detail. The result is a message that serves both as a specification for implementation and as a design document that captures the reasoning behind each decision.

Assumptions and Their Validity

Every engineering plan rests on assumptions, and Message 2360 is explicit about several:

  1. num_circuits == 1: The plan assumes Phase 7's per-partition dispatch, which processes one circuit per call. This is valid for the current architecture but would need revisiting if multi-circuit batching is re-enabled.
  2. 16 GiB VRAM: The VRAM budget analysis assumes a 16 GiB GPU. The lot_of_memory computation is dynamic, but the pre-allocation strategy assumes sufficient headroom. On smaller GPUs, the plan might need adjustment.
  3. cudaHostRegister cost is ~1-5ms for 2 GiB: This is a reasonable estimate based on CUDA documentation and common experience. The pinning operation itself is lightweight—it's the subsequent DMA that benefits.
  4. The sync() is the main bottleneck: This assumption is supported by the observed GPU power dips correlating with PCIe traffic. The deferred sync pattern directly addresses this correlation.
  5. DtoH is tiny (~1.5 MiB): This is correct for the bucket results (a few hundred KiB per batch). The deferred sync should complete well before the next batch needs the data. One potential issue not fully addressed: the interaction between Tier 1 and Tier 3. If Tier 1 already eliminates the PCIe traffic from the mutex, does Tier 3 still provide benefit? The answer is yes, because Tier 1 addresses the a/b/c polynomial uploads (6 GiB), while Tier 3 addresses the per-batch sync stalls in the MSM (a different bottleneck). They are complementary.

Input Knowledge Required

To fully understand Message 2360, the reader needs:

Output Knowledge Created

Message 2360 produces several forms of knowledge:

Conclusion

Message 2360 is far more than a plan waiting for approval. It is the crystallization of a deep investigation into GPU performance behavior, the product of iterative refinement through code reading and constraint discovery, and a detailed specification for the next phase of optimization in a high-performance proving pipeline.

The message demonstrates what makes great systems engineering: the ability to move from a high-level observation (GPU power dips correlate with PCIe traffic) to a precise, implementable solution (pre-stage a/b/c with cudaHostRegister and async copies; defer MSM batch syncs with double-buffered host buffers). It navigates the tension between simplicity and effectiveness, choosing the more complex pre-upload approach because it unlocks overlap with the other worker's computation. It accounts for VRAM budgets, third-party dependency risks, and edge cases like multi-circuit batching.

Most importantly, it represents the moment when a system that had reached a GPU-bound plateau—where conventional wisdom says "you can't go faster without better hardware"—was shown to have hidden inefficiencies waiting to be eliminated. The 7-16% improvement it promises may not sound dramatic, but in the context of a system already running at full GPU utilization, breaking through that plateau is a significant achievement. It is a reminder that even at the limits of hardware utilization, there is always another layer of optimization for those willing to look at what the GPU is waiting for, not just what it is doing.