The First Cut: Implementing Phase 9's PCIe Transfer Optimization in the cuzk SNARK Proving Engine

Introduction

In the high-stakes world of Filecoin storage proving, every millisecond of GPU idle time translates directly into lost revenue. The cuzk SNARK proving engine, a custom-built Groth16 prover for Filecoin's Proof-of-Replication (PoRep), had already been through eight optimization phases by the time the assistant reached message [msg 2374]. Phase 8 had achieved a remarkable 37.4 seconds per proof with perfect GPU utilization at the scheduling level — the GPU was never idle between partitions. But within each partition's CUDA kernel region, periodic dips in GPU power draw and SM utilization told a different story: the GPU was stalling, waiting for data to arrive over the PCIe bus.

Phase 9, documented in the 395-line c2-optimization-proposal-9.md design spec, targeted two root causes of these stalls. The first was the 6 GiB of non-pinned host-to-device (HtoD) transfers for the a, b, and c polynomials that occurred inside the GPU mutex, serialized through a 32 MB bounce buffer at half bandwidth. The second was the per-batch hard sync stalls in the Pippenger MSM, where the GPU fully drained after each of eight batches to let the CPU collect bucket results. Together, these stalls consumed an estimated 5–10% of the ~3.75 seconds of CUDA time per partition — a savings opportunity of 2–4 seconds per proof.

Message [msg 2374] is the moment where theory becomes practice. It is the first implementation step of Phase 9, the instant the assistant's cursor touches the code. This article examines that single message in depth: why it was written, what decisions it embodies, what assumptions it rests on, and how it fits into the larger arc of the optimization journey.

The Message

The message itself is brief and to the point:

[assistant] Starting with Change 1 in groth16_ntt_h.cu. I need to add: 1. execute_ntts_prestaged() — like execute_ntts_single() but skips HtoD, waits on a CUDA event instead 2. execute_ntt_msm_h_prestaged() — like execute_ntt_msm_h() but takes pre-allocated device pointers and events [edit] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_ntt_h.cu Edit applied successfully.

Two sentences of intent, a bullet list of two function signatures, and a confirmation that the edit succeeded. The actual content of the edit — the C++ code added to the file — is not shown in the message, but the assistant's reasoning is laid bare in the preceding context. The message is a bridge between planning and execution, between the abstract design in the spec and the concrete code that will run on the GPU.

Why This Message Was Written

The message exists because of a chain of reasoning that began with the Phase 8 benchmark results. The assistant had measured a system throughput of 37.4 seconds per proof with partition_workers=10, gpu_workers_per_device=2, and concurrency=3. Detailed timing logs showed that the GPU kernel time per partition was ~3,746 ms, of which the NTT+H MSM phase consumed ~2,430 ms. The system was perfectly GPU-bound at the scheduling level — but within each kernel region, GPU power draw dips correlated with 50 GB/s PCIe RX bursts.

The assistant had performed a root cause analysis that produced a complete PCIe transfer inventory: 23.6 GiB of HtoD transfers per partition, of which 6 GiB (the a/b/c polynomials) were from non-pinned host memory. Non-pinned transfers force CUDA to stage data through a small internal bounce buffer, serializing the transfer and halving effective bandwidth. A further 8 hard syncs in the Pippenger MSM created additional GPU idle gaps of 1–5 ms each, totaling 50–200 ms per partition.

The Phase 9 design spec proposed two changes. Change 1 (Tier 1) would move the a/b/c polynomial uploads outside the GPU mutex by pinning host memory with cudaHostRegister, allocating device buffers upfront, and issuing async cudaMemcpyAsync transfers on a dedicated stream with CUDA event-based synchronization. Change 2 (Tier 3) would eliminate the per-batch hard sync stalls by introducing double-buffered host result buffers and deferring the sync() call to the next iteration.

Message [msg 2374] is the first implementation step of Change 1. The assistant chose to start with groth16_ntt_h.cu because it is the leaf module in the call chain — the functions that actually perform the NTT computations. The strategy is bottom-up: add the new _prestaged variants at the lowest level first, then modify the caller in groth16_cuda.cu to use them. This is a classic software engineering pattern: build the foundation before the superstructure.

How Decisions Were Made

Several design decisions are visible in this message, even though the message itself is short.

Naming convention: The new functions are named execute_ntts_prestaged() and execute_ntt_msm_h_prestaged(), following the existing pattern of execute_ntts_single() and execute_ntt_msm_h(). The _prestaged suffix signals that the input data has already been uploaded to the device — the function skips the HtoD step and instead waits on a CUDA event that signals when the upload is complete. This naming makes the code self-documenting: any developer reading the call site will understand that pre-staging is involved.

Synchronization mechanism: The assistant chose CUDA events (cudaEventRecord/cudaEventSynchronize or cudaStreamWaitEvent) as the synchronization primitive. This is the standard CUDA mechanism for coordinating work across streams, and it integrates naturally with the existing stream-based architecture in groth16_ntt_h.cu. Events allow the upload stream and the compute stream to operate independently: the upload stream records an event after the HtoD completes, and the compute stream waits on that event before starting the NTT kernels. This avoids the need for explicit host-side synchronization (like cudaStreamSynchronize) which would block the CPU thread.

Bottom-up implementation order: The assistant started with the leaf file (groth16_ntt_h.cu) rather than the caller (groth16_cuda.cu). This is a deliberate choice that minimizes risk: if the new function signatures need adjustment, the changes are localized to one file. The caller can be written later with confidence that the interface is stable. The todo list in [msg 2372] shows the full plan: first add the prestaged variants to groth16_ntt_h.cu, then modify groth16_cuda.cu to pre-stage the buffers and call the new variants, then add cleanup code, then build and test.

Preserving backward compatibility: The design spec explicitly states that the original execute_ntt_msm_h() is preserved. If the pre-staging setup fails (e.g., cudaHostRegister returns an error on some systems), the code can fall back to the original path with the upload inside the mutex. This decision is not visible in the message itself, but it informs the design: the new functions are additions, not replacements.

Assumptions Made

Every message rests on assumptions, and [msg 2374] is no exception.

The assistant assumes that the edit tool applied the changes correctly. The message reports "Edit applied successfully," but the assistant has not yet verified the result by reading the file back. This is a trust-in-the-tool assumption that is reasonable given the tool's track record, but it means the assistant is proceeding on faith until the next verification step.

The assistant assumes that the new functions will have the right interface to be called from groth16_cuda.cu. The design spec specifies the semantics — skip HtoD, wait on event, take pre-allocated device pointers — but the exact parameter types and order are not specified in the message. The assistant must infer the correct interface from the existing functions and the design intent.

The assistant assumes that CUDA event-based synchronization will work correctly in the pre-staging architecture. The uploads happen on a dedicated stream outside the mutex, and the compute kernels run on the GPU's flip-flop streams inside the mutex. The events must be recorded on the upload stream and waited on by the compute streams. This is a well-documented CUDA pattern, but it introduces new failure modes: if the event is never recorded (e.g., because the upload kernel failed), the compute stream will wait indefinitely.

The assistant assumes that the VRAM budget is sufficient. The design spec calculates a peak of ~7.6 GiB for the pre-staged buffers plus MSM working memory, which fits comfortably in the 16 GiB RTX 5070 Ti. But this calculation assumes that cudaMalloc succeeds, that the memory is not fragmented, and that no other allocations interfere. As later messages in the session reveal, these assumptions proved optimistic — the dual-worker configuration triggered OOM failures because both workers tried to pre-stage simultaneously, allocating 12 GiB each and exceeding the 16 GiB VRAM.

Input Knowledge Required

To understand this message, a reader needs substantial context from the preceding conversation.

First, the reader must understand the Phase 8 architecture: the GPU mutex that serializes CUDA kernel access across workers, the per-partition dispatch model, and the timing breakdown showing 3,746 ms of GPU time per partition. Without this context, the significance of "moving uploads outside the mutex" is lost.

Second, the reader must understand the PCIe transfer inventory: the 23.6 GiB of HtoD transfers per partition, the distinction between pinned and non-pinned memory, and the bandwidth implications of each. The design spec at [msg 2369] provides this inventory in exhaustive detail, with a table showing each transfer's size, pinning status, and the problem it creates.

Third, the reader must understand the CUDA programming model: streams, events, cudaMemcpyAsync, cudaHostRegister, and the difference between synchronous and asynchronous transfers. The message uses CUDA terminology ("HtoD", "CUDA event") that assumes familiarity with the GPU programming paradigm.

Fourth, the reader must understand the file structure of the supraseal-c2 codebase: that groth16_ntt_h.cu contains the NTT and H MSM functions, that groth16_cuda.cu contains the main generate_groth16_proofs_c function, and that pippenger.cuh contains the Pippenger MSM implementation. The assistant had read all three files in messages [msg 2370] and [msg 2371] before writing this message.

Output Knowledge Created

This message creates two new CUDA function variants that are the foundation of the Phase 9 pre-staging architecture.

execute_ntts_prestaged() is a variant of execute_ntts_single() that skips the HtoD transfer of polynomial data. Instead of copying data from host to device, it waits on a CUDA event that signals when the data has been uploaded by a separate stream. This allows the NTT computation to begin without the calling thread ever touching the PCIe bus for the polynomial data.

execute_ntt_msm_h_prestaged() is a variant of execute_ntt_msm_h() that takes pre-allocated device pointers (d_a, d_b, d_c) and upload events. The original function allocates device memory internally via dev_ptr_t and performs the HtoD transfers. The prestaged variant skips both steps, assuming the caller has already allocated the buffers and issued the uploads. The function waits on the events, runs the NTTs, then proceeds to the H MSM using the pre-uploaded data.

These functions are the leaf-level building blocks of Change 1. They do not themselves implement the pre-staging logic — that happens in groth16_cuda.cu, which will call cudaHostRegister, allocate device buffers, issue async uploads, and then pass the resulting events and pointers to these new functions. But without these variants, the pre-staging architecture cannot exist. They are the interface between the pre-staging orchestration and the computational kernels.

The Thinking Process

The assistant's thinking process is visible in the sequence of messages leading up to [msg 2374]. In [msg 2368], the assistant lays out the complete Phase 9 plan, including the todo list with Change 1 and Change 2. In [msg 2369], the user says "proceed with implementation" and the assistant reads the design spec. In [msg 2370] and [msg 2371], the assistant reads all three files that need modification. In [msg 2372], the assistant sets up the task plan with a detailed todo list. In [msg 2373], the assistant marks the first todo item as "in_progress."

Then comes [msg 2374]: the first edit.

The thinking is methodical and deliberate. The assistant does not jump into coding immediately. It first reads the design spec, then reads all the files, then creates a structured plan, then begins implementation. The bottom-up approach — leaf file first, then caller — minimizes risk and ensures that each step builds on a solid foundation.

The message also reveals the assistant's mental model of the codebase. The assistant knows that execute_ntts_single() performs both HtoD and computation, and that the new variant needs to split these concerns. The assistant knows that execute_ntt_msm_h() allocates device memory internally, and that the new variant needs to accept pre-allocated pointers. This understanding comes from reading the files in the preceding messages.

Conclusion

Message [msg 2374] is a small step in the implementation of a complex optimization, but it is a critical one. It marks the transition from design to code, from theory to practice. The assistant's methodical approach — read, plan, execute — ensures that this transition is smooth and that the foundation is solid before the superstructure is built.

The message also embodies a key insight of the Phase 9 design: that the GPU's copy engine and compute engine are independent resources that can be overlapped. By moving the polynomial uploads outside the mutex and onto a dedicated stream, the assistant is not just optimizing a single transfer — it is rearchitecting the entire proving pipeline to treat PCIe bandwidth as a resource to be managed, not a cost to be paid.

As later messages in the session reveal, this first edit was followed by OOM failures, allocator fixes, and ultimately a 14.2% throughput improvement in single-worker mode. But it all started here, with two new function variants in groth16_ntt_h.cu and the quiet confidence of an edit applied successfully.