Zero-Copy GPU Pipelines: How Reactive Backpressure Eliminated H2D Transfer Overhead in CuZK

Introduction

In high-performance GPU proving pipelines, the difference between a system that works and one that sings often comes down to a single architectural choice: how do you regulate the flow of work from CPU to GPU? In message [msg 3338], an assistant presented the results of a dramatic optimization to the CuZK zero-knowledge proving engine's pinned memory pool and dispatch system. The numbers tell a story of a system transformed: H2D (host-to-device) transfer time dropped from a staggering 1,300–12,000 milliseconds to effectively zero, total per-partition GPU time collapsed from up to 12.7 seconds to a consistent ~935 milliseconds, and pinned memory allocations plummeted from 474 to just 24 while reuse skyrocketed from 12 to 48. The message, a triumphant summary after deploying the "pinned4" build, represents the culmination of a multi-day debugging odyssey that uncovered and fixed a thundering herd problem in the GPU dispatch logic, a buffer lifecycle bug in the pinned memory pool, and a fundamental architectural flaw in how synthesis work was gated.

This article examines that single message in depth: what led to it, what decisions were made along the way, what assumptions proved wrong, and what knowledge was created by the results it reports.

The Problem: A Pipeline That Could Not Stay Fed

The context for [msg 3338] begins with a GPU utilization crisis. The CuZK proving engine had been instrumented with a pinned memory pool — a technique that allocates host-side memory pinned (page-locked) for fast DMA transfers to the GPU. The theory was sound: by reusing pinned buffers instead of allocating fresh ones for each partition, the system would avoid the expensive cudaHostAlloc calls that serialize through the CUDA driver and stall GPU activity. In practice, however, the pinned pool was thrashing: 474 allocations produced only 12 reuses (see [msg 3313]), meaning buffers were being allocated, used once, and freed — never recycled. Meanwhile, GPU utilization was erratic, with long idle periods followed by bursts of activity.

The root cause, identified through a combination of log analysis and reasoning in [msg 3311], was a dispatch burst problem. The system used a poll-based throttle: a dispatcher loop checked every 250 milliseconds whether the GPU work queue depth had fallen below a threshold (max_gpu_queue_depth = 8). When it did, all waiting synthesis jobs were dispatched at once — often 20 or more simultaneously. This thundering herd of synthesis workers each called cudaHostAlloc concurrently, serializing through the CUDA driver and blocking the GPU. The pinned pool, designed for reuse, was instead being exhausted by burst allocations: by the time any buffer was checked back in, all synthesis workers had already allocated fresh ones. The pool had 447 checkins but only 12 reuses — the buffers were being returned, but nobody was waiting for them.

The Fix: From Polling to Reactive Backpressure

The key insight, articulated in [msg 3311], was that the dispatch mechanism needed to be reactive rather than poll-based. Instead of checking "is the GPU queue below 8?", the system should respond to "GPU consumed one slot → start one new synthesis." This creates natural 1:1 modulation: the dispatch rate exactly matches the GPU consumption rate, preventing both starvation and burst overload.

The implementation, traced through messages [msg 3314] through [msg 3329], replaced the poll-based throttle with a tokio Semaphore initialized to max_gpu_queue_depth (8). The dispatcher acquires a permit before dispatching synthesis work, then intentionally forgets the permit so it remains consumed. The GPU finalizer, after processing a partition result and dropping its budget reservation, releases a permit back to the semaphore via add_permits(1). This decouples the permit lifecycle from Rust's drop semantics and ties it directly to GPU completion. Error paths also release permits, ensuring the system cannot deadlock.

The critical design decision here was the decoupling of permit acquisition from permit release. In a naive implementation, the dispatcher would acquire a permit and hold an Acquire guard that gets dropped when synthesis completes. But synthesis completion is not GPU completion — there's a whole pipeline of GPU work after synthesis finishes. By having the GPU finalizer explicitly release permits, the system ensures that a new synthesis is only dispatched when the GPU has actually finished processing the previous partition. This prevents the post-synthesis pipeline from overflowing.

The Results: A System Transformed

The message [msg 3338] presents a before-and-after comparison that is remarkable in its clarity:

| Metric | Before (pinned3) | After (pinned4) | |---|---|---| | ntt_kernels H2D transfer | 1,300–12,000 ms | 0 ms | | Total GPU time per partition | 2,000–12,700 ms | 934–994 ms | | Pinned pool allocations | 474 | 24 | | Pinned pool reuses | 12 | 48 | | Budget used for PCE | 385 GiB | 221 GiB | | Budget available | 5 GiB | 288 GiB |

The most striking number is ntt_kernels=0ms. The H2D transfer, which previously dominated the GPU time budget (taking up to 12 seconds), is now so fast it rounds to zero in millisecond-precision logging. This is the direct consequence of two compounding improvements: (1) the pinned pool is actually reusing buffers, so cudaHostAlloc is rarely called, and (2) with only ~8 partitions in flight instead of 20+, there is no memory contention on the pinned allocations that do occur. The NTT-H path (which does H2D inline) completes nearly instantly when the host buffer is already pinned and the GPU is not swamped.

The total per-partition GPU time of ~935 ms (304 ms for coset_intt_sync + 630 ms for msm_invoke) represents pure GPU compute — the transfer overhead has been eliminated entirely. This is the theoretical best case: the GPU is never waiting for data.

The pinned pool metrics tell the same story from the memory side. With 48 reuses against only 24 allocations, the pool is achieving a 2:1 reuse ratio — every buffer is used twice on average, and the 24 allocations likely represent the initial fill of the pool plus a few growth events. The previous ratio of 12 reuses against 474 allocations meant buffers were effectively single-use. The reactive dispatch fixed this because synthesis now starts only when the GPU finishes, so the returned buffer is immediately available for the next partition.

Budget headroom improved dramatically: from 5 GiB available to 288 GiB. This is because the burst dispatch pattern had all 20+ syntheses holding budget reservations simultaneously, consuming ~362 GiB. With reactive dispatch, only ~8 partitions are in flight at any time, freeing budget for PCE caching and other uses. The PCE cache itself is now using only 221 GiB instead of 385 GiB, suggesting that the reduced memory pressure allowed more efficient caching.

Assumptions, Mistakes, and Lessons Learned

The journey to [msg 3338] involved several incorrect assumptions that were corrected along the way:

Assumption 1: Budget integration in the pinned pool was correct. The original PinnedAbcBuffers::checkout() called budget.try_acquire() for memory that was already accounted for in per-partition budget reservations. This caused silent fallback to heap allocations because with 5 jobs × 16 partitions consuming ~362 GiB, the pinned allocations were denied. The fix (pinned2) removed budget from the pool entirely, allowing pinned allocations to proceed.

Assumption 2: A poll-based throttle was sufficient. The initial GPU queue depth throttle (pinned3) checked every 250ms whether the queue was below max depth. This seemed reasonable but created the burst dispatch problem because when the queue finally dropped below threshold, all waiting work was released at once. The assumption that periodic checking would smooth out dispatch was wrong — it created a sawtooth pattern of idle accumulation followed by burst release.

Assumption 3: The pinned pool thrashing was a checkin bug. When the assistant first saw 474 allocations and only 12 reuses ([msg 3313]), the natural hypothesis was that buffers were not being returned to the pool. But 447 checkins proved they were being returned — they just weren't being reused because the burst dispatch meant no synthesis worker was waiting for a returned buffer. This distinction between "checkin works" and "reuse works" is subtle but critical.

Assumption 4: cudaHostAlloc serialization was the primary bottleneck. While cudaHostAlloc serialization was indeed a problem, the deeper issue was the pattern of allocations. Even if cudaHostAlloc were perfectly parallel, 20+ simultaneous allocations would still consume enormous memory and cause cache thrashing. The fix addressed the root cause (burst dispatch) rather than the symptom (allocation serialization).

Knowledge Created

The message [msg 3338] creates several important pieces of knowledge:

1. Reactive backpressure is superior to polling for GPU dispatch. The semaphore-based approach, where GPU completion directly gates new synthesis, provides smooth modulation without tuning parameters. There is no polling interval to configure, no threshold to set, and no burst behavior. The system self-modulates to match GPU consumption rate exactly.

2. Buffer reuse requires temporal locality. A pool can have plenty of free buffers, but if no consumer is waiting when a buffer is returned, reuse doesn't happen. The dispatch pattern must create a pipeline where buffer return and buffer checkout are closely coupled in time. Reactive dispatch achieves this by stalling synthesis until GPU completion, ensuring the returned buffer is immediately available.

3. H2D transfer overhead is not inherent — it's a symptom of system design. The fact that ntt_kernels dropped to 0ms proves that when the pipeline is properly tuned, H2D transfers can be effectively free. The bottleneck was never the PCIe bandwidth or the CUDA memcpy speed — it was the allocation contention and dispatch burst that caused transfers to wait on memory management.

4. Budget pressure compounds across subsystems. The same budget exhaustion that prevented pinned allocations also prevented PCE caching, creating a vicious cycle: no PCE → slow synthesis → more budget held longer → less budget for PCE. Fixing the dispatch pattern broke this cycle, freeing budget for both pinned allocations and PCE caching simultaneously.

Conclusion

Message [msg 3338] is a moment of clarity after a complex debugging journey. It validates the pinned memory pool concept that was introduced in earlier segments ([chunk 22.0], [chunk 23.0]) and shows that the architecture is sound when the dispatch pattern is correct. The key insight — that reactive backpressure prevents thundering herd problems better than any poll-based throttle — is a general lesson for GPU pipeline design that extends well beyond this specific system.

The message also implicitly defines the remaining work: the PCE fast path still doesn't use pinned backing for a/b/c vectors, leaving some H2D transfers as a potential bottleneck once PCE synthesis becomes fully active. But for the moment, the system is running with near-constant GPU utilization, pinned pool reuse at healthy levels, and transfer overhead eliminated. The zero-copy pipeline is, at last, a reality.