From Silent Fallback to Near-Zero Transfers: The Full Arc of GPU Pinned Memory Pool Optimization

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, every millisecond of GPU idle time represents wasted computational capacity. The CuZK proving engine, a system responsible for generating Filecoin proofs at scale, had been suffering from a persistent and puzzling problem: the GPU was spending most of its time idle, waiting for data. Instrumentation revealed the culprit — Host-to-Device (H2D) memory transfers of the a/b/c vectors were taking 1,300 to 12,000 milliseconds per partition, while actual GPU compute required only ~935 milliseconds. The GPU was spending 60-90% of its time waiting for data to arrive.

The solution seemed straightforward: use CUDA's pinned (page-locked) memory, which allows the GPU to access host memory directly via DMA, eliminating the explicit transfer step. The team designed a PinnedPool — a pre-allocated pool of pinned buffers that synthesis workers could check out, fill with circuit data, and hand directly to the GPU. This zero-copy approach promised to eliminate the H2D bottleneck entirely.

But as this chunk of the coding session reveals, deploying a pinned memory pool in a complex production pipeline is anything but straightforward. The team would discover not one, but four distinct layers of problems — budget double-counting, PCE cache starvation, dispatch burst thundering herds, and buffer lifecycle thrashing — each masked by the previous, each requiring a fundamental rethinking of the pipeline's control architecture. The journey from the first deployment to the final solution is a masterclass in systems debugging, revealing how performance optimizations in complex pipelines can fail in unexpected ways, and how the deepest problems often require changing not just what the system allocates, but when and how it dispatches work.

Part I: The Silent Fallback — Budget Double-Counting

The first deployment of the pinned memory pool (pinned1) was a failure — but a silent one. The logs showed no errors, no crashes, no warnings. Every synthesis completed normally. But every single one completed with is_pinned=false, meaning the pinned pool had silently fallen back to heap allocations.

The root cause, traced through multiple layers of code in messages [msg 3218] through [msg 3229], was a budget double-counting bug. The PinnedAbcBuffers::checkout() method called budget.try_acquire() for memory that was already accounted for in per-partition budget reservations. With five jobs each dispatching 16 partitions, approximately 362 GiB of memory was reserved before any pinned allocation was attempted. The budget system, seeing only 5 GiB remaining, denied every pinned allocation. The ? operator in the checkout code silently converted these denials into heap fallbacks — no error, no warning, just a quiet path change that defeated the entire optimization.

The fix, deployed as pinned2, was surgically precise: remove budget tracking from the pinned pool entirely. The pool's allocate(), shrink(), and drop() methods were all stripped of their budget integration. The result was immediate: logs began showing pinned prover created and is_pinned=true completions for the first time. The pinned pool was finally allocating.

But the victory was short-lived. The logs also revealed that GPU utilization remained poor. The NTT kernel timings were still in the 1,300-12,000 ms range. Something else was wrong.

Part II: The Cache That Wasn't — PCE Starvation

The second problem emerged from the same root cause as the first: budget exhaustion. The Pre-Compiled Constraint Evaluator (PCE) is a caching mechanism that pre-computes the constraint system structure from the first partition's synthesis, allowing subsequent partitions to skip the expensive enforce() constraint evaluation and use a fast CSR SpMV path instead. This PCE fast path is 3-5x faster than standard synthesis — a critical optimization for pipeline throughput.

But the PCE's insert_blocking call needed to acquire 15.8 GiB of budget to cache the constraint system. With only 5 GiB remaining after the per-partition reservations, insert_blocking looped forever, never succeeding. Every synthesis was forced through the slow enforce() path, compounding the memory pressure and keeping GPU utilization low.

The diagnosis, captured in messages [msg 3264] through [msg 3271], revealed that the budget system was starving the PCE cache. The per-partition budget reservations were consuming nearly all available memory before PCE had a chance to acquire its share. The fix for the pinned pool (removing budget) had helped the pool itself, but it hadn't addressed the fundamental budget pressure that was starving PCE.

The user identified the root cause in [msg 3272]: too many partitions were being synthesized simultaneously, flooding the GPU queue and consuming budget that PCE needed. Their suggestion was a throttle — stop dispatching new synthesis jobs once more than N partitions are waiting for GPU processing.

Part III: The Throttle That Saved the Cache — GPU Queue Depth Control

The assistant implemented a GPU queue depth throttle (max_gpu_queue_depth = 8) that pauses synthesis dispatch when too many partitions are queued waiting for GPU, freeing budget for PCE caching. The implementation, traced in messages [msg 3274] through [msg 3293], involved reading the pipeline architecture, understanding the work queue and dispatch channel, and inserting a depth check before the dispatcher pops from the synthesis work queue.

The key design decision was ordering: the GPU queue depth check must happen before budget acquisition, not after. If the dispatcher acquired budget first and then checked the queue depth, it would hold budget while waiting, defeating the purpose. The correct ordering — check depth, then acquire budget, then dispatch — ensures that budget is only committed when the GPU can actually consume the work.

The deployment of pinned3, captured in [msg 3293] through [msg 3304], was a moment of genuine triumph. The logs showed:

PCE cached circuit_id=snap-32g size_gib=15 budget_used_gib=385

For the first time in the entire optimization effort, PCE caching had succeeded. The throttle had freed enough budget for the 15.8 GiB PCE to be inserted. The user's hypothesis was validated: throttling synthesis dispatch relieved the budget pressure that was starving the cache.

But the triumph was tempered. When the assistant checked whether any partition was actually using the cached PCE, the answer was devastating: zero. grep -c "using PCE fast path" returned 0; grep -c "using standard synthesis" returned 35. Every single partition had been dispatched before PCE was cached at 19:58:17. The last standard synthesis started at 19:57:45 — 32 seconds earlier.

The throttle had prevented new dispatches from overwhelming the budget, but it couldn't retroactively accelerate the work already in flight. The pipeline was still draining the initial batch of 80 partitions, all committed to the slow path before PCE was available.

Part IV: The Dispatch Burst — A Thundering Herd of Allocations

As the team watched the logs stream in, a deeper problem became visible. The user's observation in [msg 3310] cut to the heart of the issue:

"just after droppping to 8 all synths dispatched all at the same time (20ish) — the dispatch should 'modulate' somehow such that we run a more or less fixed number of synthesis with some low buffer of pending synthesis to keep GPUs fed"

The poll-based throttle was creating a classic thundering herd problem. When the GPU queue depth dropped below the threshold of 8, the dispatcher would resume popping items from the work queue — and it would race through as many as the channel capacity (28) and budget would allow. Twenty-plus syntheses would fire simultaneously, each trying to allocate pinned memory via cudaHostAlloc. The CUDA driver serializes cudaHostAlloc calls behind an internal mutex, so all 20+ threads would block each other. The GPU, waiting for those pinned buffers to be ready before it could launch kernels, saw nothing to do and sat idle.

The user's diagnosis was precise: the dispatch pattern, not the allocation strategy, was the root cause of the allocation contention. Even a perfectly tuned pinned pool would thrash if 20+ threads all tried to allocate from it simultaneously. The poll-based throttle created a binary state — open or closed — with no fine-grained control over how many items passed through when the gate opened.

The user proposed a fundamentally different approach: reactive backpressure. Instead of polling the GPU queue depth and releasing work in bursts, the dispatcher should be event-driven. Each GPU completion signals that a slot has opened up, and the dispatcher responds by starting exactly one new synthesis. If multiple slots open simultaneously (e.g., two GPUs finish at nearly the same time), the dispatcher starts exactly that many new syntheses. The system self-modulates: the dispatch rate naturally matches the GPU consumption rate.

The mechanism was described with elegant simplicity: "if we went from N -> N-1, start one job, if from N -> N-2 start two and so on — should self modulate." This is essentially a semaphore-based dispatch: a semaphore initialized to max_gpu_queue_depth is decremented when a synthesis is dispatched and incremented when a GPU completion occurs. The dispatcher blocks on acquire() before dispatching, ensuring that the number of in-flight items never exceeds the limit and that dispatch is naturally paced by completions.

Part V: The Semaphore Solution — Reactive Dispatch

The assistant implemented the semaphore-based reactive dispatch, deployed as pinned4. The results were dramatic and immediate:

Part VI: The Remaining Frontier — PCE and Pinned Integration

Despite the dramatic improvements, one architectural gap remained. The PCE fast path, which was now being used (15 GiB cached, synthesis using the fast CSR SpMV path), computed its a/b/c vectors into regular heap Vecs, completely bypassing the pinned memory pool. The assistant had left a TODO comment about this in pipeline.rs:1230 — a note to wire the pinned pool into the PCE path — but it had been forgotten in the urgency of deployment.

This meant that even with the pinned pool working perfectly and the dispatch smoothly modulated, the PCE fast path was still using unpinned memory for its vector allocations. The H2D transfers for those vectors were still happening, but they were now masked by the PCE path's 3-5x synthesis speedup. The synthesis was so much faster that the H2D transfers, while still present, consumed a smaller fraction of the total time.

The team recognized this as the next frontier. The pinned pool concept was validated — when allocations were serialized and buffers were reused, GPU utilization remained high and transfer overhead disappeared. But the PCE path needed to be wired into the pinned pool to eliminate the last H2D transfers entirely.

Lessons in Systems Debugging

This chunk of the coding session offers several enduring lessons for systems engineering:

1. Silent fallbacks are the most dangerous bugs. The budget double-counting bug produced no errors, no crashes, no warnings. Every synthesis completed normally — just 100x slower. The ? operator in Rust, which silently propagates errors, masked the failure completely. Making failures visible is often as important as fixing them.

2. Performance optimizations interact in unexpected ways. The pinned pool and the PCE cache were designed independently, but they shared a common resource: memory budget. Fixing one (by removing budget from the pinned pool) didn't help the other (which needed budget for caching). The throttle was a cross-cutting fix that addressed the shared root cause: too many in-flight partitions consuming too much budget.

3. Control mechanism matters as much as resource management. The poll-based throttle was a reasonable first attempt, but it created a thundering herd problem that defeated the pinned pool. The semaphore-based reactive dispatch was a fundamentally different approach — event-driven rather than poll-based — and it solved the problem not by changing what was allocated, but by changing when and how work was dispatched.

4. The first batch always loses. The PCE cache couldn't help the first batch of partitions because they were dispatched before the cache was populated. This is a general principle: any optimization that requires a warm-up phase is vulnerable to burst dispatch patterns. The system must either pre-warm the cache, delay dispatch until the cache is ready, or accept that the first batch will use the slow path.

5. Real-time observation beats static analysis. The user's observation of the dispatch burst — watching the logs stream in and correlating the burst with GPU idle periods — identified a problem that code review alone would never have caught. The system's dynamic behavior, not its static structure, was the bottleneck.

Conclusion

The journey from pinned1 to pinned4 transformed the CuZK proving engine's GPU utilization from a system where the GPU spent 60-90% of its time waiting for data to one where H2D transfers were effectively zero and GPU compute was near-constant. Each deployment revealed a new layer of complexity: budget double-counting masked by silent fallbacks, PCE cache starvation masked by budget exhaustion, dispatch bursts masked by poll-based throttling, and buffer thrashing masked by allocation contention.

The final solution — semaphore-based reactive dispatch — was elegant in its simplicity. By replacing a poll-based threshold check with an event-driven semaphore, the team eliminated the thundering herd problem, improved buffer reuse from 2.5% to 67%, freed 288 GiB of budget headroom, and reduced per-partition GPU time to pure compute. The pinned memory pool concept was validated: when allocations are serialized and buffers are reused, GPU utilization remains high and transfer overhead disappears.

The remaining work — wiring the pinned pool into the PCE fast path — represents the final frontier. But the fundamental architecture is now sound. The pipeline self-modulates, the GPU stays fed, and the pinned pool delivers its promised zero-copy transfers. What began as a simple "use pinned memory" optimization became a deep exploration of pipeline dynamics, control theory, and the subtle ways that performance optimizations can interact. It is a reminder that in complex systems, the deepest problems are rarely where you first look — and the most effective solutions often change not what the system does, but when it does it.