The Semaphore That Tamed the Thundering Herd: Reactive Backpressure in a GPU Proving Pipeline

In the span of a single, deceptively brief message, a fundamental architectural flaw in a high-performance GPU proving pipeline was corrected. The message, from an AI assistant in an opencode coding session, reads in its entirety:

Now replace the poll-based throttle in the dispatcher with semaphore acquire: [edit] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

This is the kind of message that, on its surface, appears to be a routine refactor — swapping one synchronization primitive for another. But beneath this terse surface lies a cascade of diagnostic reasoning, a deep understanding of GPU pipeline dynamics, and a fix that transformed a system suffering from near-zero GPU utilization into one running at near-constant efficiency. To understand why this message matters, one must understand the chain of failures that led to it.

The Dispatch Burst Problem

The context for this message is a GPU-accelerated zero-knowledge proof system called CuZK, which processes proof partitions through a pipeline: synthesis (circuit evaluation), GPU proving, and finalization. The team had recently implemented a pinned memory pool to eliminate costly host-to-device (H2D) memory transfers — a major bottleneck that had been causing GPU underutilization. The pinned pool worked by pre-allocating GPU-pinned host memory and reusing buffers across partitions, avoiding repeated cudaHostAlloc calls.

However, when the pinned pool was deployed, a new problem emerged. The system used a poll-based throttle to limit how many synthesized partitions could queue up waiting for the GPU. Every 250 milliseconds, a dispatcher checked whether the GPU work queue depth had fallen below a threshold (max_gpu_queue_depth = 8). When it did, all waiting synthesis jobs were dispatched at once — sometimes 20 or more simultaneously.

This thundering herd had devastating consequences. Each synthesis worker, upon starting, attempted to check out pinned buffers from the pool. With 20 workers all calling cudaHostAlloc at the same time, the CUDA driver serialized these allocations, blocking the GPU and causing near-zero GPU activity during the burst. The pinned pool, designed for reuse, instead became a scene of frantic allocation: logs showed 474 allocations but only 12 reuses, with buffers being allocated, used once, and never seen again because no worker was waiting when they were returned.

The user captured the problem precisely in [msg 3310]: "when all 20+ synths run there is almost zero GPU activity, likely blocking from pinned pool allocs — otherwise now after some settling when pinned polls are not allocating (are we trashing them?) GPU compute is fairly solid near constant utilization." This observation — that GPU utilization was excellent except during allocation bursts — was the key insight that pointed toward the dispatch mechanism as the root cause.

Why Polling Failed

The poll-based approach had a fundamental flaw: it decoupled dispatch decisions from GPU consumption events. The dispatcher checked the queue depth on a fixed timer, not in response to completions. This meant that when the queue finally dipped below the threshold, the dispatcher had no information about how many slots had opened up or when they had opened. It simply released everything waiting.

This is a classic control-theory problem: a polling loop with a threshold creates a binary on/off behavior rather than proportional modulation. The system oscillated between "no dispatch" and "dispatch everything," with no intermediate state. The GPU, meanwhile, could only consume partitions one at a time, so dumping 20 syntheses into the pipeline all at once guaranteed that most would find the pinned pool's free list empty and allocate fresh buffers — defeating the entire purpose of the pool.

The Semaphore Solution

The assistant's reasoning in [msg 3311] laid out the alternative: "Instead of checking 'is GPU queue < 8?', the dispatcher should be reactive: 'GPU consumed one slot → start one new synthesis'. This self-modulates to match GPU consumption rate."

The mechanism chosen was a tokio::sync::Semaphore initialized with max_gpu_queue_depth permits. The dispatcher would acquire a permit before dispatching synthesis work, blocking if none were available. The GPU finalizer would release a permit after completing a partition — right at the point where the budget reservation was dropped. This created a natural 1:1 modulation: each GPU completion triggered exactly one new synthesis dispatch.

The key design insight was to decouple the permit lifecycle from Rust's drop semantics. The dispatcher acquires a permit but then intentionally "forgets" it (via mem::forget), keeping it consumed until the GPU worker manually releases it via add_permits(1). This ensures the permit's lifetime is tied to actual GPU processing time, not to the lifetime of any Rust object.

The Edit Itself

The subject message ([msg 3320]) is the edit that implements this in the dispatcher. The preceding message ([msg 3319]) had already created the semaphore alongside the work queues and passed it into the dispatcher closure. This edit replaces the polling loop — which had checked gpu_work_queue.len() every 250ms and conditionally dispatched — with a simple semaphore.acquire().await call. The dispatcher now blocks at this point until a permit is available, which only happens when the GPU finishes a partition.

The edit itself is small — likely a single line change — but its effects are systemic. By changing when synthesis is dispatched, it changes the entire timing and ordering of buffer allocation, GPU submission, and memory reuse. The poll-based throttle was a local fix for a local problem (queue depth); the semaphore is a global coordination mechanism that links dispatch rate to GPU consumption rate.## The Diagnostic Chain

The semaphore fix did not emerge from abstract reasoning alone. It was grounded in concrete data that the assistant gathered through a diagnostic chain spanning multiple messages. In [msg 3312], the assistant ran a log analysis command that revealed the shocking allocation statistics:

=== ALLOCS ===
474
=== CHECKINS ===
447
=== REUSE ===
12

These numbers told a damning story. The pinned pool was allocating 474 new buffers but only reusing 12 — a reuse ratio of roughly 2.5%. The 447 checkins showed that buffers were being returned to the pool, but they weren't being picked up by waiting workers. The assistant's reasoning in [msg 3313] correctly diagnosed the root cause: "by the time a buffer is checked in, no synthesis worker is waiting for it — synthesis was already dispatched en masse and each one allocated fresh."

This is a crucial insight about the interaction between dispatch timing and buffer lifecycle. The poll-based throttle allowed all syntheses to start simultaneously, each requesting buffers before any had completed. The checkins happened later (during GPU finalization), but by then the next batch of syntheses had already allocated fresh buffers. The semaphore fix breaks this cycle by ensuring that synthesis only starts after a GPU completion, which means the buffer from the completed partition is already checked back in and available for reuse.

Assumptions and Their Validation

The assistant made several assumptions in designing this fix. First, it assumed that the pinned pool's checkin/checkout mechanism was working correctly — that buffers were being returned promptly after GPU completion. The 447 checkins (against 474 allocations) validated this: buffers were being returned, just not reused. Second, it assumed that cudaHostAlloc serialization was the primary cause of GPU stalls during burst dispatches. This was confirmed by the user's observation that GPU activity dropped to near-zero during allocation bursts but recovered during steady-state reuse.

A third assumption was more subtle: that the semaphore-based approach would not introduce deadlocks or starvation. The assistant carefully designed the permit lifecycle so that the dispatcher acquires before dispatch and the GPU finalizer releases after completion. This creates a closed loop where the number of in-flight partitions is bounded by max_gpu_queue_depth. The assistant verified this by tracing through the error paths in [msg 3326] and [msg 3327], adding permit releases for both the gpu_prove_start failure path and the error-handling branch.

Input Knowledge Required

To understand this message, one needs knowledge spanning several domains. First, an understanding of GPU pipeline architecture: the distinction between synthesis (CPU-bound circuit evaluation), GPU proving (CUDA kernel execution), and finalization (proof assembly). Second, familiarity with CUDA memory management — specifically, that cudaHostAlloc allocates pinned (page-locked) host memory that enables fast H2D transfers but requires GPU driver synchronization, making concurrent allocations a serialization bottleneck. Third, knowledge of Rust's async concurrency model, particularly tokio::sync::Semaphore and the mem::forget pattern for decoupling permit lifetimes from scope. Fourth, an understanding of control theory concepts like reactive vs. poll-based modulation and the thundering herd problem.

Output Knowledge Created

This message produced a concrete artifact: a code edit that replaced a polling loop with a semaphore acquire. But the output knowledge extends beyond the code change. The message validated a design principle: that in GPU pipelines, dispatch should be reactive to completions rather than predictive of queue depth. It demonstrated that buffer reuse in a memory pool depends critically on dispatch timing — that a pool's effectiveness is determined not just by its allocation strategy but by when consumers request buffers relative to when producers return them. And it established a pattern for coordinating dispatch rate with GPU consumption rate that could be applied to other pipeline stages.

The Results

The impact of this change was dramatic, as documented in subsequent messages. After deploying the semaphore-based dispatch as pinned4, H2D transfer time for NTT kernels dropped from 1,300–12,000 ms to 0 ms. Total per-partition GPU time reduced to approximately 935 ms — pure compute, with no transfer overhead. The pinned pool reuse ratio improved from 12:474 to 48:24, and budget headroom increased to 288 GiB. The PCE fast path, which had been cached but unused because no synthesis could acquire budget, became active with 15 GiB cached.

Conclusion

The message "Now replace the poll-based throttle in the dispatcher with semaphore acquire" appears, on the surface, to be a routine refactor. But it represents the culmination of a deep diagnostic process that traced a GPU utilization problem through three layers of failure: a poll-based throttle that caused burst dispatches, burst dispatches that triggered thundering-herd cudaHostAlloc calls, and allocation serialization that stalled the GPU. The fix — a single semaphore — addressed all three layers simultaneously by introducing reactive backpressure. It is a testament to the power of understanding system dynamics at multiple levels: the CUDA driver, the async runtime, the memory pool, and the pipeline architecture. Sometimes the smallest changes have the most profound effects.