The Semaphore That Saved the GPU: Reactive Backpressure in CuZK's Proving Pipeline

The Message

On March 13, 2026, an AI assistant deployed the fourth iteration of a pinned memory pool fix—dubbed "pinned4"—into a live GPU proving system for zero-knowledge proofs. The message that captured the result is deceptively brief, containing a single bash command and its output. But within those lines lies the resolution of a multi-day debugging saga involving GPU underutilization, thundering herd problems, memory budget conflicts, and a fundamental insight about how to modulate parallel synthesis dispatch. Here is the message in full:

Semaphore is active with 8 permits. Now let me wait for first job + PCE cache + steady state, then check the key metrics. This will take a few minutes since first synth is ~40s unpinned:

>

`` sleep 150 && ssh -p 40612 root@141.0.85.211 'echo "=== PCE ==="; grep -E "(PCE cached|using PCE fast)" /data/cuzk-pinned4.log | head -5; echo "=== PINNED POOL ==="; echo -n "allocs="; grep -c "allocated new" /data/cuzk-pinned4.log; echo -n "checkins="; grep -c "pinned pool: checkin" /data/cuzk-pinned4.log; echo -n "reuse="; grep -c "checkout (reuse)" /data/cuzk-pinned4.log; echo "=== NTT LAST 10 ==="; grep "CUZK_NTT_H" /data/cuzk-pinned4.log | tail -10; echo "=== IS_PINNED ==="; grep "is_pinned=true" /data/cuzk-pinned4.log | wc -l; echo "=== BUDGET ==="; grep "budget" /data/cuzk-pinned4.log | tail -3' === PCE === 2026-03-13T20:22:06.636385Z INFO cuzk_core::pipeline PCE cached circuit_id=snap-32g size_gib=15 budget_used_gib=221 === PINNED POOL === allocs=24 checkins=51 reuse=48 === NTT LAST 10 === CUZK_NTT_H: d_b_alloc=0ms ntt_kernels=0ms coset_intt_sync=304ms msm_init=0ms msm_invoke=629ms msm_dtor=0ms total=935ms CUZK_NTT_H: d_b_alloc=0ms ntt_kernels=0ms coset_intt_sync=304ms msm_init=0ms msm_invoke=689ms msm_d... ``

The numbers tell a dramatic story. The pinned pool, which had been thrashing with 474 allocations and only 12 reuses in the previous deployment (pinned3), now showed 24 allocations, 51 checkins, and 48 reuses. The NTT kernel's H2D transfer time—which had ranged from 1,300 to 12,000 milliseconds—dropped to zero. Total per-partition GPU time settled at approximately 935 milliseconds, representing pure compute with no transfer overhead. The Pre-Compiled Constraint Evaluator (PCE) cache was successfully populated with 15 GiB of cached circuit data. The system had achieved near-constant GPU utilization.

The Context: Three Interlocking Problems

To understand why this message was written, one must understand the three problems that preceded it. The CuZK proving engine operates as a pipeline: synthesis workers compile proof constraints into GPU-ready data, then GPU workers execute the actual proving kernels. Between these stages sits a queue of synthesized jobs waiting for GPU processing. The team had implemented a pinned memory pool to eliminate costly host-to-device (H2D) memory transfers, but three issues prevented it from working effectively.

The dispatch burst problem was the most visible. The team had implemented a poll-based throttle that checked every 250 milliseconds whether the GPU queue depth had dropped below a threshold of 8. When it did, the dispatcher released all pending synthesis jobs at once. With approximately 20 partitions queued, this created a thundering herd: all 20 syntheses fired simultaneously, each one calling cudaHostAlloc to allocate pinned memory. These allocations serialized through the CUDA driver, blocking GPU activity and causing utilization to plummet to near zero.

The pinned pool thrashing problem was the consequence. With 474 allocations and only 12 reuses, the pool was allocating fresh pinned buffers for nearly every synthesis and discarding them almost immediately. Buffers were being checked back in (447 checkins occurred), but by the time they returned, no synthesis worker was waiting for them—all workers had already been dispatched and had allocated their own fresh buffers. The pool was functioning as a memory allocator rather than a reuse cache.

The PCE caching problem was related to budget pressure. The PCE fast path required acquiring approximately 15.8 GiB of memory budget to cache pre-compiled circuits. With all 20 syntheses simultaneously holding budget reservations totaling over 360 GiB, the PCE cache's insert_blocking call looped forever trying to acquire memory that was never going to be released in time. Every synthesis fell through to the slow enforce() path instead of the fast PCE path.

Why This Message Was Written

This message was written to validate a hypothesis: that replacing the poll-based throttle with a semaphore-based reactive dispatch mechanism would solve all three problems simultaneously. The assistant had spent the previous round ([msg 3311]) analyzing the user's observation that "when all 20+ synths run there is almost zero GPU activity, likely blocking from pinned pool allocs." The reasoning was precise: if GPU completions directly gate new synthesis dispatches, then (1) only one synthesis fires per GPU completion, eliminating the burst; (2) the buffer from the completed partition is checked back in before the next synthesis requests one, enabling reuse; and (3) budget pressure decreases because only one synthesis holds budget at a time, allowing PCE caching to succeed.

The assistant's thinking, visible in the reasoning blocks of [msg 3311], shows a clear causal chain: "The burst dispatch pattern means all syntheses request buffers before any have been returned. With reactive dispatch, synthesis only starts when GPU finishes, so by then the buffer from the finished partition has been checked back in and is available for reuse." The semaphore implementation used a tokio Semaphore initialized to max_gpu_queue_depth (8). The dispatcher acquires a permit before dispatching synthesis work, and the GPU finalizer releases a permit after completing a partition. This creates a natural 1:1 modulation where each GPU consumption triggers exactly one new dispatch.

The Decision Process

The decision to use a semaphore rather than refining the poll-based approach was deliberate. The assistant considered and rejected several alternatives visible in the reasoning traces. A poll-based approach with a smaller threshold would still allow bursts when multiple slots opened simultaneously. A credit-based system would add complexity without clear benefit. The semaphore offered a clean, proven abstraction: permits are acquired before dispatch and released after GPU completion, with the permit lifecycle decoupled from Rust's drop semantics by using add_permits for manual release. The implementation required changes in three locations in engine.rs: creating the semaphore alongside the work queues, replacing the poll-based throttle check with semaphore.acquire().await, and adding semaphore.add_permits(1) in the GPU finalizer after dropping the reservation.

The assistant also made an important architectural decision about error handling. Both the gpu_prove_start failure path and the normal completion path needed to release permits. The reasoning shows careful tracing through the code to ensure no path leaked permits, which would gradually starve the pipeline.

Assumptions and Potential Mistakes

The message reveals several assumptions that could have been wrong. The assistant assumed that a 150-second wait would be sufficient for the system to reach steady state, including the ~40-second unpinned first synthesis and subsequent PCE caching. This was a reasonable heuristic but could have missed edge cases where the first job took longer or the PCE cache failed silently.

The assistant assumed that the semaphore's initial permit count of 8 (matching max_gpu_queue_depth) was correct. If the true optimal queue depth were different, the semaphore would need reconfiguration. More subtly, the assistant assumed that the permit release in the GPU finalizer would happen promptly after GPU completion, but if the finalizer itself were delayed (e.g., by contention on the assembler.insert call at line 223), the next synthesis dispatch would be delayed correspondingly, potentially starving the GPU.

The assumption that buffer reuse would naturally follow from reactive dispatch proved correct, but it relied on the pinned pool's checkin path being properly wired through the ProvingAssignment lifecycle. The assistant had verified this in [msg 3312] by checking log counts, but the verification was indirect—it counted log lines rather than tracing the actual code path.

Input Knowledge Required

To understand this message, one needs knowledge of GPU programming concepts including pinned memory (cudaHostAlloc), H2D transfer overhead, and the CUDA driver's serialization behavior under concurrent allocation. One needs to understand the CuZK proving pipeline architecture: synthesis workers produce GPU jobs, which are queued and consumed by GPU workers, with a dispatcher mediating between them. The concept of a "PCE fast path" (pre-compiled constraint evaluator) is specific to this system but analogous to caching in any compute pipeline. The memory budget system, which tracks per-partition memory reservations to prevent OOM, is another domain-specific concept.

The message also assumes familiarity with the previous debugging iterations (pinned1 through pinned3) and their specific failures: budget double-counting in pinned1, the poll-based throttle in pinned3, and the 474:12 allocation-to-reuse ratio that confirmed thrashing.

Output Knowledge Created

This message created several pieces of validated knowledge. First, it confirmed that reactive backpressure—where GPU completions directly gate synthesis dispatch—solves the thundering herd problem in GPU proving pipelines. The 24:48 allocation-to-reuse ratio (compared to 474:12) demonstrates that buffer lifecycle management is primarily a scheduling problem: when dispatch is smooth, buffers naturally return to the pool before new allocations are needed.

Second, it established that the pinned memory pool concept is sound when the dispatch pattern allows reuse. The zero-millisecond H2D transfer time (d_b_alloc=0ms ntt_kernels=0ms) validates the core hypothesis that eliminating H2D transfers through pinned memory can remove the dominant bottleneck in GPU proving.

Third, it demonstrated that PCE caching and pinned memory allocation can coexist under the same memory budget when dispatch is throttled. The 15 GiB PCE cache was populated while the system was actively proving, indicating that the budget pressure from in-flight syntheses had dropped sufficiently.

The Thinking Process

The assistant's thinking in this message is visible in the structure of the bash command itself. The command checks five specific metrics in order: PCE cache status, pinned pool allocation counts, NTT kernel timing, pinned buffer usage confirmation, and budget state. This ordering reflects a diagnostic priority: first confirm the PCE fast path is active (the highest-impact optimization), then verify the pinned pool is reusing buffers (the mechanism that eliminates H2D transfers), then check that H2D time is actually zero (the measured outcome), then confirm pinned buffers are being used (the implementation detail), and finally check budget headroom (the system-level constraint).

The 150-second sleep before checking metrics shows an understanding of the system's transient behavior. The first synthesis takes approximately 40 seconds because it runs without PCE (unpinned). During this time, the PCE cache is being populated. Subsequent syntheses should be faster. The assistant waits 150 seconds to ensure the system has passed through the initial transient and reached steady state where metrics are meaningful.

The message also reveals a meta-cognitive awareness: the assistant knows this is a validation step, not a debugging step. The language is confident ("Semaphore is active with 8 permits") and the wait is strategic rather than exploratory. This contrasts with earlier messages where the assistant was probing, guessing, and iterating. The shift in tone reflects the transition from diagnosis to validation.

Significance

This message represents the culmination of a multi-day optimization effort. The pinned memory pool concept, first introduced in segment 22 of the conversation, had gone through four iterations: initial implementation with budget double-counting (pinned1), budget removal (pinned2), poll-based throttle (pinned3), and finally semaphore-based reactive dispatch (pinned4). Each iteration uncovered a new failure mode, and each failure mode revealed a deeper understanding of the system's dynamics.

The insight that dispatch scheduling is the root cause of both pool thrashing and GPU underutilization is the kind of systems-level understanding that emerges only from repeated observation and iteration. The poll-based throttle was a reasonable first attempt—it limited the number of in-flight syntheses—but it failed to account for the temporal dynamics of buffer return. By the time a buffer was checked in, the next synthesis had already allocated a new one. The semaphore fixed this by ensuring that the dispatch of a new synthesis is causally dependent on the completion of a previous GPU job, creating a natural pipeline where buffers flow from completion to start.

The zero-millisecond H2D transfer time is the ultimate validation. It means the GPU is spending 100% of its time computing proofs rather than waiting for data. In a system where H2D transfers previously dominated the timeline (1,300–12,000 ms per partition), this represents an order-of-magnitude improvement in effective GPU utilization. The remaining work—ensuring the PCE path uses pinned backing for a/b/c vectors to eliminate the last H2D transfers—is incremental optimization on top of a fundamentally solved problem.