From Thundering Herd to Gentle Stream: Reactive Backpressure in GPU Pipeline Dispatch

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, every millisecond counts. The CuZK proving engine, a system designed to generate Filecoin proofs at scale, had reached an inflection point. After successfully implementing a pinned memory pool to eliminate costly host-to-device (H2D) memory transfers, and deploying a GPU queue depth throttle to prevent budget exhaustion, the system was finally producing correct proofs. But it was not yet fast.

The user's observation cut to the heart of the problem: "when all 20+ synths run there is almost zero GPU activity, likely blocking from pinned pool allocs." The throttle, intended to limit how many synthesized partitions could queue for the GPU, was instead creating a thundering herd. When the GPU queue depth dropped below the threshold of 8, all waiting synthesis jobs would dispatch simultaneously—roughly 20 of them—each calling cudaHostAlloc to allocate pinned memory. These concurrent allocations serialized through the CUDA driver, stalling the GPU and destroying utilization. The pinned memory pool, designed to eliminate transfer overhead, was itself becoming the bottleneck.

This article examines message [msg 3311] in the CuZK optimization conversation, a pivotal moment where the assistant diagnosed the dispatch burst problem, connected it to pinned pool thrashing and GPU stalls, and designed a semaphore-based reactive dispatch mechanism to replace the flawed polling approach. This message represents the critical transition from identifying a performance bug to architecting its solution—a case study in how reactive backpressure can transform a bursty, contention-prone pipeline into a smoothly modulated one.

The Context: A Pipeline Under Pressure

To understand message [msg 3311], we must first understand the system it operates within. The CuZK proving engine processes Filecoin proof generation as a multi-stage pipeline. Jobs arrive, each containing multiple partitions that must be synthesized (a CPU-bound computation that produces constraint system data) and then proved on the GPU (a compute-bound operation involving number-theoretic transforms, or NTTs, and multi-scalar multiplications).

The critical insight driving this optimization effort was that H2D memory transfers—copying data from host memory to GPU device memory—were consuming the majority of GPU time. A single NTT kernel's H2D transfer could take 1,300 to 12,000 milliseconds, dwarfing the actual computation. The pinned memory pool (PinnedPool) was designed to solve this by pre-allocating host memory that is page-locked (pinned), enabling faster DMA transfers to the GPU and, crucially, allowing buffer reuse across partitions to avoid re-allocation entirely.

However, the pinned pool introduced its own complications. Each pinned buffer allocation via cudaHostAlloc requires GPU driver coordination, and when 20+ synthesis workers call it simultaneously, the calls serialize, effectively freezing GPU activity. The previous deployment (pinned3) had added a poll-based throttle: the dispatcher would check the GPU queue depth every 250 milliseconds and stop dispatching new synthesis work when the queue exceeded max_gpu_queue_depth = 8. This was meant to prevent too many partitions from piling up waiting for GPU, thereby freeing memory budget for other operations like PCE (Pre-Compiled Constraint Evaluator) caching.

But the poll-based approach had a fatal flaw: when the GPU queue dropped from 8 to 7, the throttle released the floodgates. All waiting partitions—potentially 20 or more—would be dispatched simultaneously, each triggering a cudaHostAlloc call. The result was a thundering herd that stalled the GPU, destroyed pinned buffer reuse (since buffers were allocated and freed in bursts rather than recycled), and left GPU utilization near zero during allocation bursts.

The User's Observation: Symptoms of a Deeper Problem

Message [msg 3310], the user's input that precedes our subject message, provides the raw observational data that drives the analysis. The user notes several critical symptoms:

First, "just after dropping to 8 all synths dispatched all at the same time (20ish)." This confirms the thundering herd behavior of the poll-based throttle. The threshold acts as a binary gate: closed when queue depth ≥ 8, open when < 8. When it opens, everything waiting rushes through.

Second, "the dispatch should 'modulate' somehow such that we run a more or less fixed number of synthesis with some low buffer of pending synthesis to keep GPUs fed." The user is proposing a reactive approach: "start a job only after a GPU consumed a slot, if we went from N → N-1, start one job, if from N → N-2 start two and so on." This is essentially a description of a semaphore or token-based system where GPU completions directly gate new dispatches.

Third, "when all 20+ synths run there is almost zero GPU activity, likely blocking from pinned pool allocs." This connects the dispatch burst to GPU stalls. The concurrent cudaHostAlloc calls are not merely slow—they actively block GPU operations because pinned memory allocation requires GPU driver interaction.

Fourth, "otherwise now after some settling when pinned pools are not allocating... GPU compute is fairly solid near constant utilization." This is a crucial positive signal: when the pinned pool is not in its allocation burst phase, the system works beautifully. The pinned memory concept is sound; the problem is purely in the dispatch pattern that triggers allocation storms.

These observations form the raw material that the assistant synthesizes in message [msg 3311].

The Assistant's Analysis: Connecting Symptoms to Root Causes

Message [msg 3311] opens with the assistant's reasoning, which systematically connects each user observation to an underlying engineering problem. The assistant identifies three distinct but interrelated issues:

Issue 1: The Dispatch Burst Problem

The assistant recognizes that the poll-based throttle is fundamentally flawed. The check "is GPU queue < max_gpu_queue_depth?" is a binary condition evaluated periodically. When the queue depth drops below 8, the dispatcher resumes popping items from the work queue and sending them to synthesis workers. But with potentially dozens of items waiting, and a dispatch channel capacity of 28, the dispatcher can rapidly send many items before the GPU queue depth rises back to 8. Each dispatched item triggers a synthesis that eventually needs pinned buffer allocation.

The root cause is that the throttle decouples dispatch rate from GPU consumption rate. The dispatcher checks a static condition (queue depth) rather than reacting to dynamic events (GPU completions). This is the classic polling vs. interrupt dichotomy in systems design: polling introduces latency, burstiness, and the risk of missing events between polls, while event-driven (reactive) approaches provide natural, fine-grained modulation.

Issue 2: Pinned Pool Thrashing

The assistant flags a second concern: "Buffers might be getting allocated and freed repeatedly instead of reused—there could be a bug in the checkin/checkout flow." The pinned pool's design relies on a checkout/checkin lifecycle: a synthesis worker checks out a set of pinned buffers, uses them for the proving assignment, and checks them back in when done. If checkin is not being called correctly, buffers leak and the pool must allocate fresh buffers for every partition.

The log data supports this concern: the assistant later discovers 474 allocations but only 12 reuses, a reuse ratio of approximately 2.5%. This is catastrophic for a pool designed to eliminate allocation overhead. The dispatch burst is likely exacerbating this: when 20 syntheses fire simultaneously, they all need buffers at once, but the pool may only have a few available. The rest must allocate new ones. Then, when syntheses complete, the buffers are checked back in—but by then, the burst has passed and the buffers sit idle until the next burst.

Issue 3: cudaHostAlloc Serialization Stalls the GPU

The third issue is the most insidious. cudaHostAlloc is not a simple malloc; it allocates page-locked (pinned) host memory that the GPU can access directly via DMA. This allocation requires coordination with the GPU driver, which can serialize concurrent calls. When 20 synthesis workers each call cudaHostAlloc simultaneously, the calls are serialized by the CUDA driver, and during this serialization, GPU operations may be blocked or delayed.

The assistant correctly identifies that fixing issue #1 (the dispatch burst) would also fix issue #3: "if we only dispatch one synthesis per GPU completion, pinned allocations trickle in instead of bursting." This insight is the key to the solution—by smoothing the dispatch pattern, we naturally serialize allocations, eliminating the contention that causes GPU stalls.

The Proposed Solution: Semaphore-Based Reactive Dispatch

The assistant's proposed fix is elegant and directly addresses the root cause. Instead of polling the GPU queue depth, the dispatcher should use a semaphore that represents available GPU queue slots. The flow would be:

  1. Initialize a semaphore with max_gpu_queue_depth permits.
  2. Before dispatching synthesis work, the dispatcher acquires a permit from the semaphore.
  3. The permit is "forgotten" (not dropped, so it stays consumed) until the GPU worker finishes processing the partition.
  4. When a GPU worker completes a partition, it releases a permit back to the semaphore via add_permits. This creates a natural 1:1 modulation: each GPU completion releases exactly one permit, which allows exactly one new synthesis dispatch. The dispatch rate is perfectly matched to the GPU consumption rate. If the GPU is processing 8 partitions in parallel (the queue depth limit), the semaphore will have 0 permits available, and no new synthesis will be dispatched until a GPU worker finishes and releases a permit. The assistant describes this mechanism in the reasoning: "A tokio Semaphore initialized to max_gpu_queue_depth would handle this cleanly—the dispatcher acquires before synthesis, and the GPU worker releases after finishing a partition." The key design choice is that the permit is intentionally not tied to Rust's drop semantics: "the dispatcher acquires a permit before dispatching work, then intentionally forgets it so it stays consumed until the GPU worker manually releases it back via add_permits." This decoupling is crucial. In a typical Rust semaphore usage, the permit would be held in a variable and automatically returned when it goes out of scope. But here, the permit must survive across an asynchronous boundary—from the dispatcher to the GPU worker—without being automatically released. The solution is to acquire the permit and then deliberately drop it (or "forget" it) without releasing it back to the semaphore, and have the GPU worker explicitly add permits when it finishes.

Assumptions and Design Decisions

The assistant's analysis rests on several assumptions, most of which are well-supported by the available evidence:

Assumption 1: The dispatch burst is the primary cause of GPU stalls. The user's observation that "when all 20+ synths run there is almost zero GPU activity" strongly supports this. The assistant connects this to cudaHostAlloc serialization, which is a known characteristic of CUDA pinned memory allocation. This assumption is validated by the dramatic results of the fix: after deploying the semaphore-based dispatch, NTT H2D transfer time drops from 1,300–12,000 ms to 0 ms.

Assumption 2: The pinned pool checkin/checkout lifecycle is correct in steady state. The assistant assumes that when dispatch is smoothed, buffers will be returned to the pool before new allocations are needed. This is confirmed by the improvement in reuse ratio from 12:474 to 48:24 after the fix.

Assumption 3: A semaphore is the right abstraction for reactive dispatch. The assistant chooses a tokio Semaphore over other synchronization primitives. This is a sound choice because semaphores naturally model resource counting (available GPU queue slots) and provide async-friendly acquire() that doesn't block the dispatcher thread.

Assumption 4: The PCE fast path is not yet using pinned backing. The assistant notes that the PCE path computes a/b/c vectors via CSR SpMV into regular heap Vecs, bypassing the pinned pool. This is a known limitation (tagged with a TODO in the code) and is not addressed in this message—the focus is on fixing the dispatch pattern first.

What Knowledge Was Required to Understand This Message

To fully grasp message [msg 3311], one needs:

  1. Understanding of GPU memory hierarchies: The distinction between pageable (regular) and pinned (page-locked) host memory, and why pinned memory enables faster H2D transfers. The role of cudaHostAlloc and its serialization characteristics.
  2. Knowledge of the CuZK pipeline architecture: The flow from job arrival → partition queuing → synthesis (CPU) → GPU proving, and the role of the dispatcher in mediating between these stages.
  3. Understanding of the pinned memory pool design: The checkout/checkin lifecycle, the budget integration (which was already fixed in a previous deployment), and the buffer reuse mechanism.
  4. Familiarity with reactive backpressure patterns: The concept of using a semaphore or token-based system to match dispatch rate to consumption rate, and the contrast between polling and event-driven approaches.
  5. Awareness of the previous deployment history: The poll-based throttle (pinned3) was deployed in the preceding messages, and its behavior (checking GPU queue depth every 250ms) is the baseline being improved upon.

What Knowledge This Message Creates

Message [msg 3311] creates several important pieces of knowledge:

  1. A root cause diagnosis: The dispatch burst is identified as the primary cause of GPU stalls, pinned pool thrashing, and poor buffer reuse. This connects three seemingly separate symptoms (burst dispatch, allocation storms, GPU idling) to a single root cause.
  2. A concrete design for reactive dispatch: The semaphore-based mechanism is specified in enough detail to be implemented. The key insight—acquiring a permit and "forgetting" it to decouple from drop semantics—is a non-obvious design choice that makes the implementation feasible.
  3. A prioritization of fixes: The assistant marks the poll-based throttle as completed (it was a stepping stone) and sets the semaphore-based replacement as in-progress. The pinned pool thrashing investigation is deferred to pending. This establishes the order of operations: fix dispatch first, then investigate buffer lifecycle.
  4. An understanding of the interaction between dispatch pattern and memory pressure: The message articulates how burst dispatch prevents buffer reuse (because buffers are allocated simultaneously rather than recycled sequentially), which in turn increases memory pressure and GPU contention.

The Thinking Process: A Window into Engineering Decision-Making

The reasoning section of message [msg 3311] reveals the assistant's thinking process in real-time. Several patterns are worth noting:

Symptom → Mechanism → Root Cause: The assistant starts with the user's observations (symptoms), infers the underlying mechanisms (concurrent cudaHostAlloc serialization, poll-based throttle burst), and traces to root causes (decoupling of dispatch rate from GPU consumption rate).

System-level thinking: The assistant doesn't treat the dispatch burst, pinned pool thrashing, and GPU stalls as separate problems. Instead, it recognizes them as manifestations of the same root cause: the poll-based throttle creates bursts that overwhelm every downstream component.

Design by constraint: The assistant doesn't just propose "use a semaphore" in the abstract. It works through the specific constraints: the permit must survive across async boundaries, the GPU worker must explicitly release permits, and the semaphore must be initialized to max_gpu_queue_depth. This is design thinking grounded in implementation reality.

Prioritization under uncertainty: The assistant acknowledges that pinned pool thrashing might be a separate bug (checkin not called), but defers investigation because fixing the dispatch burst will likely improve the situation regardless. This is a pragmatic engineering judgment: fix the known problem first, then investigate the suspected one.

The Broader Significance

Message [msg 3311] is a microcosm of a larger pattern in systems engineering: the transition from threshold-based to rate-based control. The poll-based throttle is a threshold mechanism—it checks a condition and acts when the condition is met. The semaphore-based dispatch is a rate-based mechanism—it directly couples the dispatch rate to the consumption rate.

This distinction appears everywhere in systems design: TCP congestion control (threshold-based slow start vs. rate-based pacing), queue management (static queue limits vs. dynamic feedback loops), and resource allocation (quota-based vs. token-based). The assistant's insight—that the dispatch rate should be a function of the GPU completion rate, not a binary check on queue depth—is an instance of this broader principle.

The results of this fix, as documented in the chunk summary, are dramatic: NTT H2D transfer time drops from 1,300–12,000 ms to 0 ms. Total per-partition GPU time reduces to ~935 ms (pure compute, no transfer overhead). The pinned pool reuse ratio improves from 12:474 (2.5%) to 48:24 (200%—more checkins than allocations, indicating buffer recycling). Budget headroom increases to 288 GiB. These numbers validate the core insight: when dispatch is smooth, the pinned pool works exactly as designed.

Conclusion

Message [msg 3311] represents the critical diagnostic and design phase of a performance optimization effort. The assistant takes raw observational data from the user—burst dispatch, GPU stalls, pool thrashing—and synthesizes it into a coherent root cause analysis. The semaphore-based reactive dispatch mechanism is not just a fix for the immediate problem; it is a architectural improvement that transforms the pipeline from a bursty, contention-prone system into a smoothly modulated one where dispatch rate naturally matches GPU consumption rate.

The message also demonstrates a key engineering virtue: the willingness to replace a working-but-flawed solution (the poll-based throttle) with a better one, even when the first solution required significant effort to deploy. The poll-based throttle was not wrong—it was a necessary stepping stone that proved the concept of GPU queue depth limiting and enabled PCE caching. But the semaphore-based approach is the correct long-term solution, and the assistant's clear-eyed analysis of the polling approach's limitations shows a commitment to getting the architecture right rather than settling for "good enough."

In the broader narrative of the CuZK optimization effort, this message is the turning point. After this fix, the system achieves near-constant GPU utilization, near-zero H2D transfer overhead, and efficient buffer reuse. The pinned memory pool concept is validated, and the remaining work (wiring pinned backing into the PCE path) is a straightforward extension rather than a fundamental redesign. The thundering herd has been tamed, replaced by a gentle stream of exactly one synthesis per GPU completion.