From Permits to Targets: Redesigning GPU Pipeline Dispatch in the CuZK Proving Engine

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proving, every millisecond of GPU idle time is lost revenue and every excess gigabyte of memory pressure risks OOM kills. The CuZK proving engine, a custom Rust codebase for generating Filecoin proof types (SnapDeals, WindowPoSt, WinningPoSt), sits at the intersection of these competing pressures: it must keep the GPU fed with synthesized work while simultaneously minimizing memory consumption to leave room for PCE (Pre-Compiled Constraint Evaluator) caching and SRS (Structured Reference String) data. Message [msg 3386] represents a pivotal moment in this balancing act — the assistant's summary of a fundamental redesign of the GPU pipeline dispatch mechanism, replacing a semaphore-based permit system with a waiting-target dispatch model built on Tokio's Notify primitive.

This article examines that message in depth: the reasoning that motivated the change, the assumptions embedded in the new design, the decisions made during implementation, and the knowledge produced by this single, dense summary.

The Message in Full

The assistant writes:

Replaced semaphore-based dispatch with waiting-target dispatch in engine.rs:

- Old model: Semaphore(N) permits — dispatcher acquires permit before synthesis, GPU finalizer returns permit via add_permits(1). Limits total in-flight work (synthesis + GPU) to N.

- New model: Notify + gpu_work_queue.len() check — dispatcher counts partitions actually waiting for GPU. If waiting >= target, sleep on gpu_done_notify. If waiting < target, dispatch work. GPU finalizer calls notify_one() to wake dispatcher for re-evaluation.

Behavior: - Ramp-up: waiting=0, deficit=N, dispatcher rapidly dispatches N items (budget/workers are the natural backpressure) - Steady state: GPU finishes one → waiting drops to N-1 → dispatcher dispatches one → converges to 1:1 - Self-regulating: if synthesis is slower than GPU, waiting stays low, more concurrent syntheses run. If GPU is slower, waiting hits target, syntheses pause. The system finds the minimum concurrency needed.

Config field unchanged (max_gpu_queue_depth = 8 in TOML still works), just updated the doc comment to reflect target semantics.

At first glance, this reads as a straightforward technical summary. But beneath the surface lies a rich story of iterative debugging, a fundamental shift in control philosophy, and a carefully reasoned design that balances competing system constraints.

The Motivation: Why the Semaphore Was Wrong

To understand why this change was necessary, we must first understand what the semaphore model was doing wrong. The user's critique in <msg id=3356 is> blunt and precise: "Didn't want 8 basic 'permits', that's actually wrong way to modulate scheduling." The semaphore with N permits limited the total number of in-flight partitions — meaning partitions currently being synthesized plus partitions waiting for the GPU plus partitions actively being processed by the GPU. This is a coarse, undifferentiated throttle. It cannot distinguish between "synthesis is running" and "synthesis is done and waiting for GPU attention."

The user's insight was that the correct control signal is the number of partitions that have completed synthesis and are waiting in the GPU work queue. This is the true measure of pipeline pressure. If too many partitions are waiting, memory pressure increases because each waiting partition holds its synthesized data (a/b/c vectors) in pinned GPU buffers. If too few are waiting, the GPU starves and utilization drops. The semaphore conflates these two regimes, making it impossible to tune for both memory efficiency and GPU utilization simultaneously.

The assistant's reasoning in [msg 3361] reveals the depth of this analysis. The assistant walks through multiple design iterations in their head: first considering a simple deficit-check loop, then worrying about over-dispatch during ramp-up, then considering an in-flight counter, and finally arriving at the clean Notify-based design. The key insight is that the system should be event-driven by GPU completions rather than gated by permits. When a GPU finishes a job, the dispatcher wakes up, checks how many synthesized partitions are waiting, calculates the deficit against the target, and dispatches exactly that many new synthesis tasks. This creates a natural feedback loop.

The Design Decisions Embedded in the Summary

The message makes several important design decisions, some explicit and some implicit:

Choice of Notify over Semaphore: The assistant chose Tokio's Notify primitive — a simple wake-one-on-notify mechanism — rather than the more complex semaphore. This is appropriate because the new model doesn't need to track permit counts. The dispatcher doesn't acquire a resource; it simply waits for a signal that conditions may have changed. Notify provides exactly this: a one-shot wakeup that can be called multiple times before the waiter actually waits (the notifications are stored and consumed). This is crucial because multiple GPU completions may occur while the dispatcher is actively dispatching work for a previous deficit.

The deficit calculation: The dispatcher computes deficit = target - gpu_work_queue.len() and dispatches that many items. During ramp-up, when the queue is empty, this means the dispatcher rapidly dispatches N items in a burst. The assistant notes that "budget/workers are the natural backpressure" — meaning the memory budget and worker pool limits will naturally constrain how aggressively this burst can proceed. This is a deliberate design choice: don't add artificial limits during ramp-up; let the existing constraints handle it.

Steady-state convergence: The assistant predicts that in steady state, the system will converge to a 1:1 dispatch ratio — one GPU completion triggers one new synthesis. This is the ideal behavior: exactly enough work to keep the GPU fed without building up excess queue depth. The system finds the minimum concurrency needed, which directly addresses the user's goal of "running minimum number of synthesis reducing memory pressure/trashing."

Self-regulation under imbalance: The summary describes how the system adapts to different bottleneck scenarios. If synthesis is slower than GPU, the waiting count stays low, so the dispatcher keeps dispatching more work — effectively increasing synthesis concurrency to match GPU demand. If GPU is slower, waiting builds up to the target, and the dispatcher pauses. This asymmetry is important: the system naturally biases toward keeping the GPU fed, because GPU idle time is the more expensive failure mode.

Config backward compatibility: The assistant chose to keep the existing max_gpu_queue_depth config field name despite the semantic change from "max in-flight" to "target waiting." This is a pragmatic decision — changing config names would break existing deployments and require coordination. The doc comment is updated to reflect the new semantics, which is sufficient for anyone reading the configuration.

Assumptions and Their Implications

Every design embeds assumptions, and this one is no exception:

The queue length is a reliable feedback signal: The assistant assumes that gpu_work_queue.len() accurately reflects how many partitions are waiting for GPU processing. This is true only if the queue is drained by GPU workers in FIFO order and if no partitions are being actively processed (the "on-GPU" state). The assistant implicitly treats "on-GPU" as a separate state that doesn't count toward waiting — which is correct, because those partitions are already being serviced.

Budget and workers provide sufficient backpressure: The assistant assumes that during ramp-up, the existing memory budget and worker pool limits will prevent unbounded dispatch. This is a reasonable assumption, but it creates a dependency on those subsystems being correctly tuned. If the budget is too large or the worker pool too generous, the ramp-up burst could overshoot significantly.

The target of 8 is a reasonable starting point: The user suggested 8 as a default, and the assistant accepted this without question. The summary doesn't discuss how to tune this parameter or what the tradeoffs are at different values. A lower target means less memory pressure but more risk of GPU starvation on sudden load spikes. A higher target means better GPU utilization but more memory consumed by waiting partitions.

Notifications are not missed: The assistant assumes that the Notify mechanism is reliable — that every notify_one() call from the GPU finalizer will eventually wake the dispatcher. Tokio's Notify does guarantee this (notifications are stored if no waiter is currently waiting), so this assumption is well-founded.

The Knowledge Produced

This message creates several forms of knowledge:

Explicit knowledge: The summary itself documents the architectural change, the old and new models, the expected behavioral characteristics, and the config compatibility. This serves as a design document for anyone reviewing the codebase.

Implicit knowledge about system dynamics: The description of ramp-up, steady-state, and self-regulation under imbalance provides a mental model for how the system will behave under different workload conditions. This is crucial for debugging future issues — if GPU utilization drops, the engineer now knows to check whether the waiting count is hitting the target or whether synthesis is genuinely slow.

Validation knowledge: The message confirms that the change compiles cleanly (referencing the build check in [msg 3385]), establishing that the implementation is syntactically correct and ready for deployment.

Operational knowledge: The assistant offers to "commit this, build it, or deploy it," signaling that the change is in a deployable state and that the next steps are operational rather than developmental.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in [msg 3361] is a masterclass in iterative design thinking. The assistant starts by understanding the user's critique, then works through multiple design approaches:

Initial reaction: Simple deficit check — dispatch N - waiting items when GPU finishes.
Complication: Worries about over-dispatch during ramp-up because dispatched items won't appear in the queue until synthesis completes.
In-flight counter consideration: Considers adding an atomic counter to track items being synthesized, so the deficit calculation accounts for in-flight work.
Back to basics: Realizes that the user specifically said to count waiting items only, not in-flight. The budget and worker pool provide natural limits.
Notification mechanism: Settles on Notify as the right primitive — the dispatcher sleeps when waiting >= target and wakes on GPU completion.
Edge case analysis: Walks through scenarios like multiple GPU completions during active dispatch, empty work queues, and disabled throttling. This thinking process reveals the assistant's deep understanding of concurrent systems. The concern about over-dispatch during ramp-up is legitimate — if the dispatcher checks the deficit and finds it's 8, then dispatches 8 items, those 8 items won't appear in the GPU queue until synthesis finishes. In the meantime, the dispatcher might check again and see the same deficit, dispatching another 8. The assistant's resolution — that budget and worker limits provide natural backpressure — is correct but subtle. The memory budget limits how many synthesis tasks can run concurrently, and the worker pool limits how many CPU cores are available for synthesis. These constraints prevent runaway dispatch without requiring explicit in-flight tracking.

The Broader Context

This message sits within a larger arc of GPU pipeline optimization. The preceding work (segments 22-24) focused on fixing GPU underutilization caused by slow H2D (host-to-device) PCIe transfers. The pinned memory pool eliminated those stalls, but it also changed the system dynamics — with near-zero H2D overhead, the GPU could consume work faster, potentially causing new bottlenecks. The dispatch redesign in this message addresses the scheduling side of the equation: how many synthesis tasks should be in flight to keep the GPU fed without wasting memory.

The assistant's summary in [msg 3386] is the culmination of this design iteration. It represents a shift from a push-based model (semaphore permits gating dispatch) to a pull-based model (GPU completions pulling new work into the pipeline). This is a fundamental change in control philosophy, and the message captures both the what and the why with remarkable concision.

Conclusion

Message [msg 3386] is far more than a simple "here's what I did" summary. It is a carefully crafted design document that captures a significant architectural shift in the CuZK proving engine's GPU pipeline dispatch. The assistant replaced a coarse semaphore-based throttle with a precise waiting-target dispatch mechanism, using Tokio's Notify primitive to create an event-driven feedback loop that naturally converges to the minimum synthesis concurrency needed to keep the GPU fed. The message distills hours of reasoning, multiple design iterations, and careful edge-case analysis into three concise paragraphs — a testament to the assistant's ability to communicate complex system design with clarity and precision. For anyone seeking to understand how modern GPU-accelerated proving engines balance utilization against memory pressure, this message provides both a concrete implementation pattern and a mental model for thinking about pipeline control.