The Pivot from Permits to Pipeline Depth: A Design Critique in GPU Dispatch Scheduling

In the high-stakes world of GPU-accelerated zero-knowledge proving, every millisecond of GPU idle time is lost revenue. The CuZK proving engine, a high-performance GPU-accelerated prover for Filecoin's proof-of-spacetime, had already solved its most critical bottleneck — the pinned memory pool fix that eliminated multi-second H2D transfer stalls — but a subtler problem remained: how to schedule synthesis work to keep the GPU fed without overwhelming the system's memory budget. Message [msg 3356] captures a pivotal moment where the user, acting as the system architect, identifies a fundamental design flaw in the existing dispatch mechanism and articulates a precise alternative. This single message, just a few sentences long, contains a complete control-theoretic reframing of the scheduling problem — one that would drive the next several rounds of implementation and deployment.

The Context: A System Freshly Committed

The message arrives immediately after the assistant has committed the two most significant changes of the session: the bellperson pinned memory backing ([msg 3347]) and the CuZK pinned memory pool with reactive dispatch throttle ([msg 3352]). The assistant, in [msg 3355], has just summarized the state of affairs — both commits clean, the system deployed and running on a remote machine as /data/cuzk-pinned4 with near-zero H2D overhead — and is asking for next steps. The user's response in [msg 3356] is not a simple "let's move on" but a sharp critique of the very dispatch mechanism that was just committed.

This timing is crucial. The reactive dispatch throttle, committed minutes earlier, used a semaphore-based approach: 8 "permits" that limited the total number of in-flight partitions (synthesis + GPU processing combined) to 8. The dispatcher would acquire a permit before starting synthesis, and the GPU finalizer would release a permit after prove_finish completed. On paper, this seemed reasonable — limit concurrency to prevent memory exhaustion. But the user saw deeper.

The Flaw in Permits

The user's opening line — "Didn't want 8 basic 'permits', that's actually wrong way to modulate scheduling" — is a direct and unambiguous rejection of the semaphore model. The reasoning is subtle but critical. A permit-based system limits the total number of things in flight, but it does not distinguish between synthesizing and waiting-for-GPU. These two states have very different resource profiles:

Synthesizing consumes CPU and memory (for circuit synthesis, constraint generation, witness computation)
Waiting-for-GPU consumes memory (for the synthesized partition data sitting in pinned buffers) but zero CPU A permit that treats both the same way cannot differentiate between "the GPU is busy and we have a healthy backlog" and "the GPU is idle because synthesis can't keep up." Worse, when the GPU finishes a job and releases a permit, the semaphore allows one new synthesis to start. But if the GPU just finished a job and there are zero waiting partitions, the GPU will go idle while that one synthesis runs — a classic pipeline stall. The user's insight is that the correct control variable is not the number of in-flight permits but the number of partitions waiting for the GPU. This is the queue depth — the backlog of synthesized, ready-to-prove work sitting in pinned memory, waiting for the GPU to become available. By targeting a specific queue depth, the system can ensure the GPU always has work to consume without overprovisioning synthesis capacity.

The Proposed Mechanism: Deficit-Driven Dispatch

The user's proposed mechanism is elegant in its simplicity:

"When GPU finishes a job, we should look at how many synthesis are Waiting, then if that number is greater than target (N, default 8) don't start new synths, if it's anything below the target start this many synths as we are below the target."

This is a textbook proportional controller operating on queue depth. The error signal is target_waiting - actual_waiting. If the error is positive (too few waiting), dispatch that many new syntheses. If the error is negative or zero (enough or too many waiting), do nothing. The system self-regulates: when the GPU consumes a waiting partition, the queue depth drops below target, triggering new syntheses to fill the gap. Over time, the system converges to a steady state where exactly N partitions are always waiting, and syntheses are spawned one-by-one as the GPU finishes jobs.

The user explicitly anticipates this steady state: "eventually we'll just be spawning synthesis one by one in a stable manner, running minimum number of synthesis reducing memory pressure/trashing and having good available memory budget for PCE/SRS caching." This is the key benefit — by running the minimum number of syntheses needed to keep the GPU fed, the system preserves memory for other critical operations like Pre-Compiled Constraint Evaluator (PCE) extraction and Structured Reference String (SRS) caching, which are essential for proof generation throughput.

Assumptions and Implicit Knowledge

The message makes several assumptions that reveal the user's deep understanding of the system:

Synthesis is the bottleneck that can be modulated. The user assumes that synthesis throughput can be dialed up or down by controlling how many partitions are being synthesized concurrently. This is true because synthesis is CPU-bound and embarrassingly parallel — more concurrent syntheses consume more CPU cores and memory.
GPU consumption rate is roughly constant per partition. The mechanism assumes that once a partition is waiting for the GPU, its GPU processing time is predictable enough that a fixed queue depth provides adequate buffering. If GPU times vary wildly, a fixed queue depth might cause stalls on long jobs.
The waiting count is a reliable feedback signal. The user assumes that "how many synthesis are Waiting" can be measured accurately and promptly. In practice, this requires careful instrumentation of the pipeline state — tracking which partitions have completed synthesis and are queued for GPU processing.
Memory pressure is the primary resource constraint. The emphasis on "reducing memory pressure/trashing" and preserving "good available memory budget for PCE/SRS caching" reveals that memory, not CPU, is the scarce resource. Each synthesized partition occupies pinned memory buffers, and too many concurrent syntheses cause thrashing as the pinned pool is exhausted.
The default N=8 is a starting point, not a final value. The user explicitly hedges: "8 is maybe high but let's start here." This acknowledges that the optimal queue depth depends on GPU speed, synthesis speed, partition size, and memory budget — and will need tuning.

What This Message Creates

This message is not merely a critique — it is a specification. It creates several things:

Output knowledge: A clear, actionable design for a new dispatch mechanism. The assistant, in the very next message ([msg 3357]), immediately understands the shift: "The shift is from a push model with permits to a pull model: when a GPU job finishes, I count how many partitions are waiting for GPU time, calculate the deficit against the target N, and dispatch that many new syntheses to fill the gap."

A control-theoretic framing: The message reframes the scheduling problem as a feedback control system. The controlled variable is GPU queue depth. The actuator is synthesis dispatch. The setpoint is N. This framing would prove essential as the system evolved — first to a P-controller (burst dispatch of the full deficit), then to a dampened P-controller (capping burst size), and eventually to a full PI controller with exponential moving average smoothing and synthesis throughput cap.

A vocabulary shift: The message introduces "waiting" as the key metric, replacing "permits" or "in-flight." This linguistic shift reflects a deeper conceptual shift — from limiting concurrency to maintaining a buffer.

The Thinking Process Visible in the Message

The user's message is dense with compressed reasoning. The opening "Didn't want 8 basic 'permits'" shows that the user had already considered and rejected the semaphore approach before the assistant implemented it — the assistant's commit was based on an earlier conversation or an implicit understanding that turned out to be incorrect. The phrase "that's actually wrong way to modulate scheduling" is a correction, not a discovery.

The progression of the message reveals the user's mental model: start with the goal (modulate scheduling), identify the right control variable (waiting count), specify the control law (dispatch deficit on GPU completion), and enumerate the benefits (stable GPU use, minimum memory pressure, preserved budget). This is a complete design argument in four sentences.

The hedging — "well N, 8 is maybe high but let's start here" — shows awareness of the empirical nature of the tuning problem. The user knows the theory is sound but the constants need real-world validation.

Impact on the Subsequent Conversation

This message set the agenda for the next several rounds of the conversation. The assistant implemented the P-controller (burst dispatch of the full deficit) in the following messages, deployed it as cuzk-pctrl1, observed it was too aggressive, then added dampening in cuzk-pctrl2. When that proved unstable due to the deep synthesis pipeline making the raw waiting count a noisy signal, the conversation evolved toward a PI controller with EMA smoothing — but the core insight from this message remained: target queue depth, not in-flight permits.

The user's message at [msg 3356] is a masterclass in concise, precise technical communication. It identifies a design flaw, proposes a correct alternative, specifies the control law, and justifies the benefits — all in under 200 words. For the engineer implementing the system, it provides everything needed to understand what to build and why.