The P-Controller Epiphany: How a User's Critique Reshaped GPU Dispatch Scheduling

The Message

In the middle of an intense coding session focused on GPU pipeline optimization for the CuZK proving engine, the user sent a brief but incisive critique:

Seems implementatino is.. imperfect? There are 6 synthesis Running, zero Waiting, and after GPU consumes one only one starts when we should start another N.. We will overshoot, but we must overshoot, to then undershoot little less hopefully and so on. Essentially like a Proportional (not quite PI/PID) controller

This message, appearing as <msg id=3387> in the conversation, is deceptively short. In just a few lines, the user identifies a fundamental design flaw in the assistant's freshly implemented dispatch mechanism, diagnoses the root cause, and prescribes the correct solution framework — all while introducing control theory vocabulary that would define the subsequent hour of development.

The Context: A Dispatch System in Need of Control

To understand why this message was written, we must examine what preceded it. The assistant had just completed a significant refactor of the GPU pipeline dispatch mechanism in engine.rs (see <msg id=3386>). The original system used a Semaphore-based throttle: the dispatcher would acquire a permit before starting synthesis for a partition, and the GPU finalizer would return the permit after GPU processing completed. This limited the total number of in-flight partitions (synthesis + GPU) to a fixed value N, but it did not distinguish between partitions actively being synthesized and those already waiting in the GPU queue.

The user had previously critiqued this semaphore model, arguing that what mattered was maintaining a target number of synthesized partitions waiting in the GPU queue — not limiting total in-flight work. The assistant responded by replacing the semaphore with a Notify-based mechanism: the dispatcher would check gpu_work_queue.len() against a target, and if below target, dispatch more work. If at or above target, it would sleep until a GPU completion notification arrived.

The assistant described the expected behavior in <msg id=3386>:

- Ramp-up: waiting=0, deficit=N, dispatcher rapidly dispatches N items (budget/workers are the natural backpressure) - Steady state: GPU finishes one → waiting drops to N-1 → dispatcher dispatches one → converges to 1:1

But this description contained a critical flaw — one the user immediately spotted.

The Flaw: One-at-a-Time Dispatch

The user's observation was precise: with six synthesis jobs running and zero partitions waiting in the GPU queue, when the GPU finished consuming one partition, only one new synthesis started. The user correctly identified that this was insufficient. If the target queue depth was, say, 8, and the current waiting count was 0, the deficit was 8 — meaning the system should start 8 new syntheses to fill the gap. But it only started one.

The user's phrase "we must overshoot" captures the key insight. If the dispatcher only starts one synthesis per GPU completion, it can never catch up to the target. The waiting queue will remain perpetually empty because each new synthesis takes time to complete, and by the time it finishes, the GPU has already consumed it. The system would stabilize at a state where the GPU is constantly starved — exactly the problem the dispatch redesign was supposed to solve.

The user recognized that the correct behavior is to overshoot: start more syntheses than the GPU can immediately consume, so that a backlog builds up in the waiting queue. Once the backlog exceeds the target, the dispatcher pauses. As the GPU drains the backlog below the target, the dispatcher starts a smaller batch. The system converges through successive overshoots and undershoots — exactly like a proportional controller.

The Control Theory Connection

The user's reference to a "Proportional (not quite PI/PID) controller" is the conceptual linchpin of the message. In control theory, a proportional controller computes an error signal (the difference between the desired setpoint and the current measurement) and applies a correction proportional to that error. The user identified that:

Setpoint: The target number of partitions waiting in the GPU queue
Measurement: The current gpu_work_queue.len()
Error: deficit = target - waiting
Correction: Start deficit new syntheses This is precisely a P-controller with a gain of 1. The "overshoot then undershoot less" behavior the user describes is the characteristic convergence of a proportional control system — it will oscillate around the setpoint, with diminishing amplitude, until it settles at a steady state where the dispatch rate matches the GPU consumption rate. The parenthetical "(not quite PI/PID)" is also significant. The user is acknowledging that a pure proportional controller has limitations — it may exhibit steady-state error or oscillation — but it is the correct first step. The "I" (integral) and "D" (derivative) terms of a full PID controller could be added later to eliminate steady-state error and dampen oscillations. But the immediate problem was that the system wasn't even proportional — it was effectively a 1:1 dispatcher that never filled the pipeline.

Why the Assistant's Implementation Failed

The assistant's reasoning in <msg id=3388> reveals a deep trace through the code to understand why only one dispatch occurred per GPU completion. The analysis uncovered multiple interacting constraints:

Budget bottleneck: Each synthesis job requires approximately 9 GiB of memory. With six syntheses running, roughly 54 GiB was already allocated. When the GPU finished a job, the finalizer released only the remaining reservation (~1.2 GiB after a/b/c buffers were already freed during prove_start). This freed budget was only enough for one new dispatch.
Loop structure: The dispatcher loop dispatched one item per iteration. After dispatching, it looped back to check the deficit again — but by then, budget was exhausted, so it blocked on budget.acquire(). The dispatcher was effectively serialized: one dispatch per budget availability event.
Missing batch semantics: The assistant's implementation never calculated the deficit and dispatched that many items as a batch. It simply looped, dispatching one at a time, with budget acting as the natural gate. The user recognized that this made the system behave like the old semaphore model — just with a different signaling mechanism. The assistant's own post-mortem in the reasoning trace is illuminating:

"The real issue is that we're acquiring budget sequentially for each dispatch, creating a bottleneck. Instead, when a GPU completion arrives, we should calculate how many items we can dispatch based on the freed budget, pop that many from the queue, and dispatch them all at once rather than one at a time."

The Assumptions Embedded in the Message

The user's message makes several implicit assumptions that reveal their mental model:

Assumption 1: The dispatcher can start multiple syntheses concurrently. The user assumes that if the deficit is 8, the system can dispatch 8 synthesis jobs in parallel. This is not trivially true — it depends on available memory budget, worker thread capacity, and the synthesis work queue having enough items. The user is implicitly asserting that these constraints should not prevent batching; if budget is insufficient, the dispatcher should dispatch as many as possible and then wait for more budget, rather than dispatching one and then blocking.

Assumption 2: Overshoot is acceptable. The user explicitly states that overshoot is necessary. This is a design choice: it prioritizes GPU utilization (keeping the GPU fed) over memory efficiency (minimizing the number of in-flight partitions). The user is willing to temporarily have 14+ partitions in flight (6 running + 8 dispatched) if it means the GPU never stalls.

Assumption 3: The system will converge. The user believes that the overshoot-undershoot dynamics will naturally converge to a steady state where dispatch rate matches GPU consumption rate. This assumes that the synthesis pipeline has enough work to sustain the dispatch rate, and that the GPU consumption rate is relatively stable.

Assumption 4: The assistant understands control theory. The user drops the P/PI/PID reference without explanation, expecting the assistant to immediately grasp the analogy. This assumption proved correct — the assistant's subsequent reasoning and implementation explicitly adopted the P-controller framework.

Input Knowledge Required

To fully understand this message, one needs:

Knowledge of the dispatch architecture: The message references "6 synthesis Running" and "zero Waiting" — these refer to the two stages of the GPU pipeline: synthesis (CPU-bound constraint generation) and GPU proving (GPU-bound computation). Understanding that gpu_work_queue.len() measures only partitions that have completed synthesis and are waiting for GPU processing is essential.
Knowledge of the memory budget system: Each synthesis job consumes memory budget (~9 GiB per partition). The budget is acquired at dispatch time and released in stages (a/b/c buffers during prove_start, remaining reservation at finalization). The interaction between budget and dispatch is the core bottleneck.
Knowledge of the previous implementation: The message builds on the assistant's description in <msg id=3386> of the Notify-based dispatch. Without that context, the user's critique would seem cryptic.
Knowledge of control theory: The P/PI/PID reference is central to the message's argument. Understanding proportional control — error signal, correction proportional to error, convergence through overshoot — is necessary to grasp why the user considers the current behavior wrong and what the fix should look like.

Output Knowledge Created

This message catalyzed a cascade of insights and actions:

Identification of the bug: The assistant realized that the dispatcher was effectively 1:1 despite the new architecture. The deficit calculation was correct, but the loop structure prevented batching.
Redesign of the dispatch loop: The assistant restructured the dispatcher into a two-phase loop: (a) wait for GPU event, (b) calculate deficit and dispatch that many items in a burst. This was implemented in <msg id=3390> and compiled cleanly in <msg id=3391>.
Introduction of control theory vocabulary: The P-controller framing became the organizing principle for subsequent iterations. The assistant would later implement dampened P-control (capping burst size) and eventually a full PI controller with EMA smoothing and synthesis throughput caps.
Deployment: The user immediately requested deployment (<msg id=3393>: "deploy to the machine"), indicating confidence that the P-controller fix was correct and ready for testing.

The Thinking Process Revealed

The user's message reveals a sophisticated debugging methodology. Rather than examining code, the user is examining system behavior — the observable counts of running syntheses and waiting partitions. This is a form of black-box analysis: the user doesn't need to trace through the dispatcher loop to know something is wrong; they can infer it from the system's response to a GPU completion event.

The progression of the user's thought is visible in the message structure:

Observation: "6 synthesis Running, zero Waiting" — a snapshot of the system state.
Event: "after GPU consumes one" — a stimulus is applied.
Response: "only one starts" — the system's reaction is measured.
Expected response: "we should start another N" — the user has a mental model of correct behavior.
Gap analysis: The actual response (1) differs from the expected response (N).
Root cause: The system is not operating as a proportional controller.
Prescription: The system must overshoot to converge. This is textbook debugging: observe, stimulate, measure, compare to expectation, diagnose, prescribe. The user does all of this in three sentences. The parenthetical "(not quite PI/PID)" is particularly revealing. It shows the user is already thinking ahead — they know that a pure P-controller may need refinement, but they're deliberately keeping the scope narrow. The message says "fix this first, we can add I and D terms later." This forward-looking perspective would prove crucial, as the team would indeed iterate through P-control, dampened P-control, and finally PI control with EMA feed-forward over the subsequent chunks.

Conclusion

Message <msg id=3387> is a masterclass in concise technical critique. In 43 words, the user identifies a design flaw, explains why it's wrong, describes the correct behavior, and provides the theoretical framework for the fix. The message succeeds because it operates at the right level of abstraction — it doesn't get lost in the details of budget acquisition or loop structure, but instead focuses on the observable system behavior and the control theory principles that should govern it.

The impact of this message rippled through the entire session. It transformed the dispatch mechanism from a reactive, one-at-a-time system into a proactive, batch-oriented proportional controller. It introduced control theory as the organizing framework for GPU pipeline scheduling. And it set the stage for the sophisticated PI-controlled pacer with synthesis throughput caps that would follow in subsequent chunks.

For anyone studying how effective technical communication drives engineering decisions, this message is a textbook example: observe precisely, critique constructively, and prescribe the conceptual framework rather than the implementation details.