The P-Controller Dispatch: A Pivotal Moment in GPU Pipeline Scheduling

Introduction

In the complex world of high-performance GPU proving for zero-knowledge proofs, the scheduling of synthesis work onto GPU resources is a critical control problem. Message [msg 3392] captures a pivotal moment in the iterative refinement of such a scheduler: the implementation of a proportional controller (P-controller) to replace a flawed one-at-a-time dispatch loop. This message, written by the AI assistant after a code edit compiled cleanly, summarizes the behavioral change that would shape the next several rounds of development. It is a deceptively short message — just a few paragraphs — but it encodes a fundamental shift in how the system thinks about dispatching work to the GPU, and it reveals the assumptions, design reasoning, and control-theoretic intuition that drove the change.

The Message

The subject message reads in full:

Compiles clean. The key behavioral change:

Before: Dispatcher loops continuously, dispatching one item per iteration, gated by budget. Result: ~1 dispatch per GPU completion (budget is the bottleneck).

After: Dispatcher works in cycles: 1. Wait for GPU event (or startup) 2. deficit = target - gpu_work_queue.len() 3. Dispatch deficit items in a burst (budget.acquire blocks naturally, pacing the burst) 4. After full burst, go back to waiting for next GPU event — don't keep dispatching

On startup, deficit = N = 8, so we burst 8 syntheses. When GPU finishes one and the burst of 8 hasn't landed yet (0 waiting), deficit = 8 again, we fire another 8. This overshoots. Eventually waiting climbs above target, deficit = 0, we skip. Then as GPU drains below target, we dispatch the small deficit. Converges to 1:1 steady state.

This message was written immediately after the assistant edited engine.rs to implement the P-controller dispatcher and confirmed it compiled with cargo check. It is addressed to the user, explaining what changed and why.

Why This Message Was Written: The Reasoning and Motivation

To understand why this message exists, we must understand the problem that preceded it. The session had been iterating on a GPU pipeline scheduler for the CuZK proving engine. The pipeline works as follows: CPU-bound synthesis workers produce "partitions" (units of proof computation), which are then dispatched to a GPU work queue. The GPU processes these partitions, and a finalizer releases resources when done. The goal is to keep the GPU fed with work while not overwhelming the system with too many concurrent synthesis jobs, which compete for memory.

The previous implementation used a semaphore-based dispatch model ([msg 3386]), which the assistant had replaced with a Notify-based waiting-target dispatch. In that model, the dispatcher would check how many partitions were waiting in the GPU queue, compare against a target (configurable, default 8), and dispatch one item if below target. However, the user observed a critical flaw ([msg 3387]): when the GPU finished one job, only one new synthesis started, even though the deficit was 8. The dispatcher was effectively still operating at 1:1 — one dispatch per GPU completion — because budget acquisition acted as a bottleneck within the tight dispatch loop.

The user's diagnosis was precise: "The bottleneck is we don't start enough synthesis" ([msg 3389]). They wanted a proportional controller — dispatch N items per GPU event, overshoot intentionally, and let the system converge. The assistant's reasoning in [msg 3388] shows the deep analysis that led to this conclusion. The assistant traced through the loop logic, identified that budget acquisition was the bottleneck, and realized that the dispatcher never "batched" based on deficit — it just looped, and budget was the bottleneck. The insight was that the dispatcher needed to calculate the deficit, dispatch that many items in a burst, and then stop and wait for the next GPU event, rather than continuously trying to dispatch one-at-a-time.

Message [msg 3392] is the culmination of that reasoning. It is the assistant's summary of the fix, written to communicate the design to the user and to confirm that the code compiles. It is a bridge between the implementation and the deployment that would follow ([msg 3393]: "deploy to the machine").

How Decisions Were Made: The P-Controller Design

The design described in this message represents several key decisions:

Decision 1: Cycle-based dispatch instead of continuous looping. The old model had the dispatcher continuously looping, checking the deficit, dispatching one item, and looping again. The new model has two distinct phases: wait for a GPU event, then dispatch the full deficit in a burst. After the burst, return to waiting. This is a fundamental structural change — it transforms the dispatcher from a greedy loop into an event-driven state machine.

Decision 2: Deficit as the control signal. The error term is deficit = target - gpu_work_queue.len(). This is the proportional term in the P-controller. The gain is implicitly 1 — dispatch exactly deficit items. The assistant recognized this as a P-controller: "the P in PID stands for Proportional — our deficit acts as the error signal, and we dispatch syntheses proportional to it with a gain of 1" ([msg 3388]).

Decision 3: Accept overshoot as intentional. The assistant explicitly acknowledges that the system will overshoot: "On startup, deficit = N = 8, so we burst 8 syntheses. When GPU finishes one and the burst of 8 hasn't landed yet (0 waiting), deficit = 8 again, we fire another 8. This overshoots." This is not a bug — it is the intended behavior of a proportional controller. The overshoot causes the waiting queue to climb above the target, at which point deficit becomes zero and the dispatcher skips. As the GPU drains the queue below target, the deficit becomes small, and the system converges to a steady state where one dispatch occurs per GPU completion.

Decision 4: Budget as natural backpressure within the burst. The assistant notes that budget.acquire blocks naturally, pacing the burst. This means that if budget is exhausted partway through the burst, the dispatcher blocks on acquisition, and GPU completions that free budget will unblock it. This creates a self-regulating loop within the burst phase.

Decision 5: The Notify mechanism for GPU events. The dispatcher uses a Notify primitive to wait for GPU completion events. The assistant chose this over the previous semaphore because it allows the dispatcher to sleep efficiently when the queue is at or above target, rather than continuously polling.

Assumptions Made

The design in this message rests on several assumptions, some of which would prove problematic in subsequent iterations:

Assumption 1: The waiting count is a good feedback signal. The P-controller uses gpu_work_queue.len() as its sole measurement. This assumes that the number of partitions waiting for the GPU is a reliable indicator of whether the pipeline is balanced. In practice, the deep synthesis pipeline made this a noisy and delayed signal — partitions take time to synthesize, so the waiting count doesn't reflect syntheses currently in progress.

Assumption 2: Budget will eventually free up to dispatch the deficit. The burst dispatches deficit items, but if budget is exhausted, it blocks on budget.acquire. The assumption is that GPU completions will free enough budget to allow the burst to complete. However, if the budget is deeply constrained (e.g., 6 syntheses already running consuming ~54 GiB), only small amounts of budget may be freed per GPU completion, limiting the burst size.

Assumption 3: The system will converge to steady state. The assistant predicts that the system will converge to a 1:1 steady state after initial overshoot. This assumes that the dynamics of synthesis and GPU processing are stable and that the proportional controller with gain 1 is not too aggressive.

Assumption 4: The Notify mechanism won't miss signals during the burst. During the burst phase, the dispatcher is not listening for GPU events (it's busy dispatching). If multiple GPU completions occur during this time, only one notification is stored (Notify stores at most one). This could cause the dispatcher to miss signals and delay its response to changing conditions.

What Went Wrong: Later Revealed Issues

The subsequent development revealed several problems with this P-controller design. The first deployment (cuzk-pctrl1) proved too aggressive, instantly filling all allocation slots ([chunk 25.0]). The user requested a dampening factor, capping the burst size at max(1, min(3, deficit * 0.75)) to limit the expansion rate per GPU event. This became cuzk-pctrl2.

However, the system remained unstable due to the deep synthesis pipeline, which made the raw waiting count a noisy and delayed feedback signal. This led to the development of a PI-controlled dispatch pacer with an exponential moving average (EMA) of the GPU inter-completion interval as a feed-forward rate ([chunk 25.1]). Later, a synthesis throughput cap was added to prevent the pacer from flooding the system with concurrent synthesis jobs when synthesis was compute-constrained.

These subsequent refinements do not invalidate the P-controller design — they represent the natural evolution of a control system as more is learned about the system dynamics. The P-controller was a necessary step that revealed the need for damping, smoothing, and feed-forward terms.

Input Knowledge Required

To fully understand this message, one needs:

Knowledge of the GPU proving pipeline: Synthesis workers produce partitions, which go into a GPU work queue. The GPU processes them, and a finalizer releases resources. The dispatcher sits between synthesis and GPU, deciding when to push work to the GPU queue.
The budget system: A memory budget limits how many concurrent syntheses can run. Each partition requires ~9 GiB of memory. The budget is acquired at dispatch time and released when synthesis completes (or when the GPU finalizer drops the reservation).
The previous semaphore-based dispatch: The system previously used a Semaphore(N) to limit total in-flight work. This was replaced with a Notify-based waiting-target model before the P-controller was implemented.
Control theory basics: Understanding of proportional control, error signals, overshoot, and convergence is necessary to appreciate the design.
The Rust async/Notify primitive: The tokio::sync::Notify mechanism allows one task to notify another that an event has occurred. It stores at most one notification.

Output Knowledge Created

This message creates several pieces of knowledge:

The P-controller dispatch algorithm: A clear, documented algorithm for dispatching synthesis work in bursts based on deficit. This algorithm is the foundation for all subsequent refinements.
The convergence model: An understanding of how the system will behave under the new algorithm — initial overshoot, then convergence to steady state.
The design rationale: An explanation of why the old model was broken (1:1 dispatch per GPU completion) and how the new model fixes it (burst dispatch with wait-for-event cycles).
A shared mental model: The message creates a shared understanding between the user and assistant about how the dispatcher should work, enabling further collaboration on refinements.

The Thinking Process Visible in the Reasoning

While the subject message itself is a concise summary, the reasoning that produced it is visible in the preceding messages. In [msg 3388], the assistant engages in an extensive chain of reasoning:

Problem identification: The assistant traces through the current dispatch loop and identifies that budget acquisition is the bottleneck, causing 1:1 dispatch per GPU completion.
Hypothesis formation: The assistant considers multiple hypotheses — is it the channel capacity? The budget constraint? The Notify mechanism? The loop structure?
Design exploration: The assistant explores several design alternatives — dispatching the full deficit at once, using a select statement to wake on either GPU completion or new work, handling the startup case separately, dealing with edge cases like empty work queues.
Control theory framing: The assistant explicitly recognizes the problem as a control theory problem: "the P in PID stands for Proportional — our deficit acts as the error signal, and we dispatch syntheses proportional to it with a gain of 1, which is exactly what a P-controller does."
Mental simulation: The assistant simulates the startup sequence: "Initially there's nothing waiting, so the deficit is 8, and we dispatch that first batch. Once the GPU finishes its first item, we check the deficit again — if all 8 syntheses completed and only 1 was consumed, we'd have 7 waiting and dispatch just 1 more."
Edge case analysis: The assistant identifies potential deadlocks (what if the initial fill finds no work? what if the synthesis queue empties mid-burst?) and designs mitigations. This reasoning process is characteristic of complex systems engineering: iterative refinement, mental simulation, and the application of formal theory (control theory) to a practical scheduling problem.

Conclusion

Message [msg 3392] is a small but dense artifact of a larger engineering process. It captures the moment when a team — a human user and an AI assistant — realized that their GPU pipeline scheduler needed to be reframed as a control problem, and implemented a proportional controller to solve it. The message is remarkable for its clarity and concision: in a few paragraphs, it explains the old model's failure, the new model's design, the expected behavior, and the convergence properties.

The P-controller described in this message would not be the final solution — it would be followed by dampening, PI control with EMA feed-forward, and a synthesis throughput cap. But it was the essential first step that established the control-theoretic framework for all subsequent work. Without this message, the later refinements would have had no foundation. It is a testament to the power of iterative engineering: start with a simple model, observe its behavior, and refine.