From P-Controller to PI Pacer: Diagnosing Pipeline Instability in GPU Dispatch Scheduling

The Message

"Still not stable, this is a pretty deep pipeline. The Waiting count is still the best signal we have from scheduling theory, but the input is too coarse. Consider a 'pacer' approach with PI control on a short EMA-of-Waiting or gpu consumption rate or combined signal"

This single message from a user to an AI assistant marks a critical inflection point in an extended debugging session. The team had been iteratively refining a GPU dispatch scheduling system for the cuzk zero-knowledge proving engine, and had just deployed a dampened proportional controller (P-controller) to regulate how many synthesis jobs were dispatched to the GPU pipeline. The dampened controller was the third attempt at solving the problem, and it too had failed to achieve stable operation. This message is not merely a complaint — it is a precise diagnostic statement that identifies the root cause of instability and prescribes a more sophisticated control strategy drawn from classical control theory.

The Context: Three Iterations of a Scheduling Problem

To understand why this message was written, one must understand the problem it addresses. The cuzk proving engine processes zero-knowledge proofs through a multi-stage pipeline: synthesis (CPU-bound work that generates circuit constraints) followed by GPU proving (GPU-bound work that computes the actual proof). Between these stages sits a queue of "waiting" items — synthesized partitions that have been completed by CPU workers and are ready for GPU processing. The scheduling challenge is to keep the GPU fed with work without overwhelming the system with too many concurrent synthesis jobs, which would cause CPU contention and degrade overall throughput.

The team had already tried three approaches. The first was a semaphore-based reactive dispatch model that limited total in-flight partitions. This failed because it didn't target a specific queue depth of synthesized partitions waiting for the GPU — it was too coarse and didn't maintain a stable pipeline. The second approach was a P-controller that replaced the semaphore with a Notify-based two-phase loop: wait for a GPU completion event, then dispatch the full deficit in a burst to intentionally overshoot and converge on a steady state. This was deployed as cuzk-pctrl1 but proved too aggressive, instantly filling all allocation slots. The third approach was a dampened P-controller (cuzk-pctrl2) that capped the burst size at max(1, min(3, deficit * 0.75)), limiting the expansion rate per GPU event to at most three dispatches.

The user's message comes immediately after the deployment of cuzk-pctrl2. The dampening helped but did not solve the underlying instability.

The Diagnostic Insight: Pipeline Depth and Signal Coarseness

The user's message contains two crucial observations. First: "this is a pretty deep pipeline." This is the root cause of the instability. In a deep pipeline, there is a significant delay between when a synthesis job is dispatched and when it completes and appears in the "waiting" count. During that delay, the dispatcher is operating on stale information — it sees a low waiting count and dispatches more work, but those earlier dispatches haven't landed yet. By the time they do land, the system has already overshot. This is a classic problem in control theory: delay in the feedback loop causes oscillation.

Second: "The Waiting count is still the best signal we have from scheduling theory, but the input is too coarse." The user is making a nuanced distinction between the signal (waiting count) and the quality of that signal. The waiting count is conceptually the right metric — it directly measures how much work is queued for the GPU. But in practice, the raw instantaneous value is too noisy and too delayed to serve as a stable feedback signal for a simple proportional controller. A P-controller reacts to the current error (deficit = target - waiting), but when the error signal itself is a lagging indicator, the controller inevitably overcorrects.

The Proposed Solution: PI Control on a Smoothed Signal

The user proposes a "pacer" approach with PI control (Proportional-Integral control) operating on a smoothed input signal. This is a sophisticated insight that draws on classical control theory. A PI controller has two terms: the proportional term reacts to the current error (like the existing P-controller), while the integral term accumulates error over time. The integral term is crucial for a system with delay: even if the instantaneous error is small (because the waiting count hasn't updated yet), the integral of past errors provides a memory of the system state that prevents oscillation.

The user suggests three options for the smoothed input signal:

1. A short EMA of the Waiting count — Exponential Moving Average filters out high-frequency noise from the raw waiting count, providing a smoother feedback signal that better represents the underlying trend.
2. GPU consumption rate — Instead of measuring queue depth, measure the rate at which the GPU consumes work. This is a derivative signal that can provide earlier indication of imbalance.
3. A combined signal — Some weighted combination of queue depth and consumption rate, which would provide both position and velocity information. The phrase "pacer" is evocative: rather than reacting to events (GPU completions triggering bursts of dispatches), a pacer would work like a metronome, dispatching work at a computed interval that matches the system's sustainable throughput. This is a fundamental shift from reactive to predictive scheduling.

Assumptions and Knowledge Boundaries

The message makes several implicit assumptions. It assumes that the pipeline depth is a structural property that cannot be easily reduced — that the synthesis time is inherently long and variable. It assumes that the waiting count, despite its coarseness, remains the best available signal because it directly reflects the state the system cares about (how much work is ready for the GPU). It assumes that a PI controller is the right tool for the job, which in turn assumes that the system dynamics are approximately linear and time-invariant enough for classical control to work.

There is also an assumption that the EMA window should be "short" — long enough to filter noise but short enough to remain responsive. The user does not specify the exact time constant, leaving that as a tuning parameter to be discovered empirically.

The message does not explicitly address the possibility of using feed-forward control (predicting future GPU demand based on incoming work characteristics) or adaptive control (dynamically adjusting controller parameters based on observed performance). These are more advanced techniques that might be needed if the PI approach also proves insufficient.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. First, the architecture of the cuzk proving pipeline: the two-stage synthesis-then-prove model, the role of the waiting queue, and the budget system that constrains concurrent allocations. Second, the history of the three previous dispatch strategies and why each failed. Third, basic control theory: the difference between P and PI control, the concept of integral windup, the role of signal smoothing via moving averages, and the problem of delay in feedback loops. Fourth, the practical context of GPU programming: the cost of H2D transfers, the pinned memory pool that was the foundation of the earlier fix, and the performance characteristics of GPU kernel execution.

Output Knowledge Created

This message creates a new conceptual framework for the scheduling problem. It reframes the issue from "how many items to dispatch per event" (a discrete, event-driven question) to "what is the sustainable dispatch rate" (a continuous, time-domain question). It introduces the language of control theory — PI control, EMA, smoothing, pacer — as the vocabulary for discussing the solution. It also establishes a clear diagnosis: the pipeline depth is the root cause of instability, not the controller gain or the dampening factor.

The message also implicitly defines the next iteration of work: implement a DispatchPacer struct that computes a dispatch interval based on a PI correction of a smoothed error signal, with an EMA filter on the input. This is precisely what the assistant went on to implement in the following chunk, creating a PI-controlled pacer with a synthesis throughput cap and anti-windup protection.

The Thinking Process

The user's reasoning in this message reveals a sophisticated mental model of the system. They have moved beyond treating the scheduling problem as a simple counting exercise (dispatch N items per event) and are now thinking in terms of continuous-time control. The phrase "the input is too coarse" shows they understand that the waiting count is a sampled, delayed measurement of a continuous process — and that the sampling noise and delay are what's causing the instability.

The suggestion of EMA smoothing shows an understanding of signal processing: the raw waiting count contains high-frequency noise (individual synthesis completions and GPU completions) that obscures the low-frequency trend. An EMA acts as a low-pass filter, extracting the trend while suppressing the noise.

The mention of "GPU consumption rate" as an alternative signal shows an understanding of duality in measurement: instead of measuring the stock (queue depth), one could measure the flow (consumption rate). In control theory, this is the difference between position and velocity feedback. A combined signal would give both, like a PID controller with derivative term.

The user's decision to propose rather than dictate — "Consider a 'pacer' approach" — leaves room for the assistant to exercise judgment in implementation. This is characteristic of effective human-AI collaboration: the human provides the conceptual breakthrough and the AI handles the implementation details.

Conclusion

This message is a turning point in the GPU dispatch scheduling saga. It diagnoses the fundamental problem (pipeline depth causing delayed feedback), identifies the right class of solution (PI control on smoothed signals), and sets the direction for the next iteration of work. The message demonstrates that effective debugging is not just about fixing bugs but about developing increasingly accurate mental models of system dynamics. The user's ability to abstract from the specific failure of the dampened P-controller to the general principle of feedback delay in deep pipelines is what makes this message valuable — and what makes the resulting PI-controlled pacer a more robust solution than any of the three attempts that preceded it.