Taming Latency: Designing a PI-Controlled Dispatch Pacer for a 60-Second GPU Pipeline

Introduction

In the world of high-performance computing, few challenges are as deceptive as controlling a pipeline with a 60-second feedback delay. When a single synthesis operation takes 20 to 60 seconds to complete before its results reach the GPU, any naive control scheme—dispatch a batch, wait for it to finish, dispatch another—guarantees either crippling underutilization or catastrophic overshoot. This is the precise problem addressed in message <msg id=3429> of an ongoing coding session, where an AI assistant designs and implements a PI-controlled dispatch pacer for a GPU proving pipeline in the CuZK zero-knowledge proving engine.

The message is a turning point in a longer arc of iterative refinement. Earlier sessions had already resolved the root cause of GPU underutilization—a costly host-to-device (H2D) memory transfer bottleneck—by implementing a zero-copy pinned memory pool. But even with near-zero transfer times, the pipeline still suffered from unstable scheduling. A semaphore-based dispatch model had failed to maintain a steady queue depth. A P-controller (proportional-only) had proven too aggressive, instantly filling all allocation slots. A dampened P-controller had improved behavior but remained unstable due to the deep synthesis pipeline introducing a long, noisy feedback delay. Now, in this message, the assistant takes the decisive step of implementing a full PI (proportional-integral) controller with an exponential moving average (EMA) feed-forward, a bootstrap phase, and—in a later refinement—a synthesis throughput cap with anti-windup.

This article examines message <msg id=3429> in depth: the reasoning that drove its design, the control-theoretic assumptions it makes, the knowledge it consumes and produces, and the thinking process that shaped its implementation. It is a case study in applying classical control theory to a modern distributed computing problem, and a reminder that even the most sophisticated algorithms must be grounded in an accurate understanding of system dynamics.

The Context: Why This Message Was Written

To understand why <msg id=3429> exists, one must understand what came before it. The session's broader goal was to maximize GPU utilization in a zero-knowledge proving pipeline. The pipeline has two main stages: synthesis (CPU-bound, 20–60 seconds per partition) and GPU proving (GPU-bound, ~1 second per partition). The GPU can process partitions much faster than the CPU can produce them, so to keep the GPU fed, multiple synthesis jobs must run concurrently. The challenge is to dispatch synthesis jobs at exactly the right rate—fast enough to keep the GPU busy, but not so fast that the system is flooded with in-flight synthesis work, causing CPU contention and memory pressure.

Earlier in the session, the team had deployed a pinned memory pool that eliminated H2D transfer overhead, but the dispatch scheduling remained problematic. The first attempt was a semaphore that limited total in-flight partitions. The user criticized this approach because it limited total in-flight work rather than targeting a specific number of synthesized partitions waiting for the GPU. The assistant then implemented a P-controller: on each GPU completion event, it would dispatch the full deficit in a burst, intentionally overshooting to converge on a steady state.

The P-controller failed. Its first deployment (cuzk-pctrl1) instantly filled all allocation slots. A dampened version (cuzk-pctrl2) capped burst sizes but remained unstable. The root cause was the long feedback delay: a dispatch decision takes 20–60 seconds to manifest as a change in the queue depth that the controller observes. With a pure proportional controller, this delay causes oscillation—the controller overcorrects, waits, sees no effect, overcorrects again, and by the time the corrections arrive, the system is wildly out of balance.

The user then proposed a PI controller operating on a smoothed signal. This brings us to <msg id=3429>. The user's preceding message (index 3428) had noted: "the system needs quite some time to stabilize from under/overshoot, multiple minutes (single synth is 20-60s depending on contention so from dispatch to gpu)." The assistant's response—message 3429—is a direct reaction to this observation. It internalizes the 20–60 second delay as the central design constraint and builds a control system around it.

The Thinking Process: Control Theory Meets Systems Engineering

The message opens with the assistant's reasoning, which is essentially a control theory lecture applied to a real system:

Good context — single synthesis takes 20-60s, so the pipeline delay is very long. This means: 1. The PI controller needs to be very conservative (low gains) because the feedback delay is 20-60s 2. The EMA smoothing needs to be slow (low alpha) to filter out noise over these long timescales 3. The integral term is critical for eliminating steady-state error, but must wind up slowly 4. Bootstrap needs to dispatch target items, then wait for GPU data before pacing

This reasoning reveals a deep understanding of control theory. The assistant recognizes that a long delay in a feedback loop imposes severe constraints on controller gains. In classical control theory, the phase lag introduced by a delay reduces the stability margin; high gains cause oscillation or instability. The standard remedy is to reduce the proportional gain (Kp) and use the integral term (Ki) to eliminate steady-state error over many time constants.

The assistant then performs a quantitative analysis:

With 20-60s synthesis time and ~1s GPU time: - Steady-state needs ~20-60 concurrent syntheses to keep GPU fed - But I want only target=8 waiting at any time, so I need to dispatch at GPU rate (1/s) in steady state

This is a critical insight. The assistant distinguishes between concurrent syntheses in flight (which can be 20–60) and synthesized partitions waiting for the GPU (target=8). The dispatch rate in steady state must match the GPU consumption rate (~1 per second), regardless of how many syntheses are running. The waiting count is the controlled variable; the dispatch interval is the control input.

The assistant then considers the control-theoretic implications:

The PI gains need to account for the 20-60s delay — with a 30s delay, Kp should be around 0.01-0.1 and Ki even smaller. Actually, I should think of this as a control system where the dispatch rate is my input, the waiting count is my output, and the synthesis time creates the delay. The GPU consumption is what I'm ultimately trying to regulate.

Here, the assistant briefly considers a more sophisticated formulation (regulating GPU consumption directly) but then pulls back:

Actually, I'm overthinking the control theory here. What the user really needs is straightforward: a pacer that maintains a steady dispatch rate, PI control that gently adjusts it, and an EMA filter to smooth out noise in the waiting signal.

This self-correction is revealing. The assistant recognizes that the user needs a practical, deployable solution, not a PhD thesis in control theory. The design settles on a feed-forward plus PI correction architecture: the feed-forward term matches the GPU rate (derived from an EMA of GPU inter-completion intervals), and the PI term makes small adjustments based on the smoothed error in the waiting queue depth.

Parameter Selection: The Art of Choosing Gains

The assistant selects specific parameters for the controller:

I'm setting Kp to 0.1 for gentle proportional response, Ki to 0.01 for gentle integral correction, alpha_waiting to 0.2 for slow EMA smoothing of the queue depth, and alpha_gpu to 0.3 for moderate EMA on the GPU rate estimate.

These choices deserve scrutiny. Kp=0.1 means that a deficit of 1 waiting partition adjusts the dispatch rate multiplier by 0.1 (i.e., 10% faster or slower). With a target of 8, a complete drain of the queue (error = -8) would increase the dispatch rate by 80%—aggressive but not insane. Ki=0.01 means the integral term accumulates slowly, at 1% of the error per iteration, which prevents windup and provides gentle long-term correction.

The EMA parameters reflect the different dynamics of the two signals. The waiting count (alpha=0.2) is noisy and delayed, so a slow filter (equivalent to a time constant of about 5 samples) smooths out transients. The GPU inter-completion interval (alpha=0.3) is faster and less noisy, so a moderately fast filter (time constant of about 3 samples) tracks changes more quickly.

The assistant then simulates the startup behavior:

With these parameters, at startup when there's an 8-item deficit, the proportional term alone gives a 0.8 correction, pushing the rate multiplier to 1.8x and cutting the dispatch interval to about 0.56 of the GPU interval—a reasonable aggressive boost to catch up.

This is a back-of-the-envelope calculation that validates the parameter choices. An 80% speedup in dispatch during startup is aggressive enough to fill the pipeline quickly but not so aggressive as to flood the system.

The Implementation: From Theory to Code

The message then shifts from reasoning to implementation. The assistant describes the structure of the DispatchPacer:

Defining the DispatchPacer struct... Still writing dispatcher logic... Writing the complete implementation... 5 dispatches with full state.

The dispatch loop has a specific structure:

I'll update the pacer with the latest observations and compute the interval, then handle the bootstrap check to determine whether to use a short interval or wait for GPU data. The main loop selects between a timer or GPU event—if the timer fires, I proceed to dispatch; if a GPU event arrives, I update the pacer on the next iteration. Then I pop work from the queue (waiting if needed and handling GPU events in the meantime), acquire the budget which may block, check for failed jobs, and finally dispatch to the worker.

This loop design is elegant. It uses a bootstrap phase that dispatches the target number of items at a fixed spacing before the first GPU completion, then switches to timer-based pacing at the PI-computed interval. The timer fires at the computed dispatch interval; GPU events update the pacer state but don't directly trigger dispatches (unlike the previous P-controller, which dispatched on every GPU event). This decoupling is crucial for stability—it prevents the controller from reacting too aggressively to individual events.

The assistant also lists a set of TODO items:

Assumptions and Their Implications

The design in <msg id=3429> rests on several assumptions, some explicit and some implicit:

Assumption 1: The GPU consumption rate is stable and measurable. The pacer uses an EMA of GPU inter-completion intervals as a feed-forward rate. This assumes that GPU time is roughly constant (~1s) and that variations are noise, not signal. If the GPU rate changes systematically (e.g., due to different proof types), the EMA will adapt, but slowly.

Assumption 2: The waiting queue depth is a meaningful control signal. The PI controller operates on the error between the actual waiting count and the target. This assumes that the waiting count is a reliable indicator of pipeline health—too few waiting means the GPU is starving, too many means the system is overprovisioned. However, the waiting count is a delayed and noisy signal because it reflects synthesis completions that happened 20–60 seconds ago.

Assumption 3: The system is linear and time-invariant (enough). Classical PI control assumes linear dynamics. The real system has nonlinearities (e.g., CPU contention causing synthesis time to increase with more concurrent jobs, memory allocation failures, GPU queue depth limits). The controller is designed to be conservative enough to handle these nonlinearities gracefully, but this is an assumption that must be validated empirically.

Assumption 4: The bootstrap phase can safely dispatch target items without pacing. This assumes that the initial burst won't overwhelm the system. With target=8 and 20–60s synthesis time, this is probably safe, but if the system has other constraints (e.g., memory budget), the burst could cause issues.

Assumption 5: The integral term won't wind up unacceptably. The assistant doesn't explicitly mention anti-windup in this message (though it's added later in the synthesis throughput cap). The assumption is that with Ki=0.01, the integral term accumulates slowly enough that windup is not a problem during normal operation.

What This Message Requires to Be Understood

To fully grasp <msg id=3429>, a reader needs knowledge in several domains:

Control theory fundamentals: Understanding of proportional and integral control, feedback delay, stability margins, gain tuning, and EMA filtering. The assistant uses terms like "Kp," "Ki," "phase lag," "steady-state error," and "windup" with precision.

GPU pipeline architecture: Knowledge of the CuZK proving pipeline, the distinction between CPU-bound synthesis and GPU-bound proving, the concept of partition-level parallelism, and the role of the dispatch loop.

Systems programming: Understanding of atomic counters, timer-based event loops, concurrent queues, and memory budgets in Rust. The assistant references AtomicU64, Notify-based synchronization, and budget acquisition.

The specific session history: Knowledge of the pinned memory pool fix, the failed semaphore and P-controller attempts, and the user's feedback about stabilization time. Without this context, the message's design decisions seem arbitrary.

What This Message Creates

Message <msg id=3429> produces several forms of output knowledge:

A concrete controller design: The DispatchPacer struct with specific gains (Kp=0.1, Ki=0.01), EMA parameters (alpha_waiting=0.2, alpha_gpu=0.3), and a bootstrap phase. This is a reusable artifact that could be applied to similar pipeline scheduling problems.

An implementation plan: The TODO items and loop structure provide a roadmap for coding the pacer. The assistant's description of the dispatch loop is detailed enough to serve as a specification.

A rationale for parameter choices: The message explains why the gains are small and the EMAs are slow, grounding the choices in the 20–60s feedback delay. This rationale is valuable for future tuning.

A control-theoretic framing of the problem: By framing the dispatch scheduling as a PI control problem with feed-forward, the message creates a vocabulary and analytical framework for discussing pipeline stability. Future iterations (like the synthesis throughput cap) build on this framing.

An understanding of tradeoffs: The assistant acknowledges that the system will take "minutes to stabilize" and that the PI tuning is "very conservative." This sets expectations for deployment and testing.

The Later Refinement: Synthesis Throughput Cap

While not part of <msg id=3429> itself, the article must note that this message's design was later extended with a synthesis throughput cap. The user identified an edge case: when synthesis is compute-constrained, the PI controller drives the dispatch interval below the GPU rate to fill the queue, flooding the system with concurrent synthesis jobs and causing CPU contention. The assistant responded by adding a cap that clamps the dispatch rate to not exceed the measured synthesis completion rate, with anti-windup that freezes the integral term when the cap is active.

This refinement validates the design philosophy of <msg id=3429>: start with a clean PI controller, deploy it, observe its behavior under real workloads, and add safeguards for edge cases. The base design was sound enough to support this extension without a fundamental redesign.

Conclusion

Message <msg id=3429> is a masterclass in applying control theory to a real-world distributed systems problem. The assistant takes a user observation about long stabilization times and translates it into a precise controller design with justified parameters. The thinking process shows a deep understanding of feedback delay, stability margins, and the practical constraints of a GPU proving pipeline.

The message's greatest strength is its balance of theoretical rigor and practical engineering. The assistant considers control-theoretic principles (low gains for long delays, integral term for steady-state error) but doesn't let theory paralyze action. It picks reasonable parameters, designs a clean implementation, and sets expectations for deployment. The later addition of the synthesis throughput cap proves that the design was extensible and that the team could iterate based on empirical observation.

For anyone building a pipeline with long feedback delays—whether in GPU computing, manufacturing, or network flow control—this message offers a valuable template. Start with a feed-forward estimate of the consumption rate, layer on gentle PI correction, smooth your signals with appropriate EMAs, bootstrap carefully, and expect to iterate. The mathematics of control theory may be complex, but the engineering wisdom is simple: respect your delays, tune conservatively, and measure everything.