The Calculus of Control: Tuning a PI Controller for GPU Pipeline Dispatch

In the middle of a high-stakes optimization session for a zero-knowledge proof acceleration pipeline, a single message captures one of the most intellectually demanding moments in the conversation: the transition from a structural redesign of a dispatch pacer to the delicate art of tuning its proportional-integral (PI) controller parameters. Message [msg 3606] is the bridge between two worlds — the coarse-grained architecture of the previous commit and the fine-grained parameter tuning that follows. It is a message about reasoning from first principles, about understanding why a control system misbehaves before attempting to fix it, and about the assumptions that can lead a well-intentioned design astray.

The Context: A Pipeline Under Pressure

To understand message [msg 3606], one must first understand the system it governs. The CuZK proving engine is a GPU-accelerated zero-knowledge proof system that processes proof partitions through a multi-stage pipeline: synthesis (CPU-bound, 30-60 seconds per partition), GPU proving (GPU-bound, ~1 second per partition), and finalization. Between synthesis and GPU proving sits a dispatch queue — a buffer of synthesized partitions waiting for GPU processing. The goal of the dispatch pacer is to maintain a target number of partitions in this queue (the target parameter, typically 8), ensuring the GPU never starves while preventing the queue from growing so large that it overwhelms memory.

The pacer uses a PI controller — a classic feedback mechanism from control theory. The proportional term (P) reacts to the current error (how far the queue depth is from target), while the integral term (I) accumulates past errors to correct for persistent biases. The controller's output is a rate_mult value that scales the base dispatch interval: if the GPU processes one partition every X milliseconds, the pacer dispatches new partitions at intervals of X / rate_mult. A rate_mult of 1.0 means dispatch at exactly the GPU consumption rate; 2.0 means dispatch twice as fast (filling the queue); 0.5 means dispatch at half speed (letting the queue drain).

The previous commit (e0ea675b, described in [msg 3604]) had just completed a major redesign of this pacer. It replaced a flawed GPU rate measurement that included idle time (creating a self-reinforcing collapse loop), removed a synthesis throughput cap that created a vicious cycle of ever-slower dispatch, and added a re-bootstrap mechanism to refill the pipeline when it drained completely. The user's response in [msg 3601] was cautiously positive — "seems pretty good" — but identified a critical problem: whenever the system hit the memory ceiling, the integral term went deeply negative, causing the pacer to back off so aggressively that the entire pipeline drained to zero before resuming. The integral was "almost always saturated," providing no useful control authority.

The Reasoning: Deconstructing the Problem

Message [msg 3606] opens with the assistant confirming the commit and then immediately pivoting to the PI tuning issues. The structure of the reasoning is revealing: the assistant enumerates the user's three complaints, then walks through a detailed mathematical analysis of the controller's behavior under various conditions.

The assistant begins by parameterizing the problem. With target=8, the error signal target - ema_waiting (where ema_waiting is an exponential moving average of the queue depth) can range from +8 (empty queue) to perhaps -20 (queue severely overfull). The integral gain ki=0.008 means that with a steady error of +8, the integral grows by only 0.064 per second — it would take 312 seconds to reach the maximum integral of 20. This seems innocuous, but the assistant correctly identifies that the time step dt matters: during bootstrap, when items are dispatched every 3 seconds and error is persistently positive, the integral accumulates faster than expected.

The critical insight comes next. The assistant traces the sequence of events during a "memory ceiling slam":

"when items slam into memory ceiling, many concurrent syntheses complete at once and flood the GPU queue. waiting spikes well above target, error goes deeply negative, and integral plummets to -20 in seconds. Then when the burst is consumed, the negative integral makes dispatch incredibly slow."

This is the root cause analysis. The integral doesn't gradually accumulate — it plummets because the error signal is large and negative during the burst. The symmetric integral clamping (±20) means the integral can go as deeply negative as it can positive. But the consequences are asymmetric: a positive integral causes slightly faster dispatch (harmless), while a negative integral causes drastically slower dispatch (catastrophic, because it drains the pipeline and starves the GPU).

The Assumptions and Their Failure

The original design of the PI controller made several assumptions that proved incorrect under real-world conditions:

Assumption 1: The error signal would stay within a bounded, symmetric range. The designer assumed that queue depth would oscillate roughly symmetrically around the target, so symmetric integral clamping made sense. In practice, the system exhibits strong asymmetry: the queue can spike far above target (when a burst of syntheses completes simultaneously) but can only go to zero below target. The negative error spikes are both larger in magnitude and more damaging in effect.

Assumption 2: The integral term would provide "gentle drift correction." With ki=0.008 and max_integral=20, the integral's maximum contribution to the correction signal is 0.008 * 20 = 0.16, compared to the proportional term's contribution of kp * error = 0.1 * 8 = 0.8 at peak. The integral was designed to be a minor player. But during a burst, the integral accumulates negative value rapidly because the error is large and sustained, and once negative, it takes a long time to wind back to zero — all while suppressing dispatch.

Assumption 3: The rate_mult clamp of [0.1, 5.0] provided sufficient range. A rate_mult of 0.1 means dispatch at 10x slower than GPU rate — essentially a near-complete stall. The assistant realizes this is far too aggressive: "a correction of 0 means dispatch at exactly GPU rate. The useful range of correction is roughly [-0.9, +4.0]." When the integral goes negative and combines with a negative P term, the correction easily exceeds -0.9, driving rate_mult to its floor of 0.1 and effectively halting dispatch.

The Proposed Solution: Asymmetric Clamping and Normalized Error

The assistant's proposed fix in [msg 3606] has three components, each addressing a specific failure mode:

1. Normalize the error by target. Instead of using raw error target - waiting (range -20 to +8), use (target - waiting) / target (range approximately -2.5 to +1.0). This makes the controller's behavior independent of the target value and bounds the error signal more tightly. A queue of 16 with target 8 produces a normalized error of -1.0, not -12.

2. Asymmetric integral clamping. Allow the integral to go positive (to +2.0) but barely negative (to -0.5). This is the key insight: a positive integral causes slightly faster dispatch, which is benign; a negative integral causes slower dispatch, which can starve the GPU. By clamping the negative side aggressively, the assistant prevents the integral from accumulating enough negative value to stall the pipeline.

3. Tighter rate_mult bounds. Change the clamp from [0.1, 5.0] to [0.3, 3.0]. This means dispatch can never slow to more than 3.3x the GPU rate (instead of 10x), preventing complete stalls even under extreme conditions.

The assistant also considers retuning the gains: lowering kp so the proportional term doesn't overreact to transient spikes, and adjusting ki to keep the integral in a useful range.

The Thinking Process Visible in the Reasoning

What makes message [msg 3606] particularly valuable is the transparency of the assistant's reasoning. The message is structured as a live thought process, not a polished report. We see the assistant working through the math step by step:

Input Knowledge Required

To fully understand this message, the reader needs:

  1. PI controller theory: Understanding that the proportional term responds to current error while the integral term accumulates past errors, and that the integral can "wind up" or "wind down" to problematic values.
  2. The CuZK pipeline architecture: Knowledge that synthesis takes 30-60 seconds per partition, GPU proving takes ~1 second, and the dispatch queue sits between them. The target parameter (8) represents the desired number of partitions in the GPU queue.
  3. The previous redesign (commit e0ea675b): Understanding that the pacer uses GPU processing time feed-forward, has a re-bootstrap mechanism, and had just been stripped of a synthesis throughput cap that caused collapse loops.
  4. The specific parameters: kp=0.1, ki=0.008, max_integral=±20, rate_mult clamp [0.1, 5.0], and the meaning of correction = kp*error + ki*integral which feeds into rate_mult = (1 + correction).clamp(0.1, 5.0).

Output Knowledge Created

This message produces several important outputs:

  1. A diagnosis: The root cause of the pipeline drain is identified as symmetric integral clamping combined with asymmetric error patterns. The integral goes deeply negative during bursts and then suppresses dispatch long after the burst has passed.
  2. A proposed fix: Three specific changes to the controller parameters, each with a clear rationale. The asymmetric clamping is the most novel insight — recognizing that positive and negative integral have different consequences and should be treated differently.
  3. A framework for tuning: The assistant establishes that the useful range of correction is roughly [-0.9, +4.0] (corresponding to rate_mult from 0.1 to 5.0), and that the integral's contribution should be a small fraction of this range. This gives future tuning efforts a clear target.
  4. A mental simulation: The assistant walks through several scenarios (steady state, queue half-empty, queue empty, queue overfull) to verify the proposed tuning would behave correctly. This simulation serves as a low-cost validation before deploying to production.

The Broader Significance

Message [msg 3606] exemplifies a pattern that recurs throughout the conversation: the assistant encounters a system that works "pretty well" but fails at the edges, and the debugging process requires reasoning from mathematical first principles rather than trial-and-error parameter tweaking. The user's observation — "integral is almost always saturated" — is a symptom, not a cause. The assistant's contribution is to trace that symptom back to its root: the symmetric clamping assumption that didn't hold under real-world conditions.

The message also illustrates the challenge of tuning feedback controllers in systems with long feedback delays. The 30-60 second synthesis time means that the controller's actions today affect the queue depth a minute later. By the time the controller sees the result of its actions, the system state may have changed entirely. This is why the integral term is so dangerous: it accumulates based on stale information and continues to influence behavior long after the conditions that caused the accumulation have passed.

In the end, the assistant's proposed fix — asymmetric clamping, normalized error, tighter bounds — is a pragmatic response to a fundamental limitation. It doesn't eliminate the feedback delay problem, but it bounds the damage the integral can do. The integral can still accumulate positive value (causing slightly faster dispatch, which is fine), but it can barely go negative (preventing the pipeline-draining backoff). This is control theory in the real world: not finding the mathematically optimal solution, but finding one that works reliably under all observed conditions.