Tuning the PI Controller: When Control Theory Meets GPU Pipeline Realities

Introduction

In the course of a high-stakes optimization session for a GPU-accelerated zero-knowledge proving pipeline, a single message from the AI assistant stands out as a masterclass in applied control theory debugging. Message [msg 3607] captures the moment when the assistant, having just committed a major redesign of the dispatch pacer, sits down to systematically analyze why the PI (Proportional-Integral) controller at the heart of the GPU dispatch system is behaving pathologically. The message is deceptively short—a few paragraphs of reasoning followed by an edit—but it encapsulates a deep diagnostic process that bridges control theory, GPU architecture, and production debugging.

Context: The Dispatch Pacer and Its Discontents

To understand message [msg 3607], we must first understand what came before. The assistant had been iterating on a "dispatch pacer"—a feedback control system that regulates how quickly synthesized proof partitions are dispatched to the GPU for proving. The pacer uses a PI controller to maintain a target number of "waiting" partitions in the GPU queue. The feed-forward term uses the measured GPU processing rate, and the feedback (PI) term corrects for deviations from the target queue depth.

The previous iteration (synthcap3, deployed at [msg 3598]) had removed a problematic synthesis throughput cap that created a vicious cycle of collapse, added re-bootstrap detection when the pipeline drained, and slowed the bootstrap interval from 200ms to 3 seconds. The user reported at [msg 3601] that this version was "pretty good" but identified two critical problems:

Integral saturation: The integral term was "almost always saturated" at its ±20 bounds, meaning it had no useful control authority—it was pegged at a limit, unable to respond dynamically.
Aggressive backoff on memory pressure: When the system hit the memory ceiling (too many concurrent syntheses consuming RAM), the integral went deeply negative, causing dispatch to slow to a crawl. This drained the entire pipeline, forcing a re-bootstrap from scratch—an expensive restart. The user explicitly asked for tuning: "we should tune it somehow so that it would be floating around a value where it has useful control authority (P might also need tuning)."

The Message: A Mathematical Autopsy

Message [msg 3607] is the assistant's response. The assistant had already committed the synthcap3 changes at [msg 3604] and read the current engine.rs file at [msg 3606]. Now, in this message, the assistant works through the PI controller math from first principles.

The message begins with a crucial structural observation: the correction feeds into rate_mult = (1 + correction).clamp(0.1, 5.0). This means the useful range of the correction term is approximately [-0.9, +4.0]—a correction of 0 means dispatch at exactly the GPU rate, negative values slow down dispatch, positive values speed it up. The clamping ensures the dispatch rate never goes below 10% of GPU rate or above 500%.

Then comes the quantitative analysis that reveals the root cause:

The P term dominates: With kp = 0.1 and an error of 8 (target queue depth is 8, actual waiting is 0), the proportional term contributes 0.1 × 8 = 0.8 to the correction. With ki = 0.008 and an integral of 20 (the saturation limit), the integral term contributes only 0.008 × 20 = 0.16. Total correction: 0.96. The integral accounts for just 17% of the total correction, even though it's pegged at its maximum value. The P term is doing all the work, and the integral is essentially dead weight.

The P term overreacts to negative errors: When waiting spikes to 20 (because a burst of syntheses completed simultaneously and flooded the GPU queue), the error becomes 8 - 20 = -12. The P term alone contributes 0.1 × (-12) = -1.2, which drives the correction below -0.9, causing rate_mult to clamp at 0.1—a 10× slowdown. This is the aggressive backoff the user observed.

The integral accumulates destructively: During the negative error spike, the integral rapidly accumulates large negative values. When the burst is consumed and waiting returns to normal, the integral remains deeply negative, continuing to suppress dispatch long after the transient has passed.

The Reasoning Process: A Window Into Diagnostic Thinking

What makes this message remarkable is the clarity of the assistant's reasoning. The assistant doesn't just tweak parameters blindly—it works through the math to understand why the controller is failing.

The assistant considers the error range: with a target of 8, error ranges from +8 (empty queue) to perhaps -20 (overloaded). It calculates integral accumulation rates: at ki = 0.008 and error = 8, the integral grows by 0.064 per second. To reach the saturation limit of 20 would take 312 seconds—but with larger dt (time step) during bootstrap ticks, it grows faster. The assistant traces through the bootstrap timeline: 8 items dispatched at 3-second intervals = 24 seconds, accumulating 1.5 integral. Then a 30-second gap before GPU completions arrive, accumulating another 1.9. Then the GPU burst processes 8 items, potentially spiking waiting above target and driving error negative.

This step-by-step simulation of the controller's behavior is the key insight. The assistant realizes that the real problem isn't the integral saturation per se—it's that the P term is so strong that it alone causes the aggressive backoff, and the integral then locks in that backoff for an extended period.

The Proposed Fix: Four Targeted Changes

Based on this analysis, the assistant proposes four changes:

Normalize error by target: Divide error by the target queue depth, so error is in the range [-1, +1] under normal conditions. This makes the controller scale-invariant—the same tuning works regardless of whether the target is 8 or 80.
Lower kp: Reduce the proportional gain so the P term doesn't overreact to transient spikes. The current kp = 0.1 with raw error of 8 produces a correction of 0.8—nearly the entire useful range. After normalization, kp can be much smaller.
Asymmetric integral clamping: Allow less negative integral than positive. The insight here is that slowing down too aggressively (negative integral) is more dangerous than speeding up too aggressively (positive integral). A negative integral that persists after a transient forces the pipeline to drain completely, which is expensive. A positive integral that overspeeds dispatch will just cause the memory budget to block, which is harmless.
Lower max_integral: Reduce the integral saturation limits so the integral stays in a range where it has useful control authority. If the integral is always saturated at ±20, it's not doing anything useful—it's just a constant offset.

Assumptions and Input Knowledge

This message assumes significant domain knowledge on the reader's part. To understand it, one needs:

PI controller theory: Understanding that the proportional term responds to current error, the integral term accumulates past error, and the two are summed to produce a control signal.
The specific architecture of the dispatch pacer: That rate_mult is a multiplier on the GPU processing rate, clamped to [0.1, 5.0], and that correction feeds into it as 1 + correction.
The GPU pipeline structure: That there's a synthesis phase (CPU-bound, memory-intensive) followed by a GPU proving phase, that multiple GPU workers run in parallel, and that the memory budget limits concurrent syntheses.
The previous iteration's parameters: kp = 0.1, ki = 0.008, max_integral = 20, target = 8. The message also assumes the user is familiar with the ongoing tuning process—the user had just reported the integral saturation issue at [msg 3601], and the assistant had committed the synthcap3 changes at [msg 3604].

Output Knowledge Created

This message creates several forms of output knowledge:

A quantitative diagnosis of the PI controller pathology: The assistant demonstrates that the P term dominates (0.8 out of 0.96 correction) and the integral is saturated and useless. This is a concrete, measurable finding.
A root cause analysis of the aggressive backoff: The P term alone, at kp = 0.1 and error = -12, produces a correction of -1.2, which is enough to clamp rate_mult at 0.1. The integral then locks in this backoff.
A set of four specific tuning recommendations: Each with a clear rationale. Normalize error so tuning is scale-invariant. Lower kp to reduce P overreaction. Asymmetric clamping to prevent destructive negative integral. Lower max_integral to keep integral in the useful range.
An edit to the source code: The message ends with an edit applied successfully to /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs. The edit implements these four changes.

Was There a Mistake?

The assistant's analysis is sound, but there's an implicit assumption worth examining: that the PI controller is the right tool for this job. The dispatch pacer is regulating a pipeline with significant nonlinearities: the memory budget introduces a hard ceiling (synthesis blocks when memory is exhausted), the GPU has a finite processing rate that varies with workload, and there are discrete events (bootstrap, re-bootstrap) that reset the controller state. A PI controller assumes linear, continuous behavior. The assistant acknowledges this implicitly by adding re-bootstrap logic and asymmetric clamping—these are workarounds for the fundamental nonlinearity of the system.

The assistant also assumes that lowering kp and normalizing error will fix the overreaction without making the controller too sluggish. This is a tuning tradeoff, not a mistake—but it's an assumption that would need to be validated empirically. Indeed, the subsequent messages in the conversation (pitune1 through pitune4, as mentioned in the chunk summary) show the assistant iterating on these exact parameters, suggesting that the initial tuning wasn't perfect and required further refinement.

The Broader Significance

Message [msg 3607] is a beautiful example of how to debug a feedback control system in production. The assistant doesn't guess at parameters or apply random tweaks. It:

Identifies the measurable symptom: Integral is always saturated.
Traces the symptom to its mechanism: The P term dominates because the error range is large relative to the useful correction range.
Simulates the dynamics: Walks through the bootstrap timeline to understand how error evolves.
Designs targeted fixes: Each fix addresses a specific failure mode. This is the difference between tuning by rote ("try lowering ki") and tuning by understanding ("the integral contributes only 17% of the correction even at saturation, so we need to either increase ki, decrease max_integral, or normalize the error so the P term doesn't dominate"). The message also reveals the assistant's mental model of the system. The assistant thinks in terms of time constants (how fast does integral accumulate?), saturation points (where does the correction clamp?), and transient responses (what happens when a burst of GPU completions arrives?). This systems-thinking approach is what makes the analysis so effective.

Conclusion

Message [msg 3607] is a turning point in the dispatch pacer optimization. It transforms the PI controller from a black box with mysterious behavior into a transparent, analyzable system. The assistant's mathematical autopsy reveals that the controller wasn't actually controlling—the integral was saturated and useless, and the P term was causing all the damage. The four proposed fixes address the root causes rather than the symptoms, and they set the stage for the successful tuning that follows in pitune1 through pitune4.

In a broader sense, this message demonstrates a principle that applies far beyond GPU pipeline optimization: when a feedback system behaves pathologically, don't just tweak the parameters—understand the math. The numbers will tell you what's broken.