Tuning the PI Controller: Diagnosing Integral Saturation in a GPU Dispatch Pacer

The Message

"Commit, seems pretty good; Whenever we 'slam' into the memory ceiling integral goes negative. Whenever integral goes negative new partitions completely start entering synthesis until we fully drain all running and waiting pipelines, essentially starting from scratch, which seems wrong, the backoff shouldn't be nearly this aggresive. Also seems like integral is almost always saturated, we should tune it somehow so that it would be floating around a value where it has useful control authority (P might also need tuning)"

This message, sent by the user at index 3601 in the conversation, is a deceptively compact diagnostic report. In just three sentences, it communicates a nuanced systems debugging observation, identifies a specific control-theoretic failure mode, and proposes a direction for remediation. To understand why this message matters, we must situate it within the broader arc of the session: the assistant had been iterating on a Proportional-Integral (PI) controller that regulates the rate at which synthesized GPU proof partitions are dispatched to workers in a zero-knowledge proving pipeline. The pacer's job is to maintain a target queue depth of partitions waiting for GPU processing, using feedback from measured queue depth and GPU processing rates. The user's message lands after the assistant deployed a re-bootstrap fix (synthcap3) and the user observed the system's behavior under memory pressure.

The Reasoning and Motivation: Why This Message Was Written

The user's motivation is straightforward but critical: they are the system's operator and the final arbiter of whether the pacer behaves correctly. Having observed the system under real workload conditions, they detected a pathological feedback loop. The message is not a casual observation — it is a precise engineering critique delivered to the assistant so that the next iteration of code can address the root cause.

The user had just confirmed that the re-bootstrap logic (synthcap3) was working, saying "Commit, seems pretty good." But immediately after that qualified approval, they pivoted to a deeper problem: when the system hits the memory ceiling, the PI controller's integral term goes deeply negative. This negative integral then forces the dispatch rate so low that the pipeline completely drains — every partition currently being synthesized finishes and no new ones start — effectively resetting the pipeline to empty. The user correctly identifies this as wrong: "essentially starting from scratch, which seems wrong, the backoff shouldn't be nearly this aggressive."

The motivation for writing this message is to prevent the assistant from committing the current code as-is. The user is saying: "Don't merge yet — there's a fundamental tuning problem that makes the controller unstable under the very condition it's supposed to handle (memory pressure)." This is a gatekeeping message, one that halts forward progress on deployment and redirects effort toward control theory debugging.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains:

Control theory fundamentals: The user references "integral" and "P" (proportional) terms, indicating familiarity with PI controller architecture. The integral term accumulates error over time and drives the control signal to zero steady-state error, but it can "wind up" or "saturate" when the error persists, causing overshoot and slow recovery. The user understands that an integral that is "almost always saturated" has lost its ability to provide useful control — it's stuck at a boundary value and cannot respond dynamically to changes.

GPU proving pipeline architecture: The proving pipeline involves multiple stages — circuit synthesis (CPU-bound, producing partitions) and GPU proving (GPU-bound, consuming partitions). The dispatch pacer sits between these stages, deciding when to send a synthesized partition to the GPU. The "memory ceiling" refers to GPU memory exhaustion: when too many partitions are queued on the GPU, memory runs out, causing errors or thrashing.

The specific implementation context: The user knows that the PI controller's integral term is the mechanism that should reduce dispatch rate when the queue is persistently too deep (indicating the GPU cannot keep up). But the observed behavior — integral going so negative that the pipeline fully drains — indicates that the integral gain (ki) is too high, the integral limits are too wide, or both.

The deployment environment: The system runs on a machine with 755 GB of RAM and a GPU, using a pinned memory pool for zero-copy transfers. The memory budget system tracks allocations and can stall synthesis when the budget is exhausted, creating the "memory ceiling" condition.

The Thinking Process Visible in the Message

The user's reasoning is compact but reveals a structured diagnostic thought process. Let me unpack it:

Step 1 — Observation: "Whenever we 'slam' into the memory ceiling integral goes negative." The user has identified a causal trigger: hitting the memory budget limit causes the integral term to swing negative. This is the entry point of the diagnosis.

Step 2 — Trace the consequence: "Whenever integral goes negative new partitions completely start entering synthesis until we fully drain all running and waiting pipelines." The user traces the chain of causation: negative integral → dispatch rate drops to near-zero → all in-flight synthesis completes → no new work arrives → pipeline empties completely. This is essentially a full system reset.

Step 3 — Normative judgment: "essentially starting from scratch, which seems wrong, the backoff shouldn't be nearly this aggressive." The user evaluates the behavior against expected system dynamics. A well-tuned controller should gently reduce dispatch rate under memory pressure, not slam the brakes so hard that the pipeline drains entirely. The backoff is disproportionate to the disturbance.

Step 4 — Meta-observation about controller state: "Also seems like integral is almost always saturated." This is a crucial observation. If the integral term is perpetually at its upper or lower bound, it means the controller is operating outside its effective range. The integral has become a binary switch (max or min) rather than a smoothly varying control signal. This indicates poor tuning of either the integral gain or the integral clamp limits.

Step 5 — Proposed remedy: "we should tune it somehow so that it would be floating around a value where it has useful control authority (P might also need tuning)." The user suggests targeting a regime where the integral term hovers in the middle of its range, with headroom to move in either direction. They also flag the proportional term as potentially needing adjustment — a recognition that the problem might not be purely integral-related.

This is sophisticated systems thinking. The user is not just reporting a symptom; they are performing a root cause analysis in real-time and communicating the essential findings to the assistant so the code can be fixed.

Mistakes and Incorrect Assumptions

The user's analysis is largely correct, but there are some implicit assumptions worth examining:

Assumption that integral saturation is purely a tuning problem: The user frames the issue as one of tuning parameters (ki, integral limits). While this is true at one level, the deeper issue might be architectural — the PI controller might be the wrong tool for a system with hard memory ceilings. A PI controller assumes smooth, continuous dynamics, but hitting a memory ceiling is a discrete, hard constraint. The integral term winds up because the controller cannot reduce the error (queue depth exceeds target) through normal means — the GPU is genuinely saturated and cannot consume faster. In such cases, anti-windup mechanisms (like clamping or back-calculation) might be needed regardless of tuning.

Assumption that the integral should "float" in the middle: The user suggests tuning so the integral hovers around a value with "useful control authority." This is a reasonable goal for steady-state operation, but it's worth noting that under persistent overload (memory ceiling), the integral should be negative — that's its job. The problem is the magnitude of the negativity causing pipeline drain, not the negativity itself. The user implicitly recognizes this by distinguishing between "useful" saturation and destructive saturation.

The "P might also need tuning" caveat: This is a hedge that shows the user is keeping an open mind. They correctly identify that the integral is the primary culprit but acknowledge that the proportional term might also contribute to the aggressive backoff. In a PI controller, the P term responds to instantaneous error, while I responds to accumulated error. Under a sustained memory ceiling, the error persists, so the I term dominates — but a poorly tuned P term could exacerbate the initial overshoot.

Output Knowledge Created by This Message

This message generates several forms of output knowledge:

Diagnostic signal for the assistant: The assistant now knows that the synthcap3 deployment, while fixing the re-bootstrap issue, has a PI tuning problem. This redirects the assistant's efforts from deployment infrastructure back to controller tuning.

Specific failure mode documented: The message documents the "integral saturation → pipeline drain" failure mode, which becomes the target for the next iteration of fixes. This failure mode will be addressed in subsequent messages where the assistant normalizes error by target, lowers ki from 0.02 to 0.001, raises the integral cap asymmetrically (positive 100, negative -20), and tightens the rate_mult clamp.

Performance criteria established: The user implicitly defines what "good" looks like: the integral should not be perpetually saturated, the pipeline should not fully drain under memory pressure, and the backoff should be proportional to the disturbance. These become the acceptance criteria for the pitune series of deployments.

Boundary of the current approach revealed: The message reveals that the PI controller, as currently tuned, cannot gracefully handle hard memory ceilings. This knowledge informs future architectural decisions, including the eventual addition of a max_parallel_synthesis cap as a separate mechanism to prevent CPU/DDR5 contention.

The Broader Context: Where This Message Fits

This message sits at a critical inflection point in the session. The assistant had just deployed synthcap3, which removed the problematic synthesis throughput cap and added re-bootstrap logic. The user's initial reaction was positive ("seems pretty good"), but real-world testing under memory pressure revealed the integral saturation issue. The assistant's response to this message — a series of PI tuning changes deployed as pitune1 through pitune4 — represents the final major tuning iteration before the system stabilized and the user said "seems to work well, commit."

Without this message, the assistant would have committed code with a fundamentally unstable controller. The pipeline drain behavior would have caused severe performance degradation in production: every time memory pressure mounted, the pipeline would empty and need to be re-filled from scratch, wasting all the parallelism benefits the zero-copy pinned memory pool was designed to provide. The user's intervention prevented this.

Conclusion

The message at index 3601 is a masterclass in concise, high-signal diagnostic communication. In 81 words, the user identifies a control-theoretic failure mode, traces its causal chain, evaluates it against expected behavior, observes a meta-problem with controller saturation, and proposes a tuning direction. The message demonstrates deep systems thinking and a collaborative debugging relationship between user and assistant. It is the kind of message that separates a superficial "it's broken" report from a useful engineering signal — one that tells the developer not just what is wrong, but why it is wrong and how to think about fixing it. The subsequent pitune deployments validated the user's analysis, and the system eventually stabilized, thanks in no small part to this precise diagnostic intervention.