The Integral That Wouldn't Float: Tuning a PI Controller for GPU Dispatch in a Zero-Knowledge Proving Pipeline

A Single Table That Captures a Control Theory Debugging Marathon

In message <msg id=3656> of an opencode coding session, the assistant delivered what appears at first glance to be a routine status update: a four-row table comparing old and new PI controller parameters for a GPU dispatch pacer in the cuzk zero-knowledge proving engine. The message reads:

Running — PID 185150. The integral should now float in a useful range:

>

| | Old | New | |---|---|---| | ki | 0.02 | 0.001 (20x lower) | | max_integral_pos | 2.0 | 100.0 (50x higher) | | max_integral_neg | -0.5 | -20.0 (40x higher) | | Max I correction | +0.04 / -0.01 | +0.10 / -0.02 | | Time to saturate (error=0.5) | ~4s / ~1s | ~200s / ~40s |

Beneath this compact table lies the culmination of an intensive debugging and tuning marathon spanning multiple deployment cycles, each with its own Docker image tag (pitune1, pitune2, pitune3, pitune4), each representing a hypothesis about how to keep a GPU fed with work in a complex pipeline where synthesis (circuit construction on CPU) and proving (GPU computation) must dance in lockstep. This message is not merely reporting numbers—it is declaring victory over a subtle control theory problem that had been causing the proving pipeline to stall and collapse under its own feedback dynamics.

The Problem: Integral Saturation in a GPU Dispatch Controller

To understand why this message matters, one must understand the system it controls. The cuzk engine is a high-performance zero-knowledge proof generator that processes proofs in a pipelined fashion. CPU-based synthesis constructs circuit assignments, which are then dispatched to GPUs for the computationally intensive proving work. Between these two stages sits the DispatchPacer—a PI (Proportional-Integral) controller that regulates how aggressively new synthesis work is dispatched to keep the GPU queue at a target depth.

The PI controller computes a rate_mult value that speeds up or slows down dispatch based on the error between the current GPU queue depth (waiting) and a target depth. The proportional term (P) reacts to the current error; the integral term (I) accumulates error over time to correct persistent biases. In theory, the integral term should "float" at a value that compensates for steady-state mismatches between synthesis rate and GPU processing rate.

In practice, the integral was saturating—hitting its clamp limits almost immediately—and therefore providing no useful control authority. As the user observed in <msg id=3645>: "Seems like higher integral cap but lower value (possibly much?) could help as integral still saturates." This observation triggered the reasoning and parameter change that produced the target message.

The Reasoning: Why the Integral Was Failing

The assistant's reasoning in <msg id=3646> reveals a deep understanding of both control theory and the specific dynamics of the GPU pipeline. The analysis identified the root cause: with ki=0.02 and max_integral_pos=2.0, the integral saturated almost immediately. At a normalized error of 1.0 (empty queue) with a 2-second bootstrap interval, the integral grew by 2.0 per tick—hitting the cap in a single update.

The assistant walked through the math explicitly:

Current: ki=0.02, max_integral_pos=2.0, max_integral_neg=-0.5 - Max I contribution: 0.02 2.0 = 0.04 (positive), 0.02 -0.5 = -0.01 (negative)

The key insight was that the integral term itself grows at norm_error * dt per update—the ki scaling only applies when the integral is used for correction, not when it accumulates. This meant the integral would pin to its limit within seconds and stay there, rendering it effectively a constant offset rather than a dynamic control signal.

The assistant considered multiple approaches: baking ki into the accumulation itself (matching standard PID formulation where u(t) = kp * e(t) + ki * ∫e(t)dt), or simply widening the bounds while lowering ki. The chosen approach was the latter: drop ki from 0.02 to 0.001 (20x lower) while expanding max_integral_pos from 2.0 to 100.0 (50x higher) and max_integral_neg from -0.5 to -20.0 (40x higher).## The Context: A Chain of Tuning Iterations

This message is not an isolated event—it is the fourth iteration in a tuning sequence. The assistant had previously deployed pitune1, pitune2, and now pitune3. Each iteration represented a hypothesis about the correct control parameters:

Assumptions and Input Knowledge

The assistant's reasoning in <msg id=3646> makes several assumptions that are critical to understanding the correctness of the solution:

Assumption 1: The integral accumulates the full error directly. The assistant states: "the integral term itself grows at norm_error * dt per update—the ki scaling only applies when we actually use the integral for correction." This assumes a specific implementation structure where the integral state variable stores the raw accumulated error, and ki is applied as a multiplier when computing the final control output. This is indeed the standard "parallel form" of a PI controller, but it's worth noting that an alternative "series form" would bake ki into the accumulation itself. The assistant recognized this distinction and considered switching forms, but ultimately chose to keep the parallel form and simply widen the bounds.

Assumption 2: The normalized error range is [-2, +1]. The assistant had previously normalized the error by dividing by the target queue depth, so an empty queue gives error=1.0, a queue at target gives error=0, and an overfull queue gives negative error. This normalization is crucial because it makes the gains target-independent—the same kp and ki values work regardless of whether the target is 5 or 50.

Assumption 3: The bootstrap interval is approximately 2 seconds. The assistant uses dt=2s in the saturation calculation, which comes from the slow bootstrap mode implemented in earlier iterations. This is important because the integral accumulation rate is proportional to the time step—a faster update loop would accumulate integral more slowly per wall-clock second.

Assumption 4: A time-to-saturate of ~200s (positive) and ~40s (negative) is sufficient. This assumes that the pipeline's transient dynamics resolve within this timeframe—that persistent biases (like a systematic synthesis rate mismatch) will be corrected by the integral before it hits the limit. If the pipeline has sustained errors lasting longer than 200 seconds, the integral would still saturate, just more slowly.

Output Knowledge and the Thinking Process

This message creates several forms of knowledge:

Explicit knowledge: The table itself is a compact specification of the new PI parameters. Anyone reading this message can immediately understand the magnitude of the change—ki dropped by 20x, limits expanded by 40-50x. The "Max I correction" row shows that despite the radical parameter changes, the maximum control authority of the integral term actually increased slightly (from +0.04 to +0.10 positive, from -0.01 to -0.02 negative). The integral can now push harder when needed, but takes much longer to reach full authority.

Implicit knowledge: The "Time to saturate" row reveals the assistant's mental model of the system's dynamics. At a sustained error of 0.5 (half-empty queue), the old parameters saturated in 4 seconds positive and 1 second negative. The new parameters take 200 seconds and 40 seconds respectively. This 50x improvement in saturation time is the core achievement—the integral can now "float" through normal pipeline transients without pinning.

Process knowledge: The message documents the deployment as PID 185150, linking the parameter change to a specific running process on the remote server. This creates an audit trail connecting theory to deployed reality.

Mistakes and Correctness Considerations

The assistant's analysis in <msg id=3646> is technically sound, but several aspects deserve scrutiny:

The integral accumulation formulation: The assistant considered but did not implement the standard PID form where ki is baked into the accumulation (integral += ki * error * dt). In the current implementation, the integral accumulates raw error and ki is applied at output time. Both forms are mathematically equivalent for constant ki, but the raw-accumulation form is more sensitive to time step variations and requires larger state variables. The assistant's chosen approach (widen bounds, lower ki) is correct but represents a patch rather than a redesign.

The asymmetric bounds: The positive limit (100.0) is 5x larger than the negative limit (-20.0). This asymmetry reflects the assistant's understanding that the pipeline is more likely to need speeding up (positive integral) than slowing down (negative integral). However, this asymmetry could mask issues where the pipeline systematically overshoots its target.

The assumption about error persistence: The 200-second saturation time assumes error=0.5 sustained. In practice, the pipeline may experience longer periods of imbalance during batch transitions or memory ceiling events. The assistant had previously observed that "after a memory ceiling slam, the 30-60s synthesis pipeline depth means the GPU inevitably sits idle" ([msg 3632]). The 200-second positive saturation window should cover this, but the 40-second negative window is tighter.

Conclusion

Message <msg id=3656> appears unassuming—a simple table comparing before and after values for five PI controller parameters. But in the context of the full conversation, it represents the resolution of a complex control theory problem in a production GPU proving pipeline. The assistant correctly diagnosed integral saturation as the root cause of poor controller behavior, reasoned through the mathematics of integral accumulation, and deployed a parameter set that allows the integral term to "float" in a useful range rather than pinning to its limits.

The message is a testament to the value of understanding control theory fundamentals in systems engineering. The assistant didn't just tweak parameters randomly—it calculated time-to-saturate, understood the relationship between integral gain and integral limits, and communicated the change in terms of control authority and saturation dynamics. For anyone debugging a similar PI controller in a production system, this message serves as a case study in how to reason about integral windup and saturation in a pipeline with significant latency between action and effect.