Tuning the PI Pacer: A Mental Simulation Before Deployment

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, the difference between a smoothly humming pipeline and a collapsed one can come down to a handful of floating-point constants. Message [msg 3613] captures a pivotal moment in the iterative refinement of the CuZK proving engine's dispatch pacer — a PI (proportional-integral) controller responsible for regulating how quickly synthesized proof partitions are fed to the GPU. After the user reported that the controller's integral term was saturating and causing the pipeline to fully drain whenever the system hit its memory ceiling, the assistant designed a set of tuning changes and, in this message, performs a detailed mental simulation to verify the new parameters before building and deploying the binary.

The Problem: Integral Saturation and Pipeline Collapse

The context for this message begins with the user's report in [msg 3601]. The dispatch pacer, which had just been substantially redesigned in the previous commit (removing a synthesis throughput cap, adding re-bootstrap logic, and switching to GPU processing-time feed-forward), was exhibiting a critical failure mode: whenever the system "slammed into the memory ceiling," the integral term of the PI controller went deeply negative. This negative integral caused the dispatch rate to become extremely slow, and the pipeline would fully drain — all waiting and in-flight partitions would complete before any new ones were dispatched. The system essentially had to start from scratch, wasting the pipeline's momentum.

The user's diagnosis was precise: the integral was "almost always saturated," meaning it was pinned at its maximum negative value and therefore had no useful control authority. The integral term, which is supposed to provide a slow, accumulating correction for persistent errors, was instead acting as a binary switch — either neutral or catastrophically negative. The user suggested that both the integral and the proportional term might need tuning.

The Assistant's Analysis

Before writing the message we're examining, the assistant had already committed the previous pacer redesign and begun analyzing the PI tuning problem in [msg 3606]. That analysis revealed the root cause with mathematical clarity. With the old parameters (kp=0.1, ki=0.008, max_integral=20, target=8), the error signal target - ema_waiting could range from +8 (empty queue) to perhaps -20 (queue massively overfull after a memory-ceiling burst). The proportional term alone, at error=-12, would produce kp * error = -1.2, which drove the rate_mult (the multiplier applied to the GPU's natural processing rate) below zero, clamping it to 0.1 — a 10x slowdown. Meanwhile, the integral term would rapidly accumulate the large negative error, plummeting to -20 and staying there.

The fix outlined in [msg 3607] had four components: normalize the error by dividing by target (so error stays in approximately [-1, +1] under normal conditions), lower the proportional gain, implement asymmetric integral clamping (allowing much less negative integral than positive), and lower the maximum integral value to keep it in a useful range.

The Mental Simulation

Message [msg 3613] is where the assistant validates these changes through a systematic mental simulation. This is a remarkable piece of reasoning — the assistant walks through five distinct operating scenarios, computing the controller's response in each one to verify that the behavior is sensible.

Steady state (waiting ≈ target = 8, normalized error ≈ 0): Both P and I terms produce zero correction. rate_mult = 1.0. Dispatch proceeds at exactly the GPU's natural processing rate. This is the ideal equilibrium — the pipeline is balanced.

Queue half-empty (waiting = 4, normalized error = 0.5): P produces 0.5 * 0.5 = 0.25. Correction ≈ 0.25, so rate_mult = 1.25. Dispatch runs 25% faster than the GPU can consume, gently refilling the queue. This is a modest, proportional response — no panic.

Queue empty (waiting = 0, normalized error = 1.0): P produces 0.5 * 1.0 = 0.5. rate_mult = 1.5. Dispatch runs 50% faster. Even in the worst case (completely empty pipeline), the controller only accelerates by 50%. This is a deliberate design choice: the assistant is prioritizing stability over raw throughput, avoiding the kind of aggressive dispatch that could trigger the memory ceiling.

Queue overfull (waiting = 16, normalized error = -1.0): P produces 0.5 * (-1.0) = -0.5. rate_mult = 0.5. Dispatch slows to half the GPU rate. Critically, with the clamp floor set at 0.3, the rate can never go below 3.3x slower than GPU rate — the pipeline never stalls entirely. This is the key behavioral change from the old parameters, where a similar error would drive rate_mult to 0.1 (10x slowdown) or effectively zero.

Integral behavior: The assistant then considers the integral term's dynamics separately. At a steady normalized error of 0.5, the integral grows at 0.01/s (ki error = 0.02 0.5 = 0.01). It would take 200 seconds to reach the positive cap of 2.0. At that point, the integral's contribution to the correction is only 0.02 * 2.0 = 0.04 — a tiny nudge. On the negative side, the integral is clamped at -0.5, so its maximum negative contribution is 0.02 * (-0.5) = -0.01 — negligible.

This is a profound design philosophy: the integral term is deliberately weakened to act as a "gentle slow-drift corrector" while the proportional term handles fast response. The asymmetric clamping (positive cap of 2.0, negative cap of -0.5) encodes the domain knowledge that slowing down is dangerous — a negative integral that persists can collapse the pipeline, while a positive integral that builds up slowly is merely slightly inefficient.

Decisions Made and Assumptions

The assistant made several key decisions in this message. First, the choice of kp=0.5 and ki=0.02 represents a deliberate trade-off. The proportional gain is moderate — high enough to respond to queue imbalances, but low enough to avoid overreaction to transient spikes. The integral gain is very low, ensuring that the integral term accumulates slowly and never dominates the controller's output.

The assumption underlying these choices is that the primary source of disturbance is transient bursts from the memory ceiling, not persistent steady-state errors. The PI controller is designed to handle brief overflows gracefully (via the P term) while using the I term only for long-term drift correction. This assumes that the system's equilibrium point is relatively stable — that the GPU's processing rate and the synthesis rate are roughly matched under normal conditions.

Another assumption is that normalized error is the right signal. By dividing by target, the assistant ensures that the controller's behavior scales with the target queue depth. If the target were changed from 8 to 16, the same absolute error would produce a smaller normalized error, keeping the response proportionate. This is a reasonable design choice, but it implicitly assumes that the system's dynamics are linear in the target — that a queue of 16 behaves similarly to a queue of 8, just scaled.

Knowledge Flow

The input knowledge required to understand this message includes: control theory fundamentals (how PI controllers work, the role of proportional and integral terms, the concept of integral windup and saturation), the architecture of the CuZK proving pipeline (the dispatch pacer, GPU workers, synthesis workers, the memory budget system), and the specific failure mode reported by the user (integral saturation causing pipeline drain).

The output knowledge created by this message is the validated tuning parameters and the behavioral expectations for the pitune1 binary. The assistant has produced a precise, scenario-by-scenario specification of how the controller should behave under various conditions. This serves as both a design document and a test oracle — if the deployed binary behaves differently from these predictions, that discrepancy is itself valuable diagnostic information.

The message also creates knowledge about the design rationale. The explicit reasoning — "integral as a gentle slow-drift corrector while P handles the fast response" — is an architectural decision that future developers can refer to when modifying the pacer. The asymmetric clamping rationale — "slowing down is dangerous" — encodes hard-won operational experience into the code's design narrative.

The Deployment Step

After completing the mental simulation, the assistant proceeds to build and deploy the binary. The Docker build output shows a clean build with image hash 85dfb1dbff94ba76db825fdb042b009ad7bb634b1a1519fa7fb9871b75ab333b, tagged as cuzk-rebuild:pitune1. This is the first in a series of tuning iterations (the conversation continues with pitune2, pitune3, and pitune4), reflecting the empirical nature of controller tuning — no amount of mental simulation can fully replace real-world testing with actual GPU workloads and memory pressure.

The fact that the assistant deploys as pitune1 rather than attempting further refinement before deployment reveals a pragmatic engineering philosophy: get the first candidate into production quickly, gather data, and iterate. The mental simulation provides confidence that the changes won't make things catastrophically worse, but the real validation comes from the live system.

Conclusion

Message [msg 3613] is a masterclass in applied control theory for systems engineering. It demonstrates how abstract mathematical reasoning — working through PI controller equations with hypothetical queue depths — can guide the tuning of a complex production system. The assistant's systematic walkthrough of five operating scenarios, each with computed corrections and rate multipliers, shows a methodical approach to validation that bridges the gap between code changes and deployed reality.

The message also illustrates a crucial insight about PI controller design for computing systems: the integral term, often treated as a mathematical convenience, must be carefully constrained when the cost of negative error is asymmetric. In the CuZK pipeline, slowing down too much is far worse than speeding up too much — a stalled pipeline loses all its parallelism and must be laboriously refilled. The asymmetric integral clamping encodes this asymmetry directly into the controller, making it a domain-aware rather than generic PI controller.

This message is not just about tuning constants. It is about understanding the physics of the system — the flow of partitions through synthesis into the GPU queue, the bursts caused by memory pressure, the time constants of bootstrap and steady-state operation — and translating that understanding into mathematical constraints that produce stable, predictable behavior.