The Compile Check That Closes the Loop: Verifying PI Controller Tuning in a GPU Dispatch Pacer

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proving, the difference between a smoothly humming pipeline and a collapsed one often comes down to a handful of floating-point constants. This article examines a single, seemingly mundane message in an opencode coding session — message index 3612 — where an AI assistant runs cargo check after implementing a series of PI (proportional-integral) controller tuning changes for a GPU dispatch pacer. On its surface, this is just a compile verification. But beneath that surface lies a rich story of control theory applied to systems engineering, the subtle art of integral anti-windup, and the iterative process of tuning a feedback controller under real-world constraints.

The Broader Context: A GPU Pipeline Under Pressure

To understand message 3612, we must first understand the system it serves. The assistant has been building a GPU-accelerated zero-knowledge proving engine called CuZK. At the heart of this engine lies a synthesis dispatch pipeline: CPU cores synthesize proof partitions, which are then dispatched to GPU workers for proving. The dispatcher must carefully regulate how many partitions are "in flight" — waiting in the GPU queue — to avoid two failure modes: starving the GPU (too few waiting partitions, leaving GPU cores idle) or overwhelming it (too many waiting partitions, causing memory pressure and latency spikes).

The assistant had already implemented a sophisticated PI controller as the DispatchPacer struct. This controller uses a feed-forward term (the measured GPU processing rate) plus a feedback correction based on the error between the target queue depth and the exponentially-weighted moving average (EMA) of actual waiting partitions. The PI controller computes a rate_mult value that scales the dispatch interval: rate_mult = (1.0 + correction).clamp(0.1, 5.0). A value of 1.0 means dispatch at exactly the GPU consumption rate; values above 1.0 speed up dispatch to fill the queue; values below 1.0 slow down dispatch to let the GPU drain.

The Problem: Integral Saturation and Pipeline Collapse

In the message immediately preceding our subject ([msg 3601]), the user reported a critical failure mode:

"Whenever we 'slam' into the memory ceiling integral goes negative. Whenever integral goes negative new partitions completely start entering synthesis until we fully drain all running and waiting pipelines, essentially starting from scratch, which seems wrong, the backoff shouldn't be nearly this aggressive. Also seems like integral is almost always saturated."

This is a textbook case of integral windup, a well-known phenomenon in control theory. When the system hits a constraint (in this case, the memory budget ceiling), the error persists, the integral term accumulates, and when the constraint is removed, the integral has grown so large that it takes an excessively long time to unwind. But the problem was even more insidious: the integral was going negative — not positive — and the negative integral was causing the dispatcher to slow to a crawl, draining the pipeline entirely before resuming synthesis.

The assistant analyzed the math in [msg 3606]:

• With target=8 (the desired number of waiting partitions), error ranges from +8 (empty queue) to -20 (overloaded queue)
• With ki=0.008 and max_integral=±20, the integral saturates at its bounds almost immediately
• When waiting spikes to 20 (error = -12), the P term alone (kp=0.1) produces a correction of -1.2, driving rate_mult to its floor of 0.1 (10× slower than GPU rate)
• The integral then rapidly accumulates even more negative value, compounding the problem The core insight: the integral was both too aggressive and too symmetric. It could accumulate just as much negative authority (slowing down) as positive authority (speeding up), but the consequences of negative integral were far more damaging — it could drain the pipeline and force a costly re-bootstrap.

The Fix: Four Interlocking Changes

In messages 3607 through 3611, the assistant implemented four changes to the PI controller:

1. Normalize the error. Instead of using raw error (target - ema_waiting), the assistant divided by target to produce a normalized error in the range [-1, +1] under normal conditions. This makes the controller's behavior independent of the target value — a crucial property for a system that might operate with different targets in different configurations.

2. Lower the proportional gain. The P term was reduced from kp=0.1 to kp=0.5 — but because the error is now normalized, the effective gain is actually lower. Previously, an error of 8 produced a P correction of 0.8. Now, a normalized error of 1.0 (queue empty) produces a P correction of 0.5. This prevents the P term from overreacting to transient spikes.

3. Asymmetric integral clamping. This is the most important change. The assistant set max_integral_pos = 2.0 and max_integral_neg = -0.5. The integral can accumulate positive correction (speeding up) up to 2.0, but can only accumulate negative correction (slowing down) to -0.5. This asymmetry reflects the engineering judgment that slowing down too aggressively is more dangerous than speeding up too aggressively. A slow dispatch can drain the pipeline and force a re-bootstrap; a fast dispatch just fills the GPU queue a bit more, which the memory budget will naturally cap.

4. Lower the integral gain. The integral gain was reduced from ki=0.008 to ki=0.02 — but again, because the error is now normalized, the effective accumulation rate is different. With normalized error of 0.5, the integral grows at 0.02 × 0.5 = 0.01 per second, reaching the positive cap of 2.0 after 200 seconds of sustained error. This makes the integral a very slow drift corrector, while the P term handles fast response.

Message 3612: The Verification Step

This brings us to the subject message itself. After making all four edits across multiple files, the assistant pauses to verify one thing: that the re-bootstrap logic already resets the integral to 0.0. This is critical because the re-bootstrap mechanism — which re-enters a slow-start phase when the pipeline drains — must clear the integral to prevent stale negative values from poisoning the new bootstrap cycle.

The assistant reads the relevant section of engine.rs and confirms:

"Good, the re-bootstrap already resets integral to 0.0, which is correct."

Then it runs cargo check — a compilation check that verifies the code is syntactically and type-correct without producing a full binary. The output shows only pre-existing warnings about private interface visibility (a Rust lint about JobTracker being less visible than the functions that use it). No errors. The changes compile cleanly.

Why This Message Matters

On its face, message 3612 is trivial: a developer verifying that a constant is correct and running a compiler check. But in the narrative of this coding session, it represents a crucial inflection point. The assistant has just completed a deep analytical pass through the PI controller's math, identified the root cause of a pipeline collapse, designed a multi-part fix involving error normalization, gain reduction, and asymmetric clamping, and implemented it across four edits. The compile check is the moment where all that analysis crystallizes into working code.

The message also reveals the assistant's engineering discipline. Before moving to the next step (building and deploying the binary), it takes the time to verify that the re-bootstrap integral reset is consistent with the new normalized integral scheme. With the old scheme, integral values could be as large as ±20, and resetting to 0 was clearly correct. With the new scheme, the integral range is [-0.5, 2.0], and 0 is still the right reset value — it represents a neutral state with no accumulated correction. The assistant explicitly confirms this rather than assuming.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the message itself. It opens with a confirmation statement ("Good, the re-bootstrap already resets integral to 0.0, which is correct"), which tells us that the assistant was thinking about consistency: "I just changed the integral range; does the reset logic still make sense?" This is a cross-cutting concern check, the kind of thing that separates careful engineering from hasty coding.

The assistant then runs cargo check — not cargo build or cargo test. This choice reveals an assumption: the changes are syntactically and type-correct, and the main risk is compilation errors from the edits. The assistant is optimizing for speed of verification. A full build would take minutes; cargo check is faster and sufficient to catch typos, type mismatches, and API misuses.

The output shows only warnings, not errors. The warnings about private_interfaces are pre-existing and unrelated to the PI tuning changes. The assistant does not address them, implicitly judging them as acceptable noise.

Assumptions and Potential Issues

The assistant makes several assumptions in this message and the preceding edits:

1. The normalized error scheme is correct. The assistant assumes that dividing by target produces a well-behaved error signal. But if target is 0 (a possible edge case), this would cause a division-by-zero. The code likely guards against this, but the assumption deserves scrutiny.
2. The asymmetric clamping bounds are appropriate. The values max_integral_pos = 2.0 and max_integral_neg = -0.5 are chosen based on engineering judgment, not formal analysis. They may need further tuning as the system encounters different workloads.
3. The re-bootstrap integral reset is sufficient. The assistant assumes that resetting the integral to 0 on re-bootstrap is correct. But if the integral had accumulated useful information about the steady-state error before the pipeline drained, resetting it loses that information. The assistant implicitly judges that the cost of stale integral values outweighs the benefit of preserved history — a reasonable but unverified assumption.
4. The compile check is sufficient validation. The assistant does not run any unit tests or integration tests for the PI controller. The only validation is that the code compiles. This is a minimal bar.

Input and Output Knowledge

Input knowledge required to understand this message includes: the structure of the DispatchPacer struct and its PI controller, the concept of integral windup in control theory, the Rust programming language and its compilation model, the CuZK GPU proving pipeline architecture, and the specific history of the pacer's development (the re-bootstrap mechanism, the EMA waiting tracker, the GPU rate feed-forward).

Output knowledge created by this message is modest but important: confirmation that the PI tuning changes compile cleanly, and confirmation that the re-bootstrap integral reset is consistent with the new normalized integral scheme. This knowledge enables the next step: building and deploying the binary as pitune1 for live testing.

Conclusion

Message 3612 is a quiet moment in a storm of iterative debugging. After diagnosing a pipeline collapse caused by integral saturation, the assistant designs and implements a multi-part fix involving error normalization, gain reduction, and asymmetric clamping. The compile check is the gate that must be passed before the fix can be deployed and tested. In its brevity — a confirmation statement and a compiler invocation — the message encapsulates the discipline of incremental engineering: analyze, implement, verify, deploy. The real story is not in the message itself but in the thinking that produced it, the control theory that informed it, and the system that it helps to stabilize.