The Integral Saturation Problem: Tuning a PI Controller for GPU Dispatch Under Memory Pressure

Introduction

In the course of building a high-performance GPU proving pipeline for Filecoin's zk-SNARK workloads, the development team behind the CuZK engine encountered a subtle and stubborn control systems problem. The GPU dispatch pacer — a PI (Proportional-Integral) controller responsible for regulating how quickly synthesized proofs are sent to the GPU for final proving — was exhibiting integral saturation. The integral term, which should provide gentle drift correction over time, was slamming into its limits almost immediately after every perturbation, rendering it useless as a control mechanism. Message [msg 3646] captures the moment when the assistant, prompted by the user's observation that "integral still saturates," works through the mathematics of the problem, considers multiple alternative approaches, and arrives at a solution: radically lower the integral gain while radically widening the integral bounds, transforming the integral from a fast-acting but quickly-saturated accumulator into a slow trim that floats in a useful range over minutes rather than seconds.

This message is a fascinating window into the real-world application of control theory to systems engineering. It shows the assistant reasoning aloud about differential equations, saturation dynamics, and the subtle interplay between integral accumulation rate and clamping limits — all in service of fixing a production GPU pipeline that was stalling under memory pressure.

Context: The GPU Dispatch Pacer

To understand message [msg 3646], we need to understand what came before it. Over the course of several days and multiple deployment iterations, the team had been battling GPU underutilization in the CuZK proving engine. The pipeline works in two stages: first, proof partitions are synthesized (CPU-bound work that produces intermediate representations), and then those synthesized jobs are dispatched to the GPU for final proving. The GPU is the expensive resource — two NVIDIA GPUs on a 755 GB RAM machine — and keeping it busy is the top priority.

The dispatch pacer is a PI controller that regulates the rate at which synthesized jobs are sent to the GPU. Its core loop is:

Measure the GPU queue depth (waiting): how many jobs are waiting for the GPU to process.
Compute a target queue depth (based on GPU processing rate and synthesis throughput).
Calculate the error: target - waiting.
Apply PI control: rate_mult = 1.0 + kp * norm_error + ki * integral.
Use rate_mult to adjust the dispatch interval: interval = base_interval / rate_mult. When waiting is too low (GPU is starving), rate_mult goes above 1.0, shortening the dispatch interval and sending jobs faster. When waiting is too high (GPU is backlogged), rate_mult drops below 1.0, lengthening the interval and slowing dispatch. The previous iteration (committed as 2bf16cd6 in [msg 3631]) had introduced several improvements: normalizing the error by the target value, adding asymmetric integral clamping (positive 2.0, negative -0.5), and fixing re-bootstrap logic. But as the user observed in [msg 3645], the integral was still saturating — hitting its +2.0 or -0.5 limits — which meant it wasn't providing useful control authority.

The Message: Reasoning Through Integral Saturation

Message [msg 3646] opens with the assistant's internal reasoning, presented as an "Agent Reasoning" block. This is the raw thinking process of the AI as it works through the problem. The assistant begins by restating the user's observation:

The user is saying the integral is still saturating at the limits (2.0 positive, -0.5 negative), which means it's not providing useful control authority. They want a higher cap (wider range) but lower ki so the integral accumulates more slowly and floats in a useful range instead of pinning to the limits.

This restatement is important because it shows the assistant correctly interpreting the user's somewhat terse message ("Seems like higher integral cap but lower value (possibly much?) could help as integral still saturates"). The user is proposing the right general approach — widen the bounds, lower the gain — but the assistant needs to work out the precise numbers.

The assistant then does the math:

Current: ki=0.02, max_integral_pos=2.0, max_integral_neg=-0.5 - Max I contribution: 0.02 2.0 = 0.04 (positive), 0.02 -0.5 = -0.01 (negative)

This reveals the core problem. With a maximum positive integral contribution of only 0.04 (added to the rate_mult multiplier), the integral term can barely influence the dispatch rate. The proportional term (kp=0.5) dominates completely. But worse, the integral saturates almost instantly:

At steady error=1.0 (empty queue), the integral accumulates by the normalized error times dt each update, so it hits the 2.0 limit in just a couple seconds rather than taking longer to settle into a useful range.

The assistant identifies the root cause: the integral accumulates as norm_error * dt per update tick. With a 2-second bootstrap interval and maximum error of 1.0, the integral grows by 2.0 in a single step — slamming into the +2.0 cap immediately. The integral is either pinned at +2.0 or at -0.5, never floating freely in between.

The Thinking Process: Three Approaches Considered

What follows is a remarkable display of iterative reasoning. The assistant considers three distinct approaches to fixing the saturation problem.

Approach 1: Bake ki into the accumulation. The assistant initially considers a structural change to the PI controller: instead of accumulating the raw error and then multiplying by ki when computing the correction, accumulate ki * norm_error * dt directly. This would match the standard PID formulation u(t) = kp * e(t) + ki * ∫e(t)dt + kd * de/dt. The assistant writes:

The cleaner approach is to bake ki into the accumulation itself: accumulate ki * norm_error * dt instead, then use the integral term directly without multiplying by ki again.

This is a valid approach — many PI controllers are implemented this way. But the assistant quickly rejects it:

Actually, the current implementation is correct as-is—the real problem is just that the clamp values are too restrictive.

This is a key insight. The assistant recognizes that the controller structure isn't broken; the parameters are just wrong. The integral accumulation rate is fine; what's broken is that the clamps cut off the integral before it can accumulate to a useful value.

Approach 2: Moderate tuning. The assistant then considers a moderate retuning:

With typical norm_error around 0.5 and dt of 1-2 seconds, the integral grows roughly 0.5-1.0 per second, so to avoid saturation within a minute I'd need max_integral around 90. That gives a maximum correction of about 0.45 with ki = 0.005, which is reasonable.

This is the first concrete proposal: ki=0.005, max_integral_pos=90. But the assistant doesn't stop there. It continues refining.

Approach 3: Aggressive retuning. The assistant then considers an even more aggressive change:

But maybe I should try something simpler: ki = 0.002 with asymmetric bounds like max_integral_pos = 50 and max_integral_neg = -10. This would cap the positive correction at 0.10 and negative at -0.02, taking about 100 seconds to saturate positive and 20 seconds negative.

This is a 10x reduction in ki (from 0.02 to 0.002) and a 25x increase in the positive bound (from 2.0 to 50). The time to saturate goes from ~2 seconds to ~100 seconds — a 50x improvement.

But the assistant pushes further still:

I'm testing ki = 0.001 with much wider bounds—max_integral_pos = 100 and max_integral_neg = -20—which gives the same max corrections but takes 100 seconds to saturate at a sustained error of 1.0, accounting for the fact that norm_error can stay elevated during pipeline fill phases. ±100. It would genuinely float around a useful value.

This is the final proposal: ki=0.001 (20x lower), max_integral_pos=100 (50x higher), max_integral_neg=-20 (40x higher). The max correction stays roughly the same (+0.10 / -0.02), but the time to saturate goes from seconds to minutes. The integral can now float freely, providing genuine drift correction rather than pinning to a limit.

The Decision: Why This Approach Wins

The assistant's reasoning converges on the aggressive retuning for several reasons:

The integral should be a slow trim, not a fast actuator. The proportional term (kp=0.5) handles immediate corrections. The integral should only compensate for steady-state bias — drift that persists over many cycles. A fast-integrating, quickly-saturating integral doesn't serve this purpose.
The asymmetric bounds reflect the asymmetric cost of error. A positive integral (speeding up dispatch) is relatively safe — the worst case is a slightly overfull GPU queue. A negative integral (slowing down dispatch) is dangerous — it can drain the pipeline entirely, causing GPU starvation. Hence the asymmetric bounds: +100 for positive, -20 for negative.
The time-to-saturate matters. With the old parameters, the integral saturated in ~2 seconds at error=1.0. With the new parameters, it takes ~100 seconds. This means the integral can actually respond to medium-term trends (changes in proof complexity, memory pressure fluctuations) rather than slamming into a wall at the first perturbation.

Assumptions and Potential Mistakes

The assistant makes several assumptions in its reasoning:

Assumption 1: The integral accumulation formula is norm_error * dt per tick. This assumes that the update interval (dt) is the bootstrap interval (2 seconds) and that norm_error is computed as (target - waiting) / target. Both are correct based on the code, but the assistant doesn't re-verify this by reading the update function — it works from memory of the implementation.

Assumption 2: The saturation behavior is the root cause of the pipeline draining. The user's observation was that "when hitting the memory ceiling, the integral went deeply negative and caused the pipeline to fully drain." The assistant assumes that fixing the saturation will prevent this. But there's a subtlety: even with a slow integral, the proportional term can still cause aggressive backoff when the memory ceiling is hit (because budget backpressure stops synthesis, which drains the GPU queue, which makes waiting drop, which makes the P term slow dispatch). The integral fix addresses the integral windup problem but may not fully prevent pipeline drain under memory pressure.

Assumption 3: The asymmetric bounds (+100 / -20) are sufficient. The assistant calculates that at error=0.5, it takes ~200 seconds to saturate positive and ~40 seconds negative. This assumes sustained error. If the error persists longer (e.g., a multi-minute memory pressure event), the integral could still saturate. The assistant implicitly assumes that such long-duration events are rare or that the proportional term will have corrected the error by then.

Potential mistake: The assistant doesn't consider derivative control. A full PID controller would add a derivative term (kd) to anticipate changes and dampen oscillations. The assistant stays with PI, which is simpler but may be more prone to overshoot. Given the success of the deployed fix (the user later says "Seems to work pretty well" in [msg 3641]), this wasn't a problem in practice, but it's a notable omission from the reasoning.

Input Knowledge Required

To fully understand message [msg 3646], the reader needs:

PI controller theory: Understanding of proportional and integral terms, how they combine, and what saturation means.
The specific CuZK pipeline architecture: That there's a two-stage pipeline (synthesis → GPU dispatch), that the pacer controls dispatch rate, and that waiting measures GPU queue depth.
The history of previous tuning attempts: The normalized error formula, the asymmetric clamping, the re-bootstrap logic — all introduced in earlier messages.
The deployment context: That the system runs on a 755 GB RAM machine with 2 GPUs, that memory budget is ~400 GB, and that hitting the memory ceiling causes synthesis to stall.
The bootstrap mechanism: That when the pipeline is empty, the pacer enters a "bootstrap" mode with a fixed 2-second dispatch interval to fill the pipeline before switching to PI control.

Output Knowledge Created

Message [msg 3646] creates several important outputs:

A clear mathematical analysis of why the integral saturates: the accumulation rate (norm_error * dt) exceeds the clamp limits in a single update tick.
A specific parameter proposal: ki=0.001, max_integral_pos=100, max_integral_neg=-20.
A design principle: The integral should be a slow trim that takes minutes to saturate, not a fast actuator that pins immediately.
A comparison table (in the follow-up message [msg 3656]) showing old vs. new parameters and their effects. The message also implicitly validates the user's intuition. The user suggested "higher integral cap but lower value" — the assistant works through the math and arrives at exactly that solution, but with specific numbers that make it work.

The Follow-Up: Implementation and Deployment

After message [msg 3646], the assistant reads the current parameters in engine.rs (the read tool call at the end of the message) and then edits them in [msg 3648]. The edit applies the new parameters. The assistant then builds and deploys as pitune3 ([msg 3650]-[msg 3654]).

The follow-up message [msg 3656] provides a clear comparison table:

| | Old | New | |---|---|---| | ki | 0.02 | 0.001 (20x lower) | | max_integral_pos | 2.0 | 100.0 (50x higher) | | max_integral_neg | -0.5 | -20.0 (40x higher) | | Max I correction | +0.04 / -0.01 | +0.10 / -0.02 | | Time to saturate (error=0.5) | ~4s / ~1s | ~200s / ~40s |

This table crystallizes the reasoning from message [msg 3646] into concrete, measurable improvements. The time-to-saturate numbers — 200 seconds vs 4 seconds — are the key metric. The integral can now actually float.

Broader Significance

Message [msg 3646] is a case study in applied control theory for systems engineering. It demonstrates several important principles:

The importance of understanding saturation dynamics. A PI controller with clamped integral is only as good as its clamp limits. If the integral hits its limits faster than the system's characteristic time constants, the integral term is effectively disabled. The assistant's key insight was recognizing that the saturation time (~2 seconds) was far shorter than the pipeline's fill time (30-60 seconds), meaning the integral was never providing useful correction during the critical pipeline-fill phase.

The value of mathematical reasoning over trial-and-error. Rather than blindly tweaking parameters and redeploying, the assistant works through the math: integral_growth = norm_error * dt. This allows precise calculation of saturation times and informed parameter selection.

The asymmetry of control in production systems. The assistant recognizes that positive and negative integral bounds should not be symmetric. Speeding up dispatch (positive integral) is relatively safe; slowing down dispatch (negative integral) risks draining the pipeline and starving the GPU. The bounds reflect this: +100 for positive, -20 for negative.

The iterative nature of tuning. This is the fourth iteration of PI tuning (pitune1 through pitune4). Each iteration reveals new dynamics: the re-bootstrap spam, the integral saturation, the synthesis concurrency cap. The assistant doesn't try to solve everything at once; it addresses each problem as it surfaces.

Conclusion

Message [msg 3646] captures a pivotal moment in the tuning of a production GPU dispatch pacer. The assistant, prompted by the user's observation that the integral was still saturating, works through a detailed mathematical analysis of the PI controller's dynamics, considers three alternative approaches, and converges on an aggressive retuning that transforms the integral from a fast-saturating, useless accumulator into a slow trim that can float freely over minutes. The reasoning is precise, the math is explicit, and the solution is grounded in control theory principles.

The message also demonstrates something more subtle: the value of an AI assistant that can reason aloud about differential equations and saturation dynamics in the context of a real production system. The assistant doesn't just apply a formula — it understands why the formula applies, when it breaks down, and how to fix it. That understanding is what makes the difference between a blind parameter tweak and a principled engineering solution.