The Integral Saturation Problem: A User's Insight That Reshaped a PI Controller

Message Context

Subject Message (msg 3645): Seems like higher integral cap but lower value (possibly much?) could help as integral still saturates.

This brief, technically dense observation arrives at a pivotal moment in an intensive tuning session for a Proportional-Integral (PI) controller governing GPU dispatch in a zero-knowledge proof system called cuzk. The user has just deployed pitune2 — a build incorporating the assistant's previous PI tuning commit (2bf16cd6) — and confirmed it "seems to work pretty well." Yet rather than declaring victory, the user drills deeper. They have been observing the system's behavior under load and noticed a subtle but critical pathology: the integral term in the PI controller is still saturating. This message proposes a specific remedy — raise the integral cap while lowering the gain — and invites the assistant to explore the parameter space.

To understand why this message matters, we must reconstruct the technical landscape in which it lands.

The Problem: GPU Dispatch Under Memory Pressure

The cuzk engine processes zero-knowledge proofs through a pipeline: synthesis (CPU-bound work that generates proof components) followed by GPU proving (GPU-bound work that performs the cryptographic heavy lifting). The dispatch pacer is a PI controller that regulates how fast synthesis work items are fed to the GPU workers. Its target is to maintain a small queue of ready work at the GPU — enough to keep the GPU busy but not so much that memory overflows.

The PI controller computes a rate_mult — a multiplier on the dispatch interval — based on two terms:

The User's Observation: Integral Saturation

The user's message — "Seems like higher integral cap but lower value (possibly much?) could help as integral still saturates" — identifies that the integral term is still hitting its clamp limits. When the integral saturates, it ceases to be a dynamic control element. It becomes pinned at the boundary, unable to accumulate further. The PI controller effectively degrades to a P-only controller, losing the steady-state correction that the integral term is meant to provide.

The user proposes a counterintuitive solution: raise the cap (give the integral more room) but lower the gain (make it accumulate more slowly). The phrase "possibly much?" suggests the user suspects the adjustment needs to be dramatic — not a minor tweak but an order-of-magnitude change.

This is a sophisticated control theory insight. In a standard PI controller, the integral term accumulates as:

$$I(t) = ki \times \int e(t) dt$$

The effective correction from the integral is bounded by ki × max_integral. With ki=0.02 and max_integral_pos=2.0, the maximum positive correction is only 0.04 — barely perceptible. But the integral hits the 2.0 cap almost immediately: at norm_error=1.0 (empty queue) with a 2-second bootstrap interval, the integral grows by 2.0 in a single update step. It saturates instantly.

The user's intuition is that by widening the cap (say to 100) and lowering ki (say to 0.001), the integral can take minutes to saturate rather than seconds. It would float freely within its range, providing genuine control authority rather than pinning uselessly at a boundary.

Assumptions and Knowledge Required

To understand this message, one needs:

  1. Control theory fundamentals: Knowledge of PI controllers, integral windup, saturation, and the relationship between gain and clamp limits. The user assumes the assistant understands that integral saturation renders the I-term ineffective.
  2. The specific system architecture: Understanding that the cuzk pipeline has a synthesis stage feeding a GPU stage, that dispatch is rate-controlled, and that memory pressure from pinned allocations can cause backpressure.
  3. The history of the pacer: Awareness of the previous iterations — the synthesis throughput cap debacle (synthcap1/synthcap2), the re-bootstrap fix (synthcap3), and the PI tuning in pitune1/pitune2. The user's message builds directly on the pitune2 deployment.
  4. The asymmetric clamp trade-off: Understanding why the negative integral was clamped so aggressively (-0.5) — to prevent pipeline-draining backoff after memory ceiling slams — and recognizing that this asymmetry might be too restrictive.
  5. The deployment context: Knowing that pitune2 is running on a remote machine (141.0.85.211) with 755 GB of RAM and that the user has been observing its behavior through logs.

What the Message Does Not Say (But Implies)

The message is deceptively brief. It does not:

With the current ki=0.02 and max_integral_pos=2.0, the integral saturates almost immediately. At norm_error=1.0 (empty queue) with dt=2s bootstrap interval, the integral grows by 2.0 per tick — hits the cap in one update.

The assistant then works through the parameter trade-space, ultimately implementing:

Mistakes and Incorrect Assumptions

The user's message contains no factual errors — it is a correct observation about integral saturation. However, it implicitly assumes that widening the cap alone is sufficient. The assistant's reasoning reveals a deeper issue: the integral accumulation itself needs to be reconsidered. The assistant briefly considers a structural change — baking ki into the accumulation itself (standard PID form) — but ultimately stays with the current architecture and just adjusts parameters.

The user also assumes that the asymmetric clamp (tight negative) is the right approach in principle, just too restrictive. But the subsequent deployment of pitune3 (with the wider asymmetric bounds) still had issues, leading to further iterations (pitune4, the synthesis concurrency cap, etc.). The integral saturation fix was necessary but not sufficient.

Output Knowledge Created

This message directly produces:

  1. A parameter change in the PI controller: The assistant edits engine.rs to modify ki, max_integral_pos, and max_integral_neg, then builds and deploys pitune3 (<msg id=3648-3655>).
  2. A documented parameter comparison: The assistant produces a table contrasting old vs. new values ([msg 3656]), which becomes part of the project's operational knowledge.
  3. A deeper understanding of the system's dynamics: The exchange surfaces the time constants involved — that the integral saturates in one update step at the current settings — which informs all subsequent tuning.
  4. A template for future tuning discussions: The user's message establishes a pattern of proposing structural parameter changes rather than just reporting symptoms, elevating the conversation from debugging to control system design.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in [msg 3646] reveals several layers of analysis:

  1. Problem identification: "The integral is still saturating at the limits (2.0 positive, -0.5 negative), which means it's not providing useful control authority."
  2. Quantitative analysis: The assistant computes the maximum I contribution (ki × max_integral) and the accumulation rate (norm_error × dt per update).
  3. Structural reconsideration: The assistant briefly considers changing the accumulation formula to standard PID form (accumulating ki × error × dt instead of raw error), demonstrating awareness that the current implementation is non-standard.
  4. Trade-space exploration: The assistant works through several parameter combinations before settling on ki=0.001, max_integral_pos=100, max_integral_neg=-20.
  5. Validation: The assistant checks that the maximum correction is reasonable (+0.10 / -0.02) and that the saturation time is appropriate (~100 seconds at error=1.0). This reasoning shows that the assistant treated the user's message not as a command but as a hypothesis to be tested against the mathematics of the system. The phrase "possibly much?" in the user's message is interpreted as an invitation to explore the parameter space, not a specific request.

Conclusion

The user's message at [msg 3645] is a masterclass in concise, high-leverage technical communication. In 13 words, it identifies a control system pathology (integral saturation), proposes a remedy (higher cap, lower gain), and opens a design space ("possibly much?"). It assumes the assistant has the domain knowledge to understand the implication — that the integral is hitting its limits and therefore providing no useful correction — and trusts the assistant to work through the mathematical implications.

The message succeeds because it speaks the language of the system: control theory, parameter bounds, time constants. It does not describe symptoms ("the GPU queue is oscillating") but rather diagnoses the mechanism ("integral still saturates"). This is the difference between reporting a bug and proposing a fix — and it's what makes this single message a turning point in the tuning session. The resulting deployment (pitune3) was not the final answer, but it was a necessary step toward a stable, well-tuned dispatch pacer that could handle the memory pressure and pipeline dynamics of production zero-knowledge proof generation.