Tuning the PI Pacer: Normalized Error, Asymmetric Integral Clamping, and the Art of Gentle Backpressure

In the high-stakes world of GPU-accelerated zero-knowledge proving, the difference between a smoothly humming pipeline and a collapsed one often comes down to a handful of floating-point constants. Message [msg 3620] captures one of those pivotal moments: the assistant, after deploying a major redesign of the dispatch pacer (committed as e0ea675b), receives feedback from the user that the PI controller's integral term is misbehaving. When the system "slams into the memory ceiling," the integral goes deeply negative, causing the pipeline to fully drain and restart from scratch. The user's diagnosis is precise: the integral is almost always saturated, the backoff is too aggressive, and the P term may also need tuning. The assistant's response—documented in this message—is a carefully reasoned set of parameter changes that transform the controller from a brittle, over-reactive system into a gentle, resilient one.

The Subject Message

The message itself is deceptively concise. After building, deploying, and starting the pitune1 binary (PID 166807), the assistant writes:

Running — PID 166807. The PI tuning changes:

>

| Parameter | Old | New | Rationale | |---|---|---|---| | Error | raw target - waiting (0-8) | normalized (target - waiting)/target (±1) | Gains independent of target | | kp | 0.1 | 0.5 | P now does the heavy lifting on normalized error | | ki | 0.008 | 0.02 | Gentle drift correction | | max_integral | ±20.0 (symmetric) | +2.0 / -0.5 (asymmetric) | Negative integral barely slows — prevents pipeline drain | | rate_mult clamp | [0.1, 5.0] | [0.3, 3.0] | Tighter bounds — max 3.3x slowdown, max 3x speedup |

>

The key change: even in the worst case (queue massively overfull), dispatch only slows to 1/0.3 = 3.3x GPU rate — it never stalls. And the integral can barely go negative (-0.5), so it can't accumulate into a pipeline-draining backoff.

Five parameters changed. A table. A one-sentence summary. But behind this compact summary lies a deep understanding of control theory, GPU pipeline dynamics, and the specific failure modes of the cuzk proving engine.

The Problem: Integral Saturation and the Pipeline Drain

To understand why this message was written, we must first understand what was happening before it. The dispatch pacer is a PI (proportional-integral) controller that regulates how fast the CPU synthesizes new proof partitions and dispatches them to the GPU queue. Its job is to maintain a target number of "waiting" partitions in the GPU queue—enough to keep the GPU busy, but not so many that memory overflows.

The previous iteration (synthcap3, deployed in the preceding messages) had introduced a major pacer redesign: it removed a problematic synthesis throughput cap that created a vicious cycle of collapse, added re-bootstrap detection for when the pipeline drains between batches, and switched to measuring actual GPU processing time for the feed-forward term. But it retained the original PI tuning, and that tuning had a fatal flaw.

The user's report in [msg 3601] identified the problem with surgical precision. When the system hits the memory ceiling—meaning the pinned memory pool is exhausted and budget.acquire() blocks—many concurrent syntheses complete at roughly the same time and flood the GPU queue. The waiting count spikes well above the target of 8, the error term goes deeply negative, and the integral term plummets to its negative saturation limit of -20. Once the burst is consumed and the queue drains, the deeply negative integral keeps the dispatch rate crippled—10x slower than GPU rate—causing the pipeline to fully drain before synthesis can resume. The system essentially "starts from scratch," losing all the pipeline parallelism it had carefully built up.

The assistant's analysis in [msg 3606] walks through the math. With target=8 and ki=0.008, a sustained error of +8 (empty queue) accumulates integral at 0.064 per second. But when the error is -12 (queue overfull at 20 waiting), the integral plummets at 0.096 per second. In seconds, the integral hits -20. And with kp=0.1, the P term alone contributes 0.1 * (-12) = -1.2 to the correction, which already drives rate_mult below zero, getting clamped to 0.1—a 10x slowdown. The integral just makes it worse and keeps it stuck.

The Reasoning: Five Interlocking Changes

The assistant's response in [msg 3607] lays out a four-point plan, which then becomes the five-parameter change set shown in the subject message. Each change is individually motivated, but they work together as a system.

Normalizing the error is the most conceptually significant change. Previously, the error was computed as target - waiting, which for a target of 8 gives a raw range of roughly -20 to +8. This meant the PI gains were implicitly calibrated to this specific target value. If the target ever changed, the gains would need retuning. By dividing by the target, the error is normalized to approximately ±1, making the gains independent of the target value. This is a standard control engineering practice—error normalization—that decouples the tuning from the operating point.

Raising kp from 0.1 to 0.5 might seem counterintuitive: if the problem is over-reaction, why make the proportional gain larger? The key is that the error is now normalized. The old kp=0.1 with raw error of 8 gave a P contribution of 0.8. The new kp=0.5 with normalized error of 1.0 gives a P contribution of 0.5. So the P term actually becomes less aggressive in absolute terms for the same queue state, while being better scaled. The assistant's rationale is that "P now does the heavy lifting on normalized error"—meaning the proportional term handles fast corrections, while the integral term is relegated to gentle drift compensation.

Lowering ki from 0.008 to 0.02 is another apparent paradox: the integral gain increases. But this is paired with a drastic reduction in the integral's maximum value. The old max_integral of ±20 meant the integral could contribute up to 0.008 * 20 = 0.16 to the correction—a tiny fraction of the total. The new max_integral of +2.0/-0.5 means the integral contributes at most 0.02 * 2.0 = 0.04 on the positive side and 0.02 * (-0.5) = -0.01 on the negative side. The integral becomes a gentle nudger, not a dominant force.

Asymmetric integral clamping is the most novel insight. The assistant recognizes that positive integral (speeding up) and negative integral (slowing down) have fundamentally different consequences. A positive integral that overspeeds the dispatch by 2x is harmless—the GPU queue fills up, the budget blocks, and the system self-corrects. But a negative integral that slows dispatch to 0.1x GPU rate is catastrophic—the pipeline drains, proving throughput collapses, and the system must re-bootstrap from scratch. So the clamping is made sharply asymmetric: +2.0 on the positive side, -0.5 on the negative side. The integral can barely slow things down at all.

Tightening the rate_mult clamp from [0.1, 5.0] to [0.3, 3.0] reinforces the same philosophy. The old clamp allowed a 10x slowdown (rate_mult=0.1), which is exactly what caused the pipeline drain. The new clamp limits the slowdown to 3.3x (rate_mult=0.3), ensuring dispatch never stalls completely. The speedup is also capped at 3x instead of 5x, but this is less critical—the system is already bounded by the memory budget.

The Mental Simulation

Before building and deploying, the assistant performs a mental simulation in [msg 3613] that reveals the depth of the reasoning. This is not a casual check; it is a systematic walkthrough of five operating points:

Assumptions and Trade-offs

The tuning makes several implicit assumptions. First, it assumes that the memory budget (budget.acquire()) is the primary safety mechanism against memory exhaustion, not the PI controller. The controller's job is to maintain queue depth for GPU utilization, not to protect memory. This is a crucial design choice: the PI controller is demoted from a safety-critical role to a performance-tuning role.

Second, it assumes that the GPU rate feed-forward term (actual GPU processing time divided by number of workers) provides a good baseline dispatch rate. The PI correction is only a small modulation around this baseline. This means the controller's behavior is dominated by the feed-forward term in normal operation, with the PI loop only correcting for transient mismatches.

Third, it assumes that the system operates in a regime where occasional queue overfull events are tolerable. The asymmetric clamping means that when the queue does spike, the controller barely slows down—it trusts that the memory budget will block synthesis before memory is actually exhausted. This is a reasonable assumption given that the budget is a hard blocking mechanism, but it means the controller provides almost no anticipatory slowdown.

The trade-off is clear: the new tuning prioritizes pipeline stability and throughput over precise queue depth regulation. The queue depth may overshoot more during bursts, but the pipeline never drains. This is the right trade-off for a proving system where throughput is the primary metric and memory protection is handled by a separate mechanism.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains. Control theory is the most obvious: understanding what a PI controller is, how the proportional and integral terms interact, what integral saturation means, and why error normalization is beneficial. Without this background, the parameter changes appear arbitrary.

GPU pipeline architecture is equally important. The concept of a "GPU queue" of synthesized partitions, the distinction between CPU-side synthesis and GPU-side proving, the role of pinned memory pools, and the dynamics of how concurrent synthesis completion can flood the queue—all of this context is necessary to understand why the integral goes negative and why that's a problem.

The specific cuzk engine architecture is also relevant. The DispatchPacer struct, the rate_mult field, the budget.acquire() mechanism, the re-bootstrap logic, and the GPU worker completion reporting—these are all components introduced in earlier messages that the assistant has been iterating on for days. The message assumes familiarity with this codebase.

Finally, some understanding of the deployment environment helps. The fact that the system runs on a vast.ai instance with 755 GB of memory, that the pinned memory pool can exhaust under high concurrency, and that the GPU has multiple workers processing in parallel—these details shape the tuning decisions.

Output Knowledge Created

This message creates a specific, actionable artifact: the pitune1 binary deployed to /data/cuzk-pitune1 on the remote machine, along with a documented set of PI parameters that can be referenced, compared, and further tuned. The table in the message serves as a permanent record of the tuning rationale—a form of documentation that is rare in fast-moving development sessions.

More broadly, the message establishes a design philosophy for the dispatch pacer: the PI controller should be gentle, the integral should be asymmetric, and the memory budget should be the primary safety mechanism. This philosophy shapes all subsequent tuning iterations (pitune2, pitune3, pitune4 mentioned in the chunk summary) and becomes the foundation for the production deployment that follows.

The message also creates knowledge about the system's failure modes. The insight that "negative integral is dangerous but positive integral is harmless" is not obvious a priori—it emerged from observing the system's behavior under memory pressure. This asymmetry is now encoded in the controller design and will inform future tuning efforts.

The Thinking Process

What is most striking about this message is what it doesn't say. The table and summary are the output of a reasoning process that is almost entirely invisible in the subject message itself. To see the thinking, one must look at the surrounding messages.

In [msg 3606], the assistant works through the math step by step, calculating integral accumulation rates for different scenarios. It identifies the root cause: "when items slam into memory ceiling, many concurrent syntheses complete at once and flood the GPU queue. waiting spikes well above target, error goes deeply negative, and integral plummets to -20 in seconds."

In [msg 3607], the assistant analyzes the useful range of the correction term and realizes that the old integral contributed almost nothing (0.16 out of 0.96) while the P term dominated. This leads to the insight that the integral should be relegated to gentle drift correction while P handles fast response.

In [msg 3613], the assistant performs the mental simulation, testing five operating points to verify the tuning behaves as expected. This simulation is the bridge between theory and practice—it catches potential issues before deployment.

The thinking process reveals a methodical, engineering-minded approach. The assistant does not guess at parameters or try random values. It works from first principles: compute the error ranges, calculate the gain contributions, identify the saturation points, and design the clamping to prevent the specific failure mode observed. This is control systems engineering applied to a real-world GPU pipeline, and the subject message is the concise summary of that engineering work.

Conclusion

Message [msg 3620] is a masterclass in targeted system tuning. In five parameter changes and a single sentence of explanation, the assistant transforms a PI controller from a pipeline-destroying liability into a gentle, stabilizing force. The changes are individually motivated, mutually reinforcing, and grounded in a deep understanding of both control theory and the specific dynamics of the cuzk proving engine. The message stands as a testament to the power of methodical reasoning in systems engineering—where the most impactful changes are often the smallest, and the best tuning is the one that prevents failure before it can happen.