The Art of PI Tuning: Deploying a Fix for Integral Saturation in a GPU Dispatch Pacer

Introduction

In the complex world of high-performance GPU proving for zero-knowledge cryptography, even a single line of code can determine whether a pipeline hums along efficiently or collapses into idle stalls. Message 3650 captures one such moment: the deployment of a PI controller tuning iteration, codenamed pitune3, for the CuZK proving engine's dispatch pacer. On its surface, the message is unremarkable—a Docker build completes, a new binary is tagged and pushed. But beneath this routine deployment lies a deep debugging journey through control theory, integral saturation, and the subtle dynamics of a multi-worker GPU pipeline.

The Context: A Saga of Dispatch Pacing

To understand message 3650, we must step back into the broader narrative. The CuZK proving engine had been suffering from GPU underutilization—the GPU would sit idle while the CPU synthesized proof constraints, creating a bottleneck that wasted expensive hardware. The solution was a dispatch pacer: a feedback controller that modulated how aggressively new synthesis jobs were sent to the GPU, aiming to keep the GPU's work queue at a target depth.

This pacer had evolved through multiple iterations. Initially a simple semaphore-based throttle, it graduated to a PI (Proportional-Integral) controller, then gained an EMA (Exponential Moving Average) feed-forward for GPU processing time estimates, and finally acquired a re-bootstrap mechanism to handle pipeline drains between proof batches. Each iteration brought improvements, but also revealed new pathologies.

By the time we reach message 3650, the pacer is on its third PI tuning iteration. The previous deployment (pitune2) had introduced normalized error computation and asymmetric integral clamping, but the user reported a persistent problem: the integral term was still saturating, pinning to its limits and losing the ability to provide useful control authority.

The Problem: Integral Saturation

The integral term in a PI controller accumulates past errors, allowing the controller to eliminate steady-state bias. But if the integral's range is too tight, it can saturate—hit its maximum or minimum value—and become effectively stuck. When saturated, the controller loses its ability to respond to changing conditions, like a thermostat that's pegged at maximum heat and can't modulate down.

In the pitune2 configuration, the parameters were:

ki (integral gain): 0.02
max_integral_pos: 2.0
max_integral_neg: -0.5 The assistant's analysis in the preceding message (message 3646) revealed the severity of the problem. With a bootstrap interval of 2 seconds and a normalized error of 1.0 (representing an empty GPU queue), the integral would grow by norm_error × dt = 1.0 × 2.0 = 2.0 in a single update tick. This meant the integral hit the positive cap of 2.0 in one update. The negative cap of -0.5 was even more restrictive, reached in a fraction of a second if the queue was overfull. When the integral saturates this quickly, it cannot serve its purpose. Instead of gradually accumulating error and providing a nuanced correction, it slams into the ceiling and stays there, effectively reducing the controller to a P-only (proportional) system. The integral term becomes a constant offset rather than a dynamic adjustment.

The Decision: Lower ki, Wider Bounds

The assistant's reasoning in message 3646 shows a careful consideration of the mathematics. The key insight was that the integral accumulation rate needed to be slowed dramatically, and the bounds needed to be widened to give the integral room to "float" in a useful range.

The chosen parameters were:

ki: 0.001 (reduced from 0.02, a 20× reduction)
max_integral_pos: 100.0 (increased from 2.0, a 50× increase)
max_integral_neg: -20.0 (increased from -0.5, a 40× increase) The assistant calculated the saturation time: at a sustained error of 1.0, the integral would take approximately 100 seconds to reach the positive limit. This is an eternity in the dispatch pacer's world—long enough that the integral would genuinely float at a useful value, providing subtle drift correction rather than slamming into a wall. The maximum correction from the integral term also changed. Previously, ki × max_integral = 0.02 × 2.0 = 0.04 for positive corrections. Now, 0.001 × 100.0 = 0.10—actually more authority, but delivered gradually over 100 seconds rather than instantly. The negative correction went from 0.02 × -0.5 = -0.01 to 0.001 × -20.0 = -0.02, doubling the available slowdown while making it accumulate slowly. This is a fundamentally different philosophy. The old parameters treated the integral as a fast-acting trim that could be slammed to its limits. The new parameters treat it as a slow, gentle drift corrector that accumulates over many seconds, never saturating under normal operation.

The Deployment: Message 3650

With the code change compiled and verified clean (message 3649), the assistant proceeds to build and deploy. Message 3650 shows the Docker build completing successfully:

DOCKER_BUILDKIT=1 docker build --no-cache -f Dockerfile.cuzk-rebuild -t cuzk-rebuild:pitune3 . 2>&1 | tail -5

The build uses DOCKER_BUILDKIT=1 for BuildKit's improved caching and performance, --no-cache to force a clean build (ensuring no stale artifacts from previous builds), and the -f Dockerfile.cuzk-rebuild flag to select the rebuild-specific Dockerfile. The image is tagged cuzk-rebuild:pitune3, continuing the naming convention that tracks each tuning iteration.

The output shows the final stages of the build: exporting layers, writing the image SHA, and naming it. The build completed in 0.1 seconds for the final export stage (the actual compilation happened in earlier stages, not shown in the tail -5 output).

This message is notable for what it doesn't show. The assistant doesn't include the scp transfer to the remote machine, the process kill, or the restart—those steps were part of the deployment ritual established in earlier iterations (see message 3635-3638 for the pattern). The message assumes the reader knows the deployment pipeline: build the Docker image, extract the binary, scp it to the remote host, kill the old process, and start the new one.

The Assumptions and Risks

Every deployment carries assumptions. The assistant assumes:

The build is correct: The cargo check that passed in message 3649 guarantees type safety and compilation, but not runtime correctness. The PI controller could still behave unexpectedly under real workload conditions.
The Docker build environment matches the target: The binary is built inside a Docker container and then extracted. If the container's CUDA runtime, glibc, or system libraries differ from the target machine (141.0.85.211), the binary could crash or exhibit undefined behavior.
The PI tuning is an improvement: The assistant believes that lower ki and wider bounds will solve the saturation problem, but this is a hypothesis. The integral might now be too slow to react, or the wider bounds might allow the integral to drift to extreme values that cause instability when the error direction reverses.
The deployment is safe: Killing the old process and starting a new one mid-workload could lose in-flight proofs. The assistant doesn't show any graceful shutdown or state preservation logic.

The Deeper Significance

Message 3650 represents a microcosm of the engineering process in high-performance systems. It's not just about changing a few constants—it's about understanding the dynamics of a feedback system, diagnosing a subtle pathology (integral saturation), reasoning about the mathematics of accumulation rates and saturation times, and iterating toward a solution.

The message also reveals the rhythm of the development cycle: analyze → hypothesize → code → compile → build → deploy → observe → repeat. Each iteration (pitune1, pitune2, pitune3) tightens the loop, with the assistant and user collaborating to narrow in on the right parameters.

For the reader unfamiliar with PID control, this message offers a glimpse into the practical challenges of applying control theory to software systems. The textbook PI controller is simple—two gains, an accumulator, a clamp—but making it work in a real system with discrete time steps, variable delays, and noisy measurements requires careful tuning and an understanding of how each parameter interacts with the system's dynamics.

Conclusion

Message 3650 is a deployment message, but it's also a milestone. It marks the third attempt to tune the PI controller's integral term, informed by a deep analysis of saturation dynamics. The assistant moved from a fast-acting integral that saturated instantly to a slow, gentle integrator with wide bounds, fundamentally changing the controller's behavior. Whether this iteration succeeds or reveals new pathologies, the process of systematic tuning, deployment, and observation is the engine that drives engineering progress in complex systems.

The next message in the conversation will tell us whether pitune3 was the fix that stuck, or whether the saga continued. But for now, message 3650 stands as a testament to the patience and precision required to tune a feedback controller for a GPU proving pipeline—one Docker build at a time.