The Art of Integral Saturation: Deploying a PI Controller Tuning for GPU Dispatch

In the high-stakes world of GPU-accelerated zero-knowledge proving, every microsecond of GPU idle time is lost revenue. The cuzk proving engine's dispatch pacer — a Proportional-Integral (PI) controller responsible for regulating how quickly synthesis tasks are fed to the GPU — had been the subject of intense iterative refinement across multiple deployments. Message [msg 3654] captures a seemingly mundane moment in that journey: the deployment of pitune3, the third tuning iteration. But beneath the simple nohup command lies a deep story about integral saturation, control theory mathematics, and the experimental discipline required to tune a production system.

The Message

The subject message is terse — a single bash command executed over SSH:

[assistant] [bash] ssh -p 40612 root@[REDACTED] 'nohup /data/cuzk-pitune3 --config /tmp/cuzk-memtest-config.toml > /data/cuzk-pitune3.log 2>&1 & echo "started PID $!"'
started PID 185150

This launches the newly built pitune3 binary on a remote machine (IP redacted for privacy) with a memory test configuration, running it in the background via nohup with output redirected to a log file. The remote shell echoes back the PID: 185150. On its face, this is routine deployment infrastructure — build a Docker image, extract the binary, copy it via SCP, kill the old process, wait for memory to drain, start the new one. But this deployment is the culmination of a careful mathematical analysis that had just taken place in the preceding messages.

The Problem: Integral Saturation

To understand why this deployment matters, we need to rewind to the previous iteration. The pitune2 binary, deployed just minutes earlier, had been running with the following PI controller parameters:

The Mathematical Root Cause

The assistant's reasoning in [msg 3646] reveals a thorough mathematical autopsy of the problem. The key insight was understanding how the integral accumulated in each update cycle. The PI controller's integral term grew at a rate of norm_error × dt per update — the normalized error multiplied by the time delta between updates. With a bootstrap interval of approximately 2 seconds (the time between dispatch pacer updates during pipeline warmup) and a maximum normalized error of 1.0 (representing a completely empty GPU queue), the integral would grow by 2.0 in a single update tick.

The problem becomes immediately clear: with max_integral_pos = 2.0, the integral hits its positive saturation limit in one update. It cannot grow further. Any sustained error beyond a single tick is invisible to the integral term. The integral is effectively disabled — it pins to the limit and stays there, providing no additional corrective action over time.

The assistant walked through several candidate solutions in its reasoning trace. One option was to restructure the controller entirely, baking ki into the accumulation itself (accumulating ki × norm_error × dt instead of norm_error × dt, then using the accumulated value directly without multiplying by ki again). This would match the standard PID formulation u(t) = kp × e(t) + ki × ∫e(t)dt. But the assistant correctly recognized that the existing structure was mathematically sound — the problem was purely one of tuning, not architecture.

The Fix: Slower Accumulation, Wider Range

The solution implemented in [msg 3648] was elegantly simple:

The Deployment Pipeline

The deployment sequence preceding the subject message reveals the operational discipline behind the tuning. In [msg 3650], the assistant builds a Docker image using DOCKER_BUILDKIT=1 docker build --no-cache -f Dockerfile.cuzk-rebuild -t cuzk-rebuild:pitune3 ., a full rebuild to ensure no caching artifacts. In [msg 3651], the binary is extracted from the image via a temporary container (docker create, docker cp, docker rm), then copied to the remote server via SCP, and made executable. In [msg 3652], the old process (PID 176416 from the pitune2 deployment) is killed. In [msg 3653], the assistant waits 90 seconds and checks free -g to confirm memory has been released by the old process before starting the new one — a thoughtful precaution to avoid memory pressure during the transition.

Only then, in the subject message, does the assistant start the new binary. This careful sequencing — build, extract, transfer, kill, wait, verify, start — reflects an understanding that production experimentation requires clean state transitions. A rushed deployment that starts the new binary before the old one has fully released its GPU memory allocations could lead to OOM kills or corrupted state.

Assumptions and Potential Pitfalls

The tuning change rests on several assumptions worth examining. First, it assumes that the normalized error calculation — (target - waiting) / target — produces a signal in the [-2, +1] range that is appropriate for integral accumulation. If the error signal has different dynamics than expected (e.g., if noise or oscillations cause rapid sign changes), the integral may still struggle to find a useful operating point.

Second, the asymmetric saturation limits (+100 / -20) encode a deliberate design choice: the system should be much more willing to speed up (positive integral) than to slow down (negative integral). This reflects the operational priority of keeping the GPU busy — GPU idle time is the primary cost. But it also means that if the system needs to apply sustained braking (e.g., during memory pressure events), the integral can only contribute -0.02 of correction, leaving the proportional term to do most of the slowdown work. The assistant's earlier comment that "backoff is P's job" makes this explicit.

Third, the tuning assumes that the 2-second bootstrap interval is representative of typical update intervals during steady-state operation. If the dispatch pacer updates more frequently during normal operation (e.g., every 500ms), the integral would accumulate faster than expected, potentially saturating sooner. Conversely, if updates are slower, the integral would accumulate more slowly, making it even less responsive.

The Broader Significance

This message is a microcosm of the experimental loop that characterizes production systems engineering in high-performance computing. The cycle is tight: hypothesize → code → build → deploy → observe → refine. Each iteration (pitune1, pitune2, pitune3, and later pitune4) represents a complete lap around this loop, with deployment times measured in minutes rather than days.

What makes this particular iteration noteworthy is the depth of the mathematical reasoning that preceded it. The assistant didn't blindly tweak parameters — it traced through the accumulation dynamics, calculated saturation times, evaluated alternative controller structures, and selected parameters that would give the integral genuine freedom to operate. The subject message, for all its surface simplicity, represents the moment when that analysis meets reality: when the binary starts, the logs begin flowing, and the hypothesis is tested against actual GPU dispatch behavior.

In the end, PID 185150 running on that remote machine would either prove the tuning correct or reveal new pathologies. That is the nature of experimental systems engineering — every deployment is a question posed to the production environment, and the answer arrives in log lines and performance metrics.