The Deployment That Carried a PI Controller Tuning: Shipping pitune1 to Production

The Message

docker rm -f cuzk-pitune1 2>/dev/null; docker create --name cuzk-pitune1 cuzk-rebuild:pitune1 /cuzk && docker cp cuzk-pitune1:/cuzk /tmp/cuzk-pitune1 && docker rm cuzk-pitune1 && scp -P 40612 /tmp/cuzk-pitune1 root@141.0.85.211:/data/cuzk-pitune1 && ssh -p 40612 root@141.0.85.211 'chmod +x /data/cuzk-pitune1 && ls -lh /data/cuzk-pitune1'
ee7b0daca856d6fefd866f1fc451ba9c75a5b3ffc918c332e3ecbc66e190c28c
cuzk-pitune1
-rwxr-xr-x 1 root root 27M Mar 13 23:41 /data/cuzk-pitune1

At first glance, this message looks like nothing more than a routine deployment script — a chain of Docker and SSH commands that extracts a binary from a container image and ships it to a remote server. But this seemingly mundane deployment is the culmination of a deep, multi-hour debugging session into the dynamics of a PI (Proportional-Integral) controller governing GPU dispatch in a zero-knowledge proof system. The binary being deployed, cuzk-pitune1, carries within it a set of carefully calculated tuning parameters intended to fix a critical instability that was causing the entire proving pipeline to collapse under memory pressure.

The Context: A PI Controller Gone Wrong

To understand why this message exists, we must step back into the problem that precipitated it. The cuzk system (a GPU-accelerated zero-knowledge proving engine) uses a PI controller called DispatchPacer to regulate how quickly synthesized proof partitions are dispatched to the GPU for processing. The controller's job is to maintain a target number of partitions waiting in the GPU queue — not too few (which would starve the GPU) and not too many (which would exceed the memory budget). The feed-forward term uses the measured GPU processing rate, and the feedback terms (P and I) correct for deviations from the target queue depth.

In the previous iteration (deployed as synthcap3 at <msg id=3598>), the user reported a critical failure mode at <msg id=3601>:

"Whenever we 'slam' into the memory ceiling integral goes negative. Whenever integral goes negative new partitions completely start entering synthesis until we fully drain all running and waiting pipelines, essentially starting from scratch, which seems wrong, the backoff shouldn't be nearly this aggresive. Also seems like integral is almost always saturated, we should tune it somehow so that it would be floating around a value where it has useful control authority."

This is a classic control theory problem: integral saturation. When the memory ceiling was hit, a burst of completed syntheses would flood the GPU queue, causing the error term (target - ema_waiting) to go deeply negative. The integral term would accumulate this negative error, saturating at its negative clamp of -20. Once the burst was consumed and the queue drained back to normal levels, the deeply negative integral would keep the dispatch rate suppressed for far too long, causing the pipeline to fully drain and forcing a costly re-bootstrap. The integral had become a liability rather than a helpful correction mechanism.

The Reasoning Behind the Fix

The assistant's response at <msg id=3606> laid out a detailed analysis of the problem. Working through the math with a target queue depth of 8, the assistant identified several issues:

The error range was too large: With raw error values ranging from +8 (queue empty) to -20 (queue overfull), the P term (kp=0.1) could produce corrections from +0.8 to -2.0, already enough to drive rate_mult to its clamp limits of 0.1 and 5.0. The integral term contributed almost nothing in the positive direction (0.16 out of 0.96 total correction) while accumulating massive negative values.
The integral was too aggressive: With ki=0.008 and max_integral=±20, the integral could saturate at ±20, but the useful range of the correction was only about [-0.9, +4.0] (corresponding to rate_mult clamping at 0.1 and 5.0). The integral was operating in a regime where most of its range was useless — it would saturate long before contributing meaningfully.
Symmetric clamping was dangerous: Allowing the integral to go equally negative as positive meant that a single memory-pressure event could poison the controller for minutes afterward, forcing a complete pipeline drain and restart. The fix, designed across messages <msg id=3607> through <msg id=3610>, involved four key changes: Normalize the error: By dividing the error by the target queue depth, the error was mapped to a [-1, +1] range under normal operating conditions. This made the controller behavior predictable regardless of the target value and prevented the P term from overreacting to large absolute errors. Lower the proportional gain: The P term was adjusted so that even at maximum normalized error (±1.0), the correction would be moderate — speeding up dispatch by at most 50% or slowing it to half the GPU rate, rather than slamming into the rate_mult clamps. Asymmetric integral clamping: The positive integral limit was set to +2.0 (allowing gentle accumulation for sustained demand), while the negative limit was set to just -0.5 (preventing the integral from ever becoming a dominant braking force). This was the critical insight: slowing down too aggressively is far more dangerous than speeding up too aggressively, because a stalled pipeline requires an expensive re-bootstrap. Lower the integral gain: With ki=0.02 and the new normalized error, the integral accumulates slowly — at a steady error of 0.5, it would take 200 seconds to reach the positive limit of 2.0. The integral becomes a gentle, slow-drift corrector for sustained biases, while the P term handles fast transient response.

The Mental Simulation

Before building and deploying, the assistant ran a mental simulation at <msg id=3613> to verify the tuning:

Steady state (waiting≈target=8, norm_error≈0): P: 0.5 0 = 0, I: 0.02 ~0 = 0, correction = 0, rate_mult = 1.0. Dispatch at exactly GPU rate.

Queue half-empty (waiting=4, norm_error=0.5): P: 0.5 * 0.5 = 0.25, correction ≈ 0.25, rate_mult = 1.25. Dispatch 25% faster than GPU rate to fill up. Gentle.

Queue empty (waiting=0, norm_error=1.0): P: 0.5 * 1.0 = 0.5, correction ≈ 0.5, rate_mult = 1.5. Dispatch 50% faster. Reasonable.

Queue overfull (waiting=16, norm_error=-1.0): P: 0.5 * (-1.0) = -0.5, correction ≈ -0.5, rate_mult = 0.5. Dispatch at half GPU rate. Slows down but doesn't stop.

Integral behavior: at steady error=0.5, integral grows 0.01/s, reaches max_integral_pos=2.0 after 200s. Contribution: 0.02 2.0 = 0.04 — small correction, just nudges. Integral can't go below -0.5, so 0.02 (-0.5) = -0.01 — negligible slowdown from I.

This simulation reveals the design philosophy: the controller should never fully stop dispatching, even under severe overload. The minimum rate_mult of 0.3 means dispatch slows to at most 3.3× slower than the GPU rate, but never stalls. The integral is deliberately neutered in the negative direction to prevent the "pipeline drain" failure mode.

The Deployment Itself

The subject message executes a chain of six commands:

docker rm -f cuzk-pitune1 2>/dev/null: Force-remove any existing container with the same name, suppressing errors if none exists. This is cleanup — ensuring a fresh start.
docker create --name cuzk-pitune1 cuzk-rebuild:pitune1 /cuzk: Create a new container from the cuzk-rebuild:pitune1 image, specifying the entrypoint as /cuzk (the binary). Critically, this uses docker create rather than docker run — the container is created but not started. This is a deliberate choice: the goal is to extract the binary, not to execute it inside Docker.
docker cp cuzk-pitune1:/cuzk /tmp/cuzk-pitune1: Copy the binary out of the container's filesystem to a local temporary directory. This is the core extraction step.
docker rm cuzk-pitune1: Remove the temporary container, leaving no dangling resources.
scp -P 40612 /tmp/cuzk-pitune1 root@141.0.85.211:/data/cuzk-pitune1: Securely copy the binary to the remote server, placing it in /data/ with a name matching the build tag.
ssh -p 40612 root@141.0.85.211 'chmod +x /data/cuzk-pitune1 && ls -lh /data/cuzk-pitune1': Set the executable permission on the remote binary and verify its size (27MB). The output confirms success: the container ID ee7b0daca856d6fefd866f1fc451ba9c75a5b3ffc918c332e3ecbc66e190c28c, the extracted binary name, and the file listing showing -rwxr-xr-x 1 root root 27M Mar 13 23:41 /data/cuzk-pitune1.

Assumptions and Their Risks

This deployment carries several assumptions, some of which would prove incorrect:

The tuning parameters are correct: The assistant assumed that the mental simulation accurately predicted real-world behavior. But the simulation used idealized conditions — steady-state errors, constant GPU rates, no interaction with the memory budget system. In practice, the real system has nonlinearities (the memory budget's acquire() blocking, the pinned memory pool's allocation behavior, the interaction between multiple GPU workers) that the simulation couldn't capture. The segment summary reveals that this was indeed pitune1 of at least four iterations, suggesting the initial tuning was not yet optimal.

The remote server is in the same state as expected: The deployment assumes the remote server is accessible, has sufficient disk space, and that no other process is interfering with the cuzk pipeline. The previous binary (cuzk-synthcap3) was still running at PID 158698 on the same server, and the deployment doesn't coordinate with stopping it — that would happen in a subsequent step.

The Docker extraction is reliable: The multi-step Docker extraction assumes the image builds correctly, the binary exists at the expected path /cuzk, and the copy operation succeeds. The build step at <msg id=3613> completed successfully, and the image SHA 85dfb1dbff94ba76db825fdb042b009ad7bb634b1a1519fa7fb9871b75ab333b was produced, so this assumption held.

The binary is self-contained: The 27MB binary is assumed to be statically linked or to have all necessary shared libraries available on the remote server. The cuzk-daemon binary links against CUDA libraries, which must be present on the target system.

Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains:

Docker container lifecycle: Understanding why docker create + docker cp + docker rm is used instead of docker run or docker export. This pattern is specific to extracting artifacts from images without executing them.
SCP and SSH deployment patterns: The use of scp for binary transfer and ssh for remote command execution is a standard but fragile deployment pattern — no rollback, no version management, no health checks.
PI controller theory: The concepts of proportional gain, integral gain, error normalization, integral saturation, and asymmetric clamping are essential to understanding why this particular binary was built and what problem it solves.
The cuzk system architecture: Knowledge that cuzk is a GPU-accelerated zero-knowledge proving engine, that it uses a dispatch pacer to regulate synthesis-to-GPU flow, and that the memory budget system provides backpressure.
The deployment naming convention: The binary name pitune1 encodes its purpose — "PI tune, iteration 1" — indicating this is the first in a series of tuning attempts.

Output Knowledge Created

This message creates a deployed artifact: a 27MB executable at /data/cuzk-pitune1 on the remote server. But more importantly, it creates the capability to test the PI controller tuning in a production-like environment. The binary is the embodiment of a hypothesis: that normalizing the error, lowering the gains, and using asymmetric integral clamping will prevent the pipeline collapse under memory pressure.

The message also documents the deployment process itself — the exact commands used, the container ID, the file size and timestamp. This serves as an audit trail: if the binary behaves unexpectedly, the operator knows exactly which build was deployed, when, and from which image.

The Deeper Narrative

What makes this message significant is not the commands themselves but what they represent: the transition from analysis to experimentation. The assistant had spent the preceding messages reasoning through control theory, performing mental simulations, and editing Rust source code. The Docker build at <msg id=3613> compiled those edits into a binary. This message completes the loop by placing that binary in the environment where it can be tested against reality.

The message is also a bet. Every deployment is a bet that the changes will improve the system rather than degrade it. The assistant's mental simulation suggested the new tuning would work, but the real proof would come from observing the system under load. The segment summary tells us that this bet would need refinement — pitune1 through pitune4 would follow, each iteration informed by observed behavior. But this message captures the moment the first iteration went live.

In the broader arc of the conversation, this message sits at a pivot point. The previous chunks had focused on structural changes to the pacer (removing the synthesis throughput cap, adding re-bootstrap detection, fixing GPU rate measurement). Now the focus shifts to the subtler art of tuning — adjusting dimensionless parameters to shape the system's dynamic response. The deployment of pitune1 marks the beginning of this tuning phase, carrying the assistant's best theoretical analysis into the arena of empirical validation.