From Collapse to Production: Tuning the GPU Dispatch Pacer Through Four Iterations

Introduction

In the previous chunk of this segment, the assistant had discovered that the synthesis throughput cap—a carefully designed feature meant to prevent the PI-controlled dispatch pacer from over-dispatching—was creating a self-reinforcing collapse loop. The cap was removed, re-bootstrap detection was added, and bootstrap timing was slowed from 200ms to 3 seconds. But removing the cap and adding re-bootstrap logic was only the beginning. The system that emerged from that redesign was fundamentally more stable, but it still exhibited pathological behaviors under memory pressure: integral saturation, re-bootstrap spam, and CPU contention from excessive concurrent syntheses.

This article traces the long tail of tuning that followed the pacer redesign. Across four iterative deployments (pitune1 through pitune4), the assistant diagnosed and fixed three distinct problems that each manifested as pipeline collapse under different conditions. The tuning phase culminated in a stable system that the user confirmed "seems to work pretty well," at which point the focus shifted to production deployment infrastructure—building the latest cuzk binary using the main Dockerfile and configuring a default memory budget for vast.ai environments.

The arc of this segment is a masterclass in control systems tuning under real-world constraints. It demonstrates that even a well-designed feedback controller can fail when its parameters are mismatched to the system's dynamics, and that the path to stability often requires multiple rounds of hypothesis, deployment, observation, and refinement.

Phase 1: The Integral Saturation Problem (pitune1)

The story begins with the user's report at <msg id=3601> after deploying the redesigned pacer (synthcap3). The user's diagnosis was precise and insightful:

"Whenever we 'slam' into the memory ceiling integral goes negative. Whenever integral goes negative new partitions completely start entering synthesis until we fully drain all running and waiting pipelines, essentially starting from scratch, which seems wrong, the backoff shouldn't be nearly this aggressive. Also seems like integral is almost always saturated, we should tune it somehow so that it would be floating around a value where it has useful control authority."

This message identified two distinct problems. First, the integral term was saturating—hitting its clamp limits and losing the ability to provide useful control. When the integral was pinned at its maximum or minimum, the PI controller effectively became a P-only controller, losing the memory and smoothing that the integral term provides. Second, when the integral went negative (indicating the controller was trying to slow down dispatch), it drove the pipeline into a complete drain—all partitions had to finish synthesis and GPU processing before any new ones could start. This was not a gentle backoff; it was a hard stop.

The assistant's response at <msg id=3607> was a textbook example of control theory applied to a real system. The assistant analyzed the PI controller's parameters and identified the root cause: the error term was not normalized. With a target queue depth of 8 and a raw error that could reach 12 or more (when waiting spiked to 20), the proportional term (kp=0.1) produced a correction of 0.1 × 12 = 1.2, which alone drove the rate_mult below 0.1, getting clamped to the minimum. The integral term (ki=0.008) with a max integral of 20 produced at most 0.008 × 20 = 0.16 of correction—almost negligible compared to the P term. The integral was saturated at its limit but contributing almost nothing useful.

The fix, deployed as pitune1, introduced three changes:

1. Normalized error: The error was computed as (target - waiting) / target instead of the raw difference. This normalized the error to a [-1, +1] range when waiting stayed within reasonable bounds, making the controller's gains work consistently regardless of the target queue depth.
2. Asymmetric integral clamping: The integral bounds were made asymmetric—positive integral was capped at +2.0 (allowing the controller to speed up), but negative integral was capped at -0.5 (limiting how much it could slow down). This reflected the engineering insight that slowing down too aggressively is more dangerous than speeding up too aggressively, because a slowed pipeline takes much longer to recover.
3. Tighter rate_mult clamp: The rate_mult clamp was tightened from [0.1, 5.0] to [0.3, 3.0], limiting the maximum slowdown to 3.3× the GPU rate rather than 10×. The deployment of pitune1 at <msg id=3615> involved killing the old process, waiting for pinned memory to drain, and starting the new binary. The assistant monitored memory drainage via SSH, confirming that the baseline usage of ~230 GiB had been reached before proceeding.

Phase 2: The Edge Case That Survived PI Tuning (pitune2)

Despite the normalized error and asymmetric clamping, the user reported at <msg id=3621> that the system still entered the edge case. A log line showed the moment of collapse:

total=460 waiting=16 ema_waiting="7.4" gpu_proc_ms="9262" gpu_eff_ms="4631"
ema_synth_ms="2212" interval_ms=4309 integral="2.00" bootstrap=false rebootstraps=42

The integral was pinned at +2.00 (its positive maximum), gpu_proc_ms was inflated to 9262ms (nearly 10× the expected ~1s) due to cudaHostAlloc stalls, and rebootstraps was at 42—the system had re-entered bootstrap mode 42 times, each time finding the GPU queue empty and trying to refill it, only to be blocked by budget exhaustion.

The assistant's deep analysis at <msg id=3623> revealed that the re-bootstrap logic itself was the problem. The condition for re-entering bootstrap was ema_waiting < 1.0—the exponential moving average of the GPU queue depth dropping below 1. But this condition only checked the GPU queue, not the entire pipeline. With 30–60 second synthesis latency, items could be in the synthesis pipeline (consuming budget, being worked on by CPU workers) but not yet visible in the GPU queue. The pacer would see an empty GPU queue, trigger re-bootstrap, dispatch more items (which would block on budget acquisition because the in-flight items already consumed it), and the cycle would repeat.

The fix, deployed as pitune2, was elegantly simple: the re-bootstrap condition was modified to check total_dispatched <= gpu_completions—ensuring the pipeline was truly empty before re-entering bootstrap. If items were in the synthesis pipeline, they would eventually reach the GPU queue; there was no need to re-enter bootstrap mode. The assistant implemented this at <msg id=3626> with a targeted edit to engine.rs.

But pitune2 also carried forward the PI tuning from pitune1, and the user observed at <msg id=3645> that "the integral still saturates." With ki=0.02 and max_integral_pos=2.0, the integral hit its cap in about 4 seconds at a modest error. It was pinned at the boundary, contributing nothing but a constant bias.

Phase 3: Reshaping the Integral (pitune3)

The user's observation at <msg id=3645> was the key insight that led to pitune3:

"Seems like higher integral cap but lower value (possibly much?) could help as integral still saturates."

The assistant's response at <msg id=3647> was a dramatic reshaping of the integral parameters. The analysis was precise: with ki=0.02 and max_integral_pos=2.0, the integral saturated at a correction of ki × max_integral = 0.02 × 2.0 = 0.04. This tiny correction was the integral's entire useful range—it could contribute at most 0.04 to the rate multiplier, which was negligible compared to the proportional term's contribution.

The fix was to lower ki from 0.02 to 0.001 (a 20× reduction) while simultaneously raising max_integral_pos from 2.0 to 100.0 and max_integral_neg from -0.5 to -20.0. This may seem counterintuitive—lowering the gain while raising the cap—but the math reveals the logic. With ki=0.001 and max_integral_pos=100.0, the maximum integral correction is 0.001 × 100.0 = 0.10, which is actually larger than before (0.04). But more importantly, the integral can now move—it takes approximately 200 seconds to saturate at a sustained error of 0.5, compared to 4 seconds before. The integral floats in a useful range, providing meaningful trim rather than slamming into a wall.

The assistant also tightened the rate_mult clamp from [0.3, 3.0] to [0.5, 2.0], further limiting the controller's range to prevent extreme behavior. The proportional gain kp was adjusted from 0.5 to 1.0 to compensate for the reduced integral contribution.

The deployment of pitune3 at <msg id=3651> used the established pattern: Docker build, binary extraction, SCP transfer, and SSH handoff. The user confirmed at <msg id=3655> that the binary was running.

Phase 4: The Synthesis Concurrency Cap (pitune4)

Between pitune3 and pitune4, the user injected a crucial observation at <msg id=3657>:

"Do we have a simple hard cap on parallel synthesis? We should just set it to default 18, and keep configurable, anything more will probably choke on ddr5 systems."

This was a different kind of fix—not a control theory refinement, but a resource management constraint. The pipeline was computing synth_worker_count as max_partitions_in_budget.min(64).max(4), which on the 755 GiB production machine yielded 28 concurrent synthesis workers. Each worker consumes CPU cycles and, critically, DDR5 memory bandwidth. With 28 workers all running synthesis simultaneously, the memory bus becomes a bottleneck—each individual synthesis slows down, and total throughput collapses. The user's intuition, based on deep knowledge of DDR5 memory subsystem behavior, was that 18 concurrent syntheses was the sweet spot for a 64-core DDR5 system.

The assistant implemented this at <msg id=3663> by adding a max_parallel_synthesis configuration field (default 18) and capping synth_worker_count with it. The change was minimal but architecturally significant: rather than adding a new synchronization primitive like a semaphore, the assistant recognized that capping the number of workers pulling from the bounded channel naturally enforces the limit. The channel itself becomes the concurrency limiter. The change was committed as 6acd3a27 with the message: "Hard cap on concurrent synthesis workers in the pipeline path."

pitune4 combined this synthesis concurrency cap with the PI tuning from pitune3. The deployment at <msg id=3671> followed the same pattern, and the user confirmed at <msg id=3675> that the system "seemed to work well."

The Architecture of a Complete Fix

What makes this segment remarkable is how multiple fixes at different levels of abstraction converged into a stable system. Each fix targeted a different layer of the system, and each was necessary for overall stability.

The PI controller tuning (pitune1–pitune3) addressed the control law—how the dispatch rate should respond to queue depth. By normalizing the error, making the integral bounds asymmetric, and reshaping the integral parameters to prevent saturation, the assistant gave the controller useful control authority across the full range of operating conditions.

The re-bootstrap fix (pitune2) addressed the state machine—when the system should enter fast-dispatch mode. By checking pipeline emptiness rather than just GPU queue depth, the assistant prevented the pacer from sabotaging its own dispatch logic with repeated re-bootstrap cycles.

The synthesis concurrency cap (pitune4) addressed the resource constraint—how many workers should compete for memory bandwidth. By capping concurrent syntheses at 18, the assistant prevented the CPU and DDR5 memory bus from becoming a bottleneck that would degrade overall throughput.

These three fixes represent complementary approaches to the same problem. The PI controller is a feedback mechanism—it measures queue depth and adjusts dispatch rate dynamically. The synthesis cap is a feedforward constraint—it limits the maximum possible load based on known hardware limitations, preventing the controller from ever asking for more than the system can deliver. The re-bootstrap fix corrected a measurement error—the pacer was making decisions based on incomplete information, and adding pipeline visibility enabled it to distinguish between a truly empty pipeline and one that was merely between stages.

The Pivot to Production

After confirming that pitune4 was stable, the user shifted the agenda at <msg id=3679>:

"We'll now be getting back to the main cuzk dockerfile and vast-manager. Main things - build latest, setup default memory budget (probably based from free -h -ish memory, tho note we're in vast.ai docker) such that we have e.g. 10gb safety margin."

This message marked a fundamental transition. The tuning phase—spanning integral saturation, re-bootstrap logic, and synthesis concurrency—was complete. The focus now shifted to production deployment infrastructure: building the latest cuzk using the main Dockerfile (not the rebuild variant used for rapid iteration) and configuring a default memory budget derived from available system memory with a ~10 GB safety margin for vast.ai environments.

The assistant began exploring the infrastructure at <msg id=3681>, reading the main Dockerfile.cuzk, the rebuild Dockerfile, and the config.rs file to understand the existing memory budget configuration. The MemoryConfig already had a safety_margin default of "5GiB", but the user wanted a larger margin and a budget derived dynamically from system memory rather than hardcoded.

The user's empty message at <msg id=3683> served as a silent signal of approval—a handoff that said, without words, "proceed." The assistant continued with the implementation, marking the end of the tuning chapter and the beginning of the deployment chapter.

Lessons in Control Systems Tuning

The four-phase tuning cycle documented in this segment offers several lessons for anyone building feedback control systems in production environments.

Integral Saturation Is a Design Choice, Not a Bug

The integral term in a PI controller is designed to accumulate error over time, providing a correction that grows as long as the error persists. But if the integral bounds are too tight, the integral saturates quickly and becomes useless—it's pinned at a boundary and can't respond to changing conditions. The assistant's fix—lowering the gain and raising the cap—seems paradoxical but is mathematically sound. The key insight is that the integral's useful range is determined by the product ki × max_integral, not by either parameter alone. By lowering ki and raising max_integral, the assistant kept the same maximum correction while dramatically increasing the time to saturation.

Normalize Your Error Terms

The original PI controller used raw error—the difference between target queue depth and actual queue depth. This meant that the controller's behavior depended on the absolute scale of the error, which varied with the target. Normalizing the error by dividing by the target made the controller's gains work consistently regardless of the operating point. This is a standard control systems practice that is often overlooked in ad-hoc implementations.

Check the Full Pipeline, Not Just the Output Buffer

The re-bootstrap spam was caused by a measurement blind spot: the pacer only looked at the GPU queue, not the entire pipeline. Items in the synthesis pipeline were invisible to the re-bootstrap logic, causing it to trigger repeatedly even though work was in flight. This is a classic monitoring error—measuring only the output buffer of a multi-stage pipeline gives an incomplete picture of system state. The fix was to track total_dispatched - gpu_completions as a measure of in-flight work, giving the pacer visibility into the full pipeline.

Hardware Constraints Are Control Constraints

The synthesis concurrency cap was not a control theory fix—it was a recognition that DDR5 memory bandwidth is a finite resource. No amount of PI tuning can overcome a hardware bottleneck. The user's intuition that 18 concurrent syntheses was the sweet spot for a 64-core DDR5 system was based on deep hardware knowledge, and the assistant's implementation was appropriately minimal: a configuration field with a sensible default, allowing operators to adjust based on their specific hardware.

Conclusion

The long tail of tuning documented in this segment represents the difference between a system that works in theory and one that works in practice. The PI controller was mathematically sound, but its parameters were wrong for the system's dynamics. The re-bootstrap logic was conceptually correct, but its condition was too narrow. The synthesis worker count was computationally reasonable, but it ignored hardware constraints.

Each fix required not just code changes but a deep understanding of the system's behavior under load—the kind of understanding that comes from careful log analysis, iterative deployment, and close collaboration between human intuition and machine analysis. The user's ability to identify integral saturation from log lines, the assistant's ability to trace the re-bootstrap spam to a pipeline visibility problem, and the joint recognition that DDR5 bandwidth imposes a hard limit on concurrent syntheses—these are the skills that transform a fragile system into a robust one.

The pivot to production deployment at the end of the segment marks the natural conclusion of this tuning phase. The system was stable, the parameters were set, and the focus could shift to building the infrastructure that would carry this work into production. The PI controller, the re-bootstrap logic, and the synthesis cap would all be baked into the production Docker image, ready for deployment on vast.ai environments with a default memory budget that kept a 10 GB safety margin. The tuning was done; the shipping could begin.## References

[1] — "The Arc of Discovery: How a Synthesis Throughput Cap Was Wired, Deployed, and Ultimately Abandoned" — The preceding chunk article that documents the synth cap redesign that set the stage for this tuning cycle.

[2] — "The Long Tail of Tuning: From Integral Saturation to Production Deployment in a GPU Dispatch Pacer" — The companion chunk article covering the same material from a different perspective.

[13] — <msg id=3601> — The user's diagnostic report that launched the PI tuning cycle, identifying integral saturation as the root cause.

[33] — <msg id=3621> — The log-line analysis showing pitune1's failure, with integral pinned at +2.00 and rebootstraps at 42.

[57] — <msg id=3645> — The user's observation that the integral still saturates, leading to pitune3's dramatic parameter reshaping.

[69] — <msg id=3657> — The user's request for a hard cap on parallel synthesis, citing DDR5 memory bandwidth constraints.

[36] — <msg id=3623> — The assistant's deep analysis tracing re-bootstrap spam to a pipeline visibility blind spot.

[38] — <msg id=3626> — The edit that fixed re-bootstrap spam by checking total_dispatched <= gpu_completions.

[84] — <msg id=3671> — The deployment of pitune4, combining PI tuning with the synthesis concurrency cap.

[90] — <msg id=3675> — The confirmation that both changes were live and working well.

[91] — <msg id=3679> — The user's directive to shift to production infrastructure, marking the end of the tuning phase.

[93] — <msg id=3681> — The assistant's exploration of the main Dockerfile and config for production deployment.

[95] — <msg id=3683> — The user's empty message as a silent handoff signal to proceed with production deployment.