The Commit That Replaced Burst Dispatch with a PI-Controlled Pacer

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proving, the difference between a system that hums along at full utilization and one that stutters and stalls often comes down to a single question: how do you decide when to send the next piece of work to the GPU? Message <msg id=3480> in this opencode session captures the moment when the assistant answered that question by committing a fundamental redesign of the dispatch scheduler — replacing a burst-based P-controller with a continuous PI-controlled pacer. The commit message itself, at 291 insertions and 56 deletions across two files, is a dense artifact that encodes weeks of iterative debugging, control theory reasoning, and hard-won operational insight.

This article examines that single message in depth: why it was written, what decisions it embodies, what assumptions it makes, what knowledge it required and produced, and how the thinking process visible in the surrounding reasoning reveals the contours of a complex engineering problem.

The Message

The subject message is a git commit, executed by the assistant in response to the user's simple command "commit" at <msg id=3478>. After staging the two modified files (config.rs and engine.rs) and verifying the diff and commit history, the assistant wrote:

git add extern/cuzk/cuzk-core/src/config.rs extern/cuzk/cuzk-core/src/engine.rs && git commit -m "cuzk: PI-controlled dispatch pacer for GPU queue depth

Replace the burst-based P-controller with a continuous pacer that
regulates synthesis dispatch rate using PI control with GPU rate
feed-forward.

DispatchPacer:
- Feed-forward: EMA of GPU inter-completion interval (measured via
  atomic completion counter incremented by GPU finalizers)
- Feedback: PI correction on (target - EMA_waiting), where EMA_waiting
  is a smoothed gpu_work_queue.len()
- Conservative gains (Kp=0.1, Ki=0.008) for the 20-60s synthesis delay
- Output: dispatch interval clamped to [50ms, 60s]

Bootstrap phase:
- Dispatches target items at 200ms spacing to prime the pipeline
- Then waits for the first GPU completion to calibrate the GPU rate EMA
- Switches to PI control once calibrated

Steady state:
- Dispatches one item per timer tick at the PI-computed interval
- GPU events update pacer state but don't directly trigger dispatch
- Converges to matching GPU consumption rate with target items waiting
- Periodic status logging every 5 dispatches

This replaces the previous burst dispatch which caused:
- Pinned pool exhaustion -> cudaHostAlloc -> GPU driver serialization
- GPU timing jitter -> pacing instability -> more burst dispatch
The steady dispatch rate keeps concurrent synthesis count stable,
so the pinned pool stays within its warm allocation."

Why This Message Was Written: The Deep Context

To understand why this commit exists, one must understand the tortured history of GPU dispatch scheduling that preceded it. The session's segment summary reveals a progression: the team started with a simple semaphore-based dispatch, moved to a reactive P-controller that dispatched in bursts on GPU completion events, then dampened that controller with a cap on burst size. Each iteration revealed a deeper pathology.

The core problem was a vicious cycle. The P-controller worked by waiting for a GPU completion, then dispatching a burst of new work to fill the deficit between the target queue depth and the current waiting count. But synthesis — the CPU-bound phase that prepares work for the GPU — takes 20–60 seconds per partition. During that long pipeline delay, the controller had no visibility into work-in-progress. It would over-dispatch, causing a flood of concurrent synthesis jobs. Those jobs would all try to allocate pinned memory buffers simultaneously, triggering expensive cudaHostAlloc calls that serialized through the GPU driver, stalling the GPU. The stalls made GPU timing jittery, which made the controller's feedback signal noisy, which caused more oscillation — and the cycle repeated.

The user diagnosed this precisely at <msg id=3470>: "And the late allocations cause GPU hickups which break the naive P-control pacing." The assistant's reasoning at <msg id=3471> elaborated: "bursty dispatch → pool exhaustion → cudaHostAlloc → GPU driver serialization → GPU timing jitter → pacing estimates wrong → more bursty dispatch."

The commit at <msg id=3480> is the culmination of this diagnostic journey. It represents a shift in control philosophy: instead of reacting to events (GPU completions) with bursts, the new design maintains a steady dispatch rate and adjusts that rate gradually using a PI controller. The key insight is that the dispatch rate should match the GPU's consumption rate, with a small correction based on queue depth, rather than trying to fill a deficit in one shot.

How Decisions Were Made

The commit message encodes several critical design decisions, each reflecting a choice informed by the failures of the previous approach.

Decision 1: Timer-based pacing instead of event-triggered dispatch. The old P-controller dispatched on GPU completion events — it was reactive. The new pacer dispatches one item per timer tick, at an interval computed by the PI controller. GPU events still update the pacer's state (the EMA of inter-completion interval, the EMA of waiting count), but they don't directly trigger dispatch. This decouples the dispatch rhythm from the noisy GPU completion signal, preventing the boom-bust cycles.

Decision 2: Feed-forward from GPU rate. The pacer uses an exponential moving average (EMA) of the GPU inter-completion interval as a feed-forward term. This means the baseline dispatch rate is anchored to the actual GPU processing rate — if the GPU completes one partition per second, the pacer naturally targets about one dispatch per second. The PI correction then fine-tunes around this baseline to maintain the target queue depth.

Decision 3: Conservative PI gains. The commit specifies Kp=0.1 and Ki=0.008. These are deliberately small values, chosen because of the long synthesis delay (20–60 seconds). Aggressive gains would cause the controller to overcorrect before the effects of previous dispatches are visible in the queue depth. The integral term is clamped to prevent windup during extended idle or saturated periods.

Decision 4: Bootstrap phase. Before any GPU completions have occurred, the pacer has no rate data. The bootstrap phase dispatches the target number of items at a fixed 200ms spacing, then blocks until the first GPU completion provides a calibration point. This avoids the problem of dispatching blindly into an empty pipeline.

Decision 5: EMA smoothing of the waiting count. The raw gpu_work_queue.len() is an integer that jumps discretely as items arrive and depart. Smoothing it with an EMA (alpha presumably around 0.3) converts this noisy signal into a continuous control variable that the PI controller can work with.

Assumptions Embedded in the Commit

Every commit carries assumptions, and this one is no exception. Some of these assumptions proved correct; others were challenged almost immediately after deployment.

Assumption 1: The PI controller's feedback signal (EMA of waiting count) is sufficient to stabilize the system. This assumes that queue depth is a reliable proxy for the balance between dispatch rate and GPU consumption rate. In practice, this holds when the GPU is the bottleneck. But as the user discovered at <msg id=3483>, when synthesis becomes the bottleneck (CPU-constrained), the pacer drives the dispatch interval below the GPU rate to fill the queue, which floods the system with concurrent synthesis jobs, causing CPU contention and degrading overall throughput. The queue depth signal alone cannot distinguish between "GPU is slow" and "synthesis is slow."

Assumption 2: Conservative gains (Kp=0.1, Ki=0.008) are appropriate. These were chosen based on the 20–60s synthesis delay. But the actual system dynamics depend on the mix of proof types, the number of CPU cores, and the memory bandwidth. The gains may need to be adaptive or configurable per deployment.

Assumption 3: The bootstrap phase of dispatching target items at 200ms spacing is sufficient to prime the pipeline. This assumes that 200ms is fast enough to get the pipeline filled without being so fast that it causes a burst of allocations. The 200ms spacing was chosen somewhat arbitrarily — it's fast enough to dispatch 8 items in 1.6 seconds, but slow enough to avoid simultaneous cudaHostAlloc calls.

Assumption 4: The pinned pool will stabilize once dispatch is steady. The commit message claims that "the steady dispatch rate keeps concurrent synthesis count stable, so the pinned pool stays within its warm allocation." This assumes that the pool's peak allocation during the bootstrap phase is sufficient for steady-state operation. If the bootstrap phase itself causes too many allocations, or if the proof type changes mid-run (requiring different buffer sizes), the pool may still need to grow.

Assumption 5: The atomic completion counter and Notify mechanism provide adequate communication between GPU finalizers and the pacer. This assumes that the Notify's single-permit semantics don't cause the pacer to miss multiple completions that occur between wakeups. The pacer compensates by sampling the counter directly on each iteration, so missed notifications only cause latency, not correctness issues.

Mistakes and Incorrect Assumptions

The most significant limitation of the committed design — one that the user identified within minutes of deployment — is that it doesn't handle the synthesis-constrained case. At <msg id=3483>, the user reported: "When synthesis doesn't keep up... the interval collapses to below gpu ema... which eventually causes us to max out memory budget with running synthesis." The pacer, in its quest to fill the GPU queue to the target depth, drives the dispatch rate higher and higher, flooding the system with synthesis jobs that compete for CPU resources and degrade overall throughput.

This is a classic control problem: the pacer optimizes for queue depth (a local metric) rather than system throughput (a global metric). When synthesis is the bottleneck, pushing more dispatch only makes things worse — the synthesis jobs contend with each other, each taking longer, and the queue never fills. The pacer's integral term keeps accumulating positive error, driving the dispatch rate ever higher, until the memory budget is exhausted.

The assistant addressed this in a subsequent iteration (described in the chunk summary) by adding a synthesis throughput cap: the dispatch rate is clamped to not exceed the measured synthesis completion rate, with the PI integral term frozen when the cap is active to prevent windup. This creates a self-regulating loop: if synthesis slows due to contention, the measured rate drops, automatically reducing the dispatch ceiling and relieving CPU pressure. But this fix is not in the commit at <msg id=3480> — it came later.

Another implicit mistake is the assumption that the bootstrap phase's 200ms spacing is sufficient to avoid allocation bursts. The deployment logs (examined at <msg id=3469>) showed 355 total pinned buffer allocations, with allocations continuing 27 minutes into the run. The pacer reduces the rate of new allocations but doesn't eliminate them entirely if the pool's peak demand exceeds its current size.

Input Knowledge Required

To understand this commit message, a reader needs knowledge spanning several domains:

Control theory: Understanding of proportional-integral (PI) control, feed-forward vs. feedback, integral windup, and the trade-offs between aggressive and conservative gains. The commit doesn't explain PI control — it assumes the reader knows that Kp and Ki are tuning parameters.

GPU computing: Knowledge of CUDA pinned memory (cudaHostAlloc), GPU driver serialization, H2D (host-to-device) transfer overhead, and the concept of a GPU work queue. The commit references "pinned pool exhaustion" and "GPU driver serialization" as known problems.

Zero-knowledge proving pipelines: Understanding that zk-SNARK proving involves a CPU-bound synthesis phase (building the circuit and witness) followed by a GPU-bound proving phase (generating the proof). The 20–60s synthesis delay is specific to the SnapDeals proof type in the CuZK engine.

Exponential moving averages: The commit uses EMA for smoothing both the GPU interval and the waiting count. The reader needs to understand that EMA is a low-pass filter that attenuates high-frequency noise.

Rust async programming: The implementation uses tokio::select!, Notify, atomic counters, and async timers. The commit message doesn't detail these, but the implementation in engine.rs does.

Output Knowledge Created

The commit creates several forms of knowledge:

A permanent design record: The commit message documents the rationale for the pacer, the structure of the bootstrap and steady-state phases, the PI gains, and the problems it solves. This is invaluable for future developers who need to understand why the dispatch system works the way it does.

A baseline for further iteration: The commit at 0ba6cbca becomes the reference point for subsequent changes. When the synthesis throughput cap was added later, the diff from this commit shows exactly what changed. The git history tells the story of the system's evolution.

Operational insights: The commit implicitly encodes operational knowledge about the system's dynamics — the 20–60s synthesis delay, the pinned pool exhaustion cycle, the need for conservative gains. These are not just code changes; they are documented observations about how the system behaves under load.

A testable hypothesis: The commit embodies the hypothesis that steady dispatch rate + PI control will stabilize the pinned pool and eliminate GPU timing jitter. This hypothesis was tested in deployment (the pacer1 binary) and partially validated — the system worked "wonderfully compared to previous state" — while also revealing the synthesis-constrained edge case that required further iteration.

The Thinking Process

The reasoning visible in the surrounding messages reveals a deeply iterative design process. At <msg id=3427>, the assistant considered and rejected several approaches before settling on the pacer design. The reasoning shows the assistant weighing alternatives: "I'm wondering if this is getting too complex... A simpler approach might be to just measure how long the GPU takes between completions, then dispatch items at that same cadence."

The assistant walked through the math of the bootstrap phase explicitly: "With a 200ms interval and 8 items to dispatch, it takes 1.6 seconds to get everything in flight. Since synthesis takes 10-30 seconds, all 8 are queued well before the first one completes around 10 seconds." This kind of quantitative reasoning — tracing through the timing of each phase — is essential for control system design.

The assistant also considered edge cases: "if the dispatcher is blocked on budget.acquire(), the notification won't interrupt that wait... However, that's actually fine — the notification gets stored in the Notify." This shows a careful consideration of async interaction patterns.

The commit message itself is a compressed form of this reasoning. It doesn't show the false starts or the rejected alternatives, but it captures the essential design decisions in a form that future readers can use to understand the system.

Conclusion

Message <msg id=3480> is more than a git commit — it's a milestone in the iterative refinement of a complex control system for GPU pipeline scheduling. It represents the moment when the team moved from reactive, event-driven dispatch to continuous, rate-based pacing. The commit encodes hard-won knowledge about the system's dynamics: the 20–60 second synthesis delay, the vicious cycle of pinned pool exhaustion and GPU timing jitter, the need for conservative gains in the face of long pipeline latency.

The message also reveals something about the nature of engineering progress. Each iteration — from semaphore to P-controller to damped P-controller to PI pacer — was a rational response to the failures of the previous approach. The PI pacer wasn't the final answer (the synthesis throughput cap came next), but it was a necessary step. The commit captures not just code, but the reasoning that produced it, the assumptions that guided it, and the problems it was designed to solve. In doing so, it becomes a permanent record of engineering judgment — a snapshot of what the team knew and believed at that moment in time.