The Moment the Pacer Started: Deploying a PI-Controlled GPU Dispatch Scheduler

On March 13, 2026, at index 3476 of a long-running opencode session, the assistant sent a brief but significant status message:

Running. GPU queue target dispatch pacer enabled gpu_queue_target=8 and synthesis priority dispatcher started (PI pacer). Bootstrap will dispatch 8 items at 200ms spacing, then wait for first GPU completion to calibrate.

Accompanied by a todowrite update marking four high-priority tasks as completed — the DispatchPacer struct, the gpu_completion_count atomic counter, the finalizer wiring, and the dispatcher loop rewrite — this message represents the culmination of an intense iterative effort to stabilize GPU pipeline scheduling in the CuZK proving engine. It is the moment a new control system went live on a production remote server, replacing a flawed P-controller that had been causing instability, excessive pinned memory allocations, and GPU timing jitter.

To understand why this message matters, one must appreciate the engineering struggle that preceded it.

The Vicious Cycle of Bursty Dispatch

The session's context reveals a system under stress. The CuZK proving engine processes zero-knowledge proofs through a pipeline: synthesis workers produce circuit partitions on CPU threads, which are then dispatched to a GPU for proving. A pinned memory pool was recently deployed to eliminate expensive H2D (host-to-device) memory transfers, and it worked — reuse rates hit 81%, and allocations dropped to zero after warmup in ideal conditions. But a scheduling problem remained.

The previous dispatch strategy, a damped P-controller (cuzk-pctrl2), used a Notify-based two-phase loop: wait for a GPU completion event, then dispatch a burst of up to max(1, min(3, deficit * 0.75)) items. This burst-based approach proved too aggressive. It caused the pinned memory pool to keep growing — 355 allocations totaling over 850 GiB of pinned memory — because bursts of dispatch would spike concurrent synthesis demand beyond what the pool had available, triggering expensive cudaHostAlloc calls deep into the run. These allocations serialized through the GPU driver, causing micro-stalls that corrupted the P-controller's timing measurements. As the user noted in [msg 3470], "the late allocations cause GPU hickups which break the naive P-control pacing."

This was a vicious cycle: bursty dispatch → pool exhaustion → cudaHostAlloc → GPU driver serialization → GPU timing jitter → pacing estimates wrong → more bursty dispatch. The system could not converge on a steady state.

Why This Message Was Written

Message 3476 is a deployment confirmation. It signals that a fundamentally different approach — a PI-controlled dispatch pacer — has been built, compiled, containerized, transferred to the remote server, and started successfully. The assistant had just executed a sequence of operations visible in the preceding messages:

The Technical Design: PI Control with EMA Feed-Forward

The DispatchPacer represents a significant architectural shift. Where the old P-controller reacted to GPU completions with bursts, the PI pacer operates on a timer-based loop that computes a precise dispatch interval. The design has three key elements:

Exponential Moving Average (EMA) of GPU inter-completion interval. Rather than using raw, noisy timing measurements, the pacer smooths the GPU completion rate with an EMA. This filters out the jitter caused by occasional cudaHostAlloc stalls and provides a stable feed-forward estimate of how fast the GPU can consume work.

PI correction on smoothed queue depth error. The controller maintains a target queue depth (here, 8 items waiting for the GPU). It measures the actual queue depth via an EMA-smoothed signal and applies proportional-integral correction to the dispatch interval. The integral term is essential: it eliminates steady-state error, ensuring the queue converges precisely to the target rather than oscillating around it.

Bootstrap phase. Before the first GPU completion arrives, there is no data to calibrate on. The pacer enters a bootstrap mode: it dispatches the target number of items (8) at a fixed 200ms spacing, then waits for the first GPU completion event. Only after that event does it switch to PI-controlled steady-state operation. This avoids the "blind dispatch" problem that plagued earlier approaches.

Assumptions and Hidden Risks

The message reveals several assumptions baked into this deployment:

  1. That a queue target of 8 is appropriate. This value determines the steady-state pipeline depth. Too low, and the GPU starves; too high, and too many synthesis workers contend for CPU resources. The choice of 8 appears to be based on empirical observation from previous runs, but it is not dynamically adaptive.
  2. That 200ms bootstrap spacing is safe. During bootstrap, the pacer dispatches without any GPU feedback. If the first GPU completion takes longer than expected (e.g., due to driver initialization or thermal throttling), the pacer could overshoot before it has data to correct.
  3. That the PI controller alone can handle all workload regimes. The chunk summary reveals that this assumption proved incorrect. When synthesis became compute-constrained — i.e., when CPU-bound synthesis workers could not keep pace with the GPU — the pacer drove the dispatch interval below the GPU rate to fill the queue, which flooded the system with concurrent synthesis jobs, causing CPU contention and degrading overall throughput. This was addressed in a subsequent iteration by adding a synthesis throughput cap with anti-windup.
  4. That the EMA smoothing parameter is well-tuned. The EMA's responsiveness to changes in GPU rate is determined by a smoothing factor (not visible in this message but presumably a constant). If the factor is too low, the pacer reacts slowly to genuine changes in GPU throughput; if too high, it overreacts to noise.

Input and Output Knowledge

To understand this message, a reader needs knowledge of: the GPU proving pipeline architecture (synthesis workers, GPU queue, finalizer); the history of failed dispatch strategies (semaphore, P-controller, damped P-controller); the pinned memory pool work and its allocation patterns; and the fundamentals of control theory (P vs PI control, EMA, bootstrap phases).

The message creates new knowledge: confirmation that the pacer1 binary is operational on the remote server; the specific parameter choices (queue_target=8, bootstrap_spacing=200ms); the bootstrap strategy; and a record of completed implementation tasks. It also implicitly establishes a baseline for subsequent analysis — the user would go on to examine the pacer's behavior and identify the synthesis throughput bottleneck that led to the next iteration.

The Broader Narrative

Message 3476 sits within a larger arc of iterative refinement. The team moved from a reactive semaphore (which limited total in-flight partitions) to a P-controller (which used GPU completions as a trigger but dispatched in bursts), to a damped P-controller (which capped burst size), and finally to this PI-controlled pacer. Each iteration was driven by empirical observation: the semaphore caused GPU starvation, the P-controller caused overshoot, the damped P-controller caused oscillations, and now the PI pacer aimed for stable convergence.

What is remarkable is the sophistication of the solution. This is not a simple heuristic or a hard-coded rate limit — it is a feedback control system with feed-forward estimation, integral correction, and a carefully designed bootstrap sequence. The engineering team treated GPU pipeline scheduling as a control problem, applying techniques from industrial process control to a distributed proving system.

The message also reveals the rhythm of the development process: build, deploy, observe, refine. The assistant and user work in tight loops, with the assistant implementing code changes, building Docker images, transferring binaries, restarting processes, and reporting results — all within minutes. The deployment of pacer1 is just one step in a sequence that would continue with the addition of a synthesis throughput cap, further tuning of PI parameters, and ongoing analysis of pinned buffer behavior.

Conclusion

Message 3476 is a milestone marker in a complex engineering effort. It records the moment when a PI-controlled dispatch pacer — a sophisticated feedback system designed to break a vicious cycle of bursty dispatch, pool exhaustion, and GPU jitter — went live on a production server. The message is brief, but it encapsulates hours of reasoning, implementation, debugging, and deployment work. It is a status update that tells a story: the old approach failed, a new approach was built, and it is now running. The next messages in the conversation would reveal whether it succeeded — and what further refinements were needed.