The Pacer Takes Over: Wiring a PI-Controlled Dispatch System into the cuzk GPU Pipeline

Introduction

In the high-stakes world of zero-knowledge proof acceleration, every millisecond of GPU idle time is lost revenue and wasted compute. The cuzk proving engine, a GPU-accelerated proof system for Filecoin's proof-of-spacetime, had been struggling with a stubborn problem: its GPU dispatch scheduler oscillated wildly, over-committing and under-committing synthesis work in boom-bust cycles that left the GPU underutilized and the pipeline unstable. After iterating through a semaphore-based reactive throttle, a P-controller, and a dampened P-controller, the team arrived at a more sophisticated solution: a PI-controlled dispatch pacer with exponential moving average (EMA) smoothing and feed-forward rate estimation.

Message [msg 3436] is the critical implementation step where this new pacer is finally wired into the dispatcher loop, replacing the old GPU queue target section. The message itself is deceptively simple — a two-line edit command and confirmation — but it represents the culmination of an extensive design process spanning thousands of words of reasoning, multiple code edits, and deep engagement with control theory principles applied to a real-world distributed systems problem.

The Problem: Why Simple Controllers Failed

The core challenge facing the cuzk pipeline is the extreme asymmetry between synthesis time and GPU proving time. Each proof partition requires 20–60 seconds of CPU-bound synthesis before it can be sent to the GPU, which then processes it in roughly one second. To keep the GPU fed, the system must maintain a pipeline of 20–60 partitions in flight at any time. However, memory constraints limit how many partitions can be waiting for the GPU — the target queue depth is just 8 items.

This creates a classic control problem with a long, variable delay in the feedback loop. When the dispatcher sends a synthesis job, it takes 20–60 seconds before that job appears in the GPU waiting queue as a completed synthesis ready for proving. By the time the dispatcher sees the result of its decision, the system state has already changed dramatically. The previous burst-based P-controller would dispatch aggressively when the queue was empty, but by the time those items finished synthesis and entered the queue, the controller had already over-committed, causing memory pressure and contention.

The user in [msg 3428] noted this explicitly: "the system needs quite some time to stabilize from under/overshoot, multiple minutes (single synth is 20-60s depending on contention so from dispatch to gpu)." This observation drove the shift from reactive burst dispatch to a continuous pacing mechanism.

The Design: A PI Controller with Feed-Forward

The assistant's reasoning in [msg 3427] and [msg 3429] reveals a deep engagement with control theory. The key insight was that instead of reacting to queue depth errors with bursts, the dispatcher should maintain a steady dispatch rate that matches the GPU's consumption rate, with gentle PI corrections to keep the queue at the target depth.

The DispatchPacer struct embodies this design. It tracks two smoothed signals: an EMA of the GPU inter-completion interval (providing a feed-forward estimate of the GPU's natural processing rate) and an EMA of the waiting queue depth (providing a filtered view of the pipeline state). The PI controller computes an error term as target_queue_depth - smoothed_waiting_count, then applies proportional and integral corrections to adjust the dispatch interval relative to the measured GPU rate.

The assistant carefully selected gains: Kp=0.1 for gentle proportional response and Ki=0.01 for slow integral correction, with alpha values of 0.2 and 0.3 for the EMA smoothers. These conservative values reflect the assistant's understanding that with a 20–60 second pipeline delay, aggressive gains would cause oscillation. The integral term includes anti-windup clamping to prevent unbounded accumulation during transient periods.

The Bootstrap Phase: Handling Startup Uncertainty

A critical design decision was how to handle the startup period before any GPU completions have occurred. Without a measured GPU rate, the pacer has no feed-forward signal. The assistant designed a bootstrap phase that dispatches the target number of items at a fixed short interval, then blocks further dispatch until the first GPU completion provides real rate data.

This bootstrap mechanism is visible in the reasoning: "I need to gate the bootstrap phase differently — maybe dispatch just the initial target batch upfront, then wait for the first GPU event before resuming the timer-based dispatch loop." The assistant recognized that without this gating, the dispatcher would fire off items at 100ms intervals during the entire 10-second synthesis phase, massively overshooting the target queue depth before any feedback arrived.## The Message Itself: A Quiet Turning Point

Message [msg 3436] reads in full:

Now replace the GPU queue target section with the new pacer setup: [edit] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

On its surface, this is almost anticlimactic — a simple edit command and a confirmation. But this message is the moment when weeks of iterative design, multiple failed controller variants, and extensive theoretical reasoning crystallize into a concrete change to the codebase. The "GPU queue target section" being replaced is the old dispatch logic — the semaphore-based throttle, the P-controller, the dampened burst dispatcher — all of it swept away in favor of the new DispatchPacer.

The message is the second-to-last step in a sequence that began with the assistant reading the engine.rs file in [msg 3430] and [msg 3435], adding the DispatchPacer struct in [msg 3431], adding the gpu_completion_count atomic counter in [msg 3433], and finally wiring it all together. The edit in [msg 3436] is the integration point — the moment the new pacer becomes the active dispatch mechanism rather than just a struct definition sitting in the code.

Assumptions and Design Decisions

Several assumptions underpin this implementation:

  1. The GPU completion rate is the correct feed-forward signal. The pacer assumes that the GPU's inter-completion interval, smoothed via EMA, represents the sustainable dispatch rate. This is reasonable in steady state but can be misleading during transient phases like proof type changes or pinned buffer reallocations.
  2. The waiting queue depth is a meaningful control signal. The PI controller uses the smoothed queue depth as its feedback variable. However, as the user later identified, this signal becomes problematic when synthesis is compute-constrained — the pacer may drive the dispatch rate below the GPU rate to fill the queue, flooding the system with concurrent synthesis jobs and causing CPU contention.
  3. Conservative gains will prevent oscillation. With Kp=0.1 and Ki=0.01, the assistant assumed these values would be gentle enough to avoid the boom-bust cycles of the previous controllers. The long pipeline delay (20–60s) was the primary justification for this conservatism.
  4. The bootstrap phase is sufficient for startup. The assistant assumed that dispatching the target number of items at a fixed spacing, then waiting for the first GPU completion, would provide a reasonable initial pipeline state. This assumption was later challenged by the user's observation that pinned allocation costs during startup can skew initial measurements.
  5. Timer-based dispatch is preferable to event-based dispatch. The pacer uses a timer to space dispatches evenly, rather than dispatching in response to GPU events. This was a deliberate choice to avoid the burst behavior that plagued the previous controllers.

What the Message Achieves

Message [msg 3436] is the implementation step that transforms the DispatchPacer from a theoretical construct into the active dispatch mechanism. Before this edit, the dispatcher loop contained the old GPU queue target logic — the semaphore-based throttle that limited in-flight partitions. After this edit, the dispatcher loop calls pacer.update() at the top of each iteration, computes a dispatch interval based on the PI controller's output, and uses tokio::select! to race between the pacing timer and GPU completion notifications.

The edit replaces approximately 50 lines of the old dispatch logic with the new pacer integration. The old code checked gpu_work_queue.len() and decided whether to dispatch based on a simple threshold comparison. The new code feeds the queue length into the pacer's EMA, computes the PI correction, and dispatches at the calculated interval — a fundamentally different approach that treats dispatch as a continuous rate control problem rather than a threshold-based on/off decision.

The Knowledge Created

This message creates several forms of knowledge:

Output knowledge: The dispatcher loop now integrates with the DispatchPacer, reading gpu_completion_count and gpu_work_queue.len() each iteration, calling pacer.update() to compute the next dispatch interval, and using timer-based pacing with GPU event wakeups. The bootstrap phase dispatches target items at a fixed spacing before the first GPU completion, then transitions to PI-controlled pacing.

Design knowledge: The message validates the approach of treating GPU dispatch as a continuous rate control problem rather than a reactive burst system. The PI controller with EMA smoothing represents a significant advancement over the previous semaphore and P-controller approaches.

Process knowledge: The message demonstrates a development methodology where control theory principles are applied iteratively to a real distributed systems problem, with each failure (semaphore, P-controller, dampened P-controller) informing the next design iteration.

The Path Forward

While [msg 3436] represents a major milestone, it was not the final word on GPU dispatch scheduling. The user later identified that the pacer's PI controller, when faced with compute-constrained synthesis, would drive the dispatch interval below the GPU rate to fill the queue, flooding the system with concurrent synthesis jobs and causing CPU contention. This led to the addition of a synthesis throughput cap — a clamp on the dispatch rate based on the measured synthesis completion rate, with anti-windup logic to freeze the integral term when the cap is active.

The message thus sits at a pivotal moment in the development arc: the point where the team moved from simple reactive control to sophisticated predictive control, but before the final refinement that would add the synthesis throughput cap. It is the bridge between the old world of burst dispatch and the new world of continuous, rate-based pipeline management.