Wiring the Feedback Loop: How a Single Atomic Counter Anchors a PI-Controlled GPU Dispatch Pacer

In the middle of a complex, multi-session effort to eliminate GPU underutilization in a zero-knowledge proof pipeline, there is a message that appears, at first glance, to be almost trivial. The assistant reads a file, inspects a few lines of code around line 2726 of engine.rs, and announces: "Now wire the counter into GPU workers. First, the clone at worker spawn." This is message [msg 3438], and it is the moment where an abstract control-theory concept — a PI controller with exponential moving average (EMA) feed-forward — touches concrete, running code. The counter in question, a gpu_completion_count of type AtomicU64, is the sensory organ of the entire dispatch pacer. Without it, the controller would be blind.

The Control Problem That Demanded a Counter

To understand why this message exists, one must understand the problem it solves. The CuZK proving engine had recently deployed a zero-copy pinned memory pool ([chunk 24.0]) that eliminated the GPU transfer bottleneck. But this fix revealed a second-order problem: the dispatch logic that feeds synthesized partitions to the GPU was unstable. The original semaphore-based throttle limited total in-flight partitions rather than targeting a specific queue depth of work waiting for the GPU, causing the pipeline to either starve or flood.

The team had iterated through several dispatch strategies. A P-controller (proportional-only) proved too aggressive, instantly filling all allocation slots. A dampened P-controller with a burst cap was better but still unstable because the deep synthesis pipeline (20–60 seconds per partition) made the raw waiting count a noisy, delayed feedback signal. The user then proposed a PI controller operating on a smoothed signal — an exponential moving average of the waiting count or GPU consumption rate ([msg 3428]).

The assistant responded with a detailed control-theory analysis ([msg 3429]), reasoning through the implications of a 20–60 second synthesis delay. With such long latency, the PI gains had to be extremely conservative: Kp around 0.1 for gentle proportional response, Ki around 0.01 for slow integral correction. The feed-forward term — matching the dispatch rate to the measured GPU completion rate — would do the heavy lifting, with PI corrections only nudging around the baseline. The assistant designed a DispatchPacer struct ([msg 3431]) that maintains an EMA of the GPU inter-completion interval, computes a PI correction on the smoothed queue depth error, and derives a dispatch interval from the combination.

The Counter as Sensory Organ

Message [msg 3438] is the step where the assistant wires the feedback signal into the system. The gpu_completion_count — an AtomicU64 shared between the dispatcher and all GPU workers — is the raw measurement that feeds the pacer's EMA. Every time a GPU worker finishes a job on the happy path, it increments this counter. The dispatcher reads it to compute the inter-completion interval, which feeds into the EMA-based GPU rate estimate. Without this counter, the pacer would have no way to know how fast the GPU is actually consuming work.

The message shows the assistant reading the file at the GPU worker spawn section (around line 2726). This is a deliberate, surgical read — the assistant already knows the file structure from previous edits and is looking for the exact location to insert the counter clone. The pattern it follows mirrors the existing gpu_done_notify clone: at worker spawn time, each GPU worker task receives a clone of the shared counter, giving it ownership of a reference that can be atomically incremented from within the async worker task.

The Clone-at-Spawn Pattern

The decision to clone the counter at worker spawn time (rather than passing it through a channel or global variable) reveals several assumptions about the system architecture. First, the GPU workers are long-lived tasks spawned once at engine initialization — they run for the entire proving session. Cloning the AtomicU64 reference at spawn time gives each worker a direct, low-overhead path to increment the counter without any locking or channel communication. Second, the clone pattern is already established for gpu_done_notify (a std::sync::Notify), so it fits naturally into the existing code structure. Third, the use of AtomicU64 (rather than a mutex-protected counter) reflects the need for lock-free, wait-free updates — the GPU finalizer runs on a hot path where any synchronization overhead could degrade throughput.

The assistant's reasoning, visible in the preceding messages, shows careful consideration of where to place the increment. The counter is incremented in the "happy path" of the GPU finalizer — the branch where GPU proving succeeds — right before calling notify_one on the gpu_done_notify signal. This ordering is intentional: the dispatcher needs to know both that a completion happened (via the notify signal) and how many completions have occurred (via the counter). By incrementing before notifying, the assistant ensures that when the dispatcher wakes up from the notify, the counter already reflects the latest completion.

Input Knowledge Required

To understand this message, one needs several layers of context. At the code level, one must understand the GPU worker spawn pattern in engine.rs — specifically the section around Phase 8 where workers are spawned with cloned references to shared state like gpu_work_queue, gpu_done_notify, and the shutdown receiver. At the concurrency level, one must understand AtomicU64 semantics, the Notify synchronization primitive, and the async task model used by the engine. At the control theory level, one must grasp the concept of a PI controller, EMA smoothing, feed-forward terms, and the stability implications of long feedback delays. And at the system level, one must appreciate the end-to-end pipeline: synthesis (20–60s per partition), GPU proving (~1s per partition), and the dispatch logic that bridges them.

Output Knowledge Created

This message, combined with the subsequent edits ([msg 3441], [msg 3444]), creates a concrete feedback mechanism. The gpu_completion_count is cloned into each GPU worker task, and the counter is incremented on every successful GPU completion. This counter feeds into the DispatchPacer's rate estimation, which computes an EMA of the inter-completion interval to derive the GPU's sustainable consumption rate. The pacer then uses this rate as a feed-forward term, with PI corrections on the queue depth error, to compute the dispatch interval. The result is a closed-loop control system that can stabilize the GPU pipeline across varying workloads.

Assumptions and Potential Pitfalls

The implementation makes several assumptions worth examining. It assumes that the happy path (successful GPU proving) is the only path that matters for rate estimation — failed jobs are not counted. This is reasonable because failed jobs represent exceptional conditions that should not influence the steady-state dispatch rate. It assumes that the atomic increment is visible to the dispatcher in a timely manner — on x86_64, AtomicU64 with Relaxed or Release ordering provides near-instant visibility, but on other architectures the memory ordering guarantees matter. It assumes that the counter will not overflow in practice — with a 64-bit counter and jobs completing at ~1 per second, overflow is effectively impossible. And it assumes that cloning the counter at spawn time is safe — each clone points to the same underlying atomic variable, so increments from any worker are visible to all readers.

The Broader Journey

This message is a small but critical step in a much larger arc. The team had already solved the GPU transfer bottleneck with the pinned memory pool ([chunk 24.0]), deployed and debugged it, and then turned to the dispatch scheduling problem. They had tried a semaphore, a P-controller, and a dampened P-controller. Each iteration revealed new insights about the system dynamics — the long synthesis delay, the noise in the waiting count signal, the need for feed-forward rather than pure feedback. The PI-controlled pacer with EMA smoothing and a synthesis throughput cap (added later in the same segment, [chunk 25.1]) represents the culmination of this iterative refinement. The gpu_completion_count wired in message [msg 3438] is the foundation upon which the entire pacer rests.

Without this counter, the pacer would be guessing at the GPU rate. With it, the pacer can measure the actual inter-completion interval, smooth it with an EMA, and compute a dispatch rate that matches the GPU's true capacity. The counter is the bridge between the abstract world of control theory and the concrete world of running GPU kernels — a single AtomicU64 that carries the signal from the GPU's completion events to the dispatcher's rate calculations.