The Todo List That Tells a Story: A Milestone in GPU Pipeline Control

Introduction

In the midst of an intense engineering session focused on stabilizing a GPU proving pipeline for zero-knowledge proofs, a single message appears that is, on its surface, almost trivial: a todo list update. Message <msg id=3446> in the conversation is a todowrite call — a structured status report showing three tasks marked "completed" and one task marked "in_progress." It reads:

[assistant] [todowrite] {"todos":[{"content":"Add DispatchPacer struct with PI control + GPU rate EMA","priority":"high","status":"completed"},{"content":"Add gpu_completion_count AtomicU64 shared state","priority":"high","status":"completed"},{"content":"Wire counter into GPU finalizer (happy path only)","priority":"high"... [ { "content": "Add DispatchPacer struct with PI control + GPU rate EMA", "status": "completed", "priority": "high" }, { "content": "Add gpu_completion_count AtomicU64 shared state", "status": "completed", "priority": "high" }, { "content": "Wire counter into GPU finalizer (happy path only)", "status": "completed", "priority": "high" }, { "content": "Rewrite dispatcher loop with pacer (bootstrap + steady-state)", "status": "in_progress", ... ]

Yet this seemingly mundane status update marks a critical inflection point in a multi-day engineering effort to solve one of the most challenging problems in high-performance GPU computing: how to schedule work in a pipeline where the CPU synthesis stage takes 20–60 seconds per unit of work, the GPU proving stage takes roughly 1 second, and the goal is to maintain a stable queue depth of exactly 8 items waiting for the GPU — no more, no less — without either starving the GPU or flooding the system with concurrent synthesis jobs.

The Journey to This Point

To understand why this message matters, we must trace the arc of the broader engineering effort. The team had already solved the fundamental bottleneck of GPU underutilization by implementing a zero-copy pinned memory pool (the PinnedPool), which eliminated costly host-to-device (H2D) transfer times. But solving the throughput problem revealed a new, subtler challenge: the dispatch scheduling logic.

The initial approach was a semaphore-based reactive throttle: limit the total number of in-flight partitions, and let the semaphore provide backpressure. But the user identified a fundamental flaw in this design — it limited total in-flight partitions rather than targeting a specific queue depth of synthesized partitions waiting for the GPU. The system needed to maintain a steady pipeline, not just cap concurrency.

The assistant then implemented a P-controller (proportional control), replacing the semaphore with a Notify-based two-phase loop: wait for a GPU completion event, then dispatch the full deficit in a burst to intentionally overshoot and converge on a steady state. The first deployment (cuzk-pctrl1) proved too aggressive, instantly filling all allocation slots. A dampening factor was added, capping burst size, but the system remained unstable due to the deep synthesis pipeline — the raw waiting count was a noisy and delayed feedback signal.

The PI Controller Design

Message <msg id=3429> reveals the assistant's deep reasoning about the control theory challenge. The key insight is that the system has a pure delay of 20–60 seconds (the synthesis time) between when a dispatch decision is made and when that work arrives in the GPU queue. This is an extraordinarily long delay for a control system — most industrial control loops operate on millisecond or second timescales. With a 30-second delay, the assistant calculated that proportional gain (Kp) should be around 0.01–0.1 and integral gain (Ki) even smaller — around 0.00014.

The assistant ultimately settled on a pragmatic design: a DispatchPacer struct that uses an exponential moving average (EMA) of the GPU inter-completion interval as a feed-forward rate estimate, with a PI correction on the smoothed GPU queue depth error. The feed-forward does the heavy lifting — it matches the dispatch rate to the GPU consumption rate in steady state — while the PI terms provide gentle corrections around that baseline. The gains were set to Kp=0.1 and Ki=0.01, with EMA smoothing factors of alpha_waiting=0.2 and alpha_gpu=0.3.

A bootstrap phase dispatches the target number of items at a fixed spacing before the first GPU completion, then switches to timer-based pacing at the PI-computed interval. This design acknowledges that the system has no information about the GPU rate until the first job completes — a full synthesis cycle must elapse before any feedback is available.

What the Target Message Reveals

Message <msg id=3446> is the assistant's status update after implementing the core pacer infrastructure. Three tasks are complete:

Add DispatchPacer struct with PI control + GPU rate EMA: The struct definition, including all the state variables (EMA filters, PI accumulator, gain parameters, timing state) has been written into engine.rs.
Add gpu_completion_count AtomicU64 shared state: A thread-safe counter that tracks how many GPU jobs have completed, shared between the dispatcher and the GPU worker finalizers.
Wire counter into GPU finalizer (happy path only): The counter is incremented in the successful completion path of each GPU job, right before the notify_one call that wakes the dispatcher. The fourth task — rewriting the dispatcher loop — is still in progress. This is the most complex piece: the actual control loop that waits for timers or GPU events, computes the next dispatch interval using the pacer, pops work from the priority queue, acquires memory budget, and dispatches to the GPU worker.

Assumptions Embedded in the Design

The PI controller design makes several assumptions that are worth examining. First, it assumes that the GPU completion rate is a stable, measurable signal that can serve as a feed-forward baseline. This works well when the GPU is the bottleneck, but breaks down when synthesis becomes the bottleneck — the GPU rate then reflects not GPU capacity but the rate at which synthesis can supply work. This is precisely the edge case the user would later identify, leading to the addition of a synthesis throughput cap.

Second, the design assumes that the EMA parameters and PI gains chosen (Kp=0.1, Ki=0.01, alpha_waiting=0.2, alpha_gpu=0.3) will work across different proof types (WinningPoSt, WindowPoSt, SnapDeals) that have different synthesis times and memory footprints. The user would later note that different proof types trigger pinned buffer re-allocations, which skew the initial measurements during startup transients.

Third, the bootstrap phase assumes that dispatching the target number of items at a fixed spacing is safe — that the system can handle the burst of synthesis jobs without overwhelming CPU resources. This assumption would prove problematic in the synthesis-constrained case.

The Mistakes That Would Surface

The most significant flaw in this design — one that the user would identify after the first deployment of the pacer (pacer1) — is the lack of a synthesis throughput cap. When synthesis is compute-constrained, the pacer drives the dispatch interval below the GPU rate to fill the queue, which floods the system with concurrent synthesis jobs, causing CPU contention and degrading overall throughput. The PI controller, designed to maintain queue depth, doesn't distinguish between "GPU is fast" and "synthesis is slow" — it sees a deficit and dispatches more aggressively, making the problem worse.

This is a classic control system failure mode: optimizing the wrong metric. The queue depth target is a proxy for the real goal, which is sustained system throughput. The assistant would later address this by adding a synthesis throughput cap that clamps the dispatch rate to not exceed the measured synthesis completion rate, with anti-windup logic that freezes the PI integral term when the cap is active.

Input and Output Knowledge

To understand this message, the reader needs knowledge of: the GPU proving pipeline architecture (synthesis → GPU queue → GPU proving), the concept of control theory (P and PI controllers, EMA filtering, feed-forward), the Rust concurrency primitives used (AtomicU64, Notify), and the history of the project including the pinned memory pool and the earlier P-controller attempts.

The message creates output knowledge about the state of the implementation: the pacer infrastructure is wired in, the completion counter is operational, and the dispatcher loop rewrite is the remaining work. It also implicitly documents the design decisions made in the preceding reasoning — the gain parameters, the EMA factors, the bootstrap strategy — which serve as a record for future debugging and tuning.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in <msg id=3429> reveals a sophisticated engineering thought process. It begins with the raw system parameters (20–60s synthesis, ~1s GPU time) and works through the implications for control theory. It calculates theoretical gains based on the delay, then pragmatically adjusts them upward for faster convergence. It considers the bootstrap problem — how to start without any feedback — and designs a two-phase approach. It evaluates the trade-off between aggressive dispatch (faster convergence) and conservative dispatch (less overshoot). The reasoning shows the assistant moving from abstract control theory to concrete implementation details, iterating on the design as it considers the real-world constraints of the system.

Conclusion

Message <msg id=3446> is a todo list update, but it represents far more than its simple structure suggests. It marks the completion of a carefully engineered PI-controlled dispatch pacer designed to solve a challenging control problem in GPU pipeline scheduling. The design embodies deep control theory knowledge, pragmatic engineering trade-offs, and an understanding of the specific characteristics of the zero-knowledge proof proving pipeline. The flaws that would later emerge — the synthesis throughput cap issue — don't diminish the achievement; they are the natural result of deploying a complex control system into a real-world environment where the assumptions are tested and refined. This message captures a moment of progress in an iterative engineering process, a checkpoint on the path from a simple semaphore to a sophisticated multi-variable control system.