The Bootstrap Timer Branch: Wiring GPU Processing Time Into the Dispatch Pacer

A Single Edit in a Chain of Precision Instrumentation

The message at index 3566 is deceptively brief. It reads:

Third (bootstrap timer branch): [edit] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

On its surface, this is merely a status notification—the assistant reporting that it has applied an edit to a Rust source file. But this message is the third of four edits that together complete a critical refactoring of the GPU dispatch pacing system in the CuZK zero-knowledge proving engine. Understanding why this particular edit exists, what it changes, and the reasoning behind it requires tracing back through a cascade of failed approaches, design iterations, and fundamental insights about how GPU pipelines actually behave under load.

The Problem: You Cannot Measure What You Cannot See

The story begins with a seemingly straightforward goal: the dispatch pacer needs to know how fast the GPU can consume work. The pacer is a PI (proportional-integral) controller that decides when to dispatch the next proof partition to the GPU. If it dispatches too slowly, the GPU sits idle and throughput suffers. If it dispatches too quickly, the GPU work queue grows unboundedly, consuming memory and potentially causing out-of-memory crashes.

The original approach measured the GPU consumption rate by tracking the inter-completion interval—the time between successive GPU job completions. This seemed natural: if completions happen every 500 milliseconds, dispatch every 500 milliseconds. But this approach had a fatal flaw exposed during pipeline fill: the first GPU completion includes not just GPU processing time but also the time spent waiting for the pipeline to fill, for synthesis to produce the first partition, and for data transfer to complete. This "pipeline fill" time could be 47 seconds, completely swamping the actual GPU processing time of roughly 1 second. The exponential moving average (EMA) would then drag downward agonizingly slowly, taking hundreds of completions to converge to the true rate.

The assistant's first attempted fix was to skip the first GPU completion and only update the GPU rate measurement when the waiting queue had items in it (waiting > 0). The intuition was sound: if the queue is non-empty when a completion fires, the GPU must have been busy, so the interval is a valid measurement of GPU throughput. But this assumption collapsed under scrutiny.

The Two-Worker Problem

The user identified the flaw in message 3541: "tracking gpu processing time isn't that simple because there are two lightly pipelined gpu workers, not just one." This single observation unraveled the entire approach.

With two interleaved GPU workers per device, the queue length becomes an unreliable proxy for GPU busyness. Consider the timeline: Worker A pops an item from the queue and starts processing (taking ~1 second). Worker B pops the next item and starts processing (also ~1 second). The queue is now empty. Worker A finishes and fires a completion event. The pacer checks waiting > 0—it's false—so it skips the measurement. Worker B finishes, same thing. Both completions are discarded even though both workers were fully utilized. The queue length reflects what is waiting, not what is active, and with two workers consuming items in parallel, the queue can be persistently empty while the GPU is running at full capacity.

The assistant's reasoning in message 3542 shows the pivot: "What I actually need is to measure the GPU's maximum throughput when it's saturated. The simplest way is to have each GPU worker track its own processing duration, then calculate effective throughput as the number of workers divided by their average processing time."

A Fundamentally Different Measurement Strategy

The new approach abandons inter-completion intervals entirely. Instead, it measures actual GPU processing duration directly from the workers. Each GPU worker already times its own processing for logging purposes—it knows exactly how long it spent computing. The insight is to publish this duration to a shared atomic counter (gpu_processing_total_ns: Arc<AtomicU64>) that workers accumulate into using fetch_add. The pacer then reads this accumulator on each completion event, computes the delta in total processing time divided by the delta in completions to get the average per-partition GPU processing time, and derives the effective dispatch interval as avg_gpu_processing / num_workers.

This approach is immune to both pipeline fill contamination (the first completion's duration is a valid measurement of actual GPU time, not pipeline wait time) and idle time contamination (idle periods simply don't produce completions, so the EMA holds its last known value). It correctly handles two interleaved workers because the formula inherently accounts for parallelism: with two workers each taking 1 second, the effective dispatch interval is 0.5 seconds, which keeps both workers saturated.

The Four Call Sites

The refactoring required updating every location where pacer.update() is called to pass the new gpu_proc_ns parameter (the accumulated GPU processing nanoseconds). There are four such call sites in the dispatcher loop, each serving a distinct purpose:

  1. GPU event completion (msg 3564): When a GPU job finishes, the pacer must update its rate measurement with the latest processing time data.
  2. Bootstrap wait loop (msg 3565): During the initial bootstrap phase, the dispatcher waits for enough GPU completions to establish a stable rate estimate. This loop also calls pacer.update() to keep the pacer state current.
  3. Bootstrap timer branch (msg 3566 — the subject message): After the bootstrap phase completes, the dispatcher enters a timer-based scheduling loop. On each timer tick, it calls pacer.update() to refresh the pacer state even when no GPU completions have occurred (e.g., to decay the EMA or update the waiting count). This is the "bootstrap timer branch" — the timer tick path that runs after the initial calibration phase.
  4. Waiting for work (msg 3567): When the dispatcher has no work to dispatch (the GPU work queue is empty and no synthesis results are ready), it enters a wait state. On each iteration of this wait loop, it calls pacer.update() to keep the pacer informed that no progress is being made, allowing the PI controller to adjust its integral term appropriately.

What This Specific Edit Achieves

The subject message applies the edit to the third call site—the bootstrap timer branch. This is the path that executes on periodic timer ticks after the bootstrap phase has completed and the pacer has entered steady-state PI control. The timer tick interval is itself computed from the pacer's feed-forward estimate, creating a feedback loop: the pacer's rate estimate determines how often the timer fires, and the timer firing updates the pacer's state.

By passing gpu_proc_ns to this call site, the assistant ensures that even on timer ticks (which may occur between GPU completions), the pacer has access to the latest accumulated processing time. This is important because the pacer's update() method computes the delta in processing time since the last call, and if a GPU completion occurred between timer ticks, the accumulated processing time needs to be visible to the pacer on the next tick. Without this parameter, the timer branch would operate with stale rate information, potentially causing the PI controller to make incorrect decisions.

Assumptions and Design Decisions

The implementation makes several key assumptions. First, it assumes that fetch_add on an AtomicU64 is an acceptable synchronization mechanism for accumulating processing times from multiple GPU workers. This is safe (atomic operations are lock-free and wait-free on x86-64), but it means the accumulator can theoretically grow without bound. In practice, the pacer reads the accumulator on every update and computes deltas, so the absolute value never matters—only the difference between successive reads.

Second, it assumes that the number of GPU workers is known and stable. The num_workers value is computed once during engine initialization as gpu_ordinals.len() * gpu_workers_per_device. If workers could be dynamically added or removed, this assumption would break, but in the current architecture, the worker pool is fixed for the lifetime of the engine.

Third, it assumes that processing time measurements from individual workers are comparable and additive. This holds because all workers run the same GPU computation on the same hardware, and the durations are measured in nanoseconds using the same clock source.

Knowledge Required and Knowledge Created

To understand this message, one needs knowledge of: GPU pipeline architecture (especially the concept of interleaved/multi-worker dispatch), PI control theory (how feed-forward and feedback terms interact in a scheduling controller), Rust atomics and concurrency primitives (Arc<AtomicU64>, fetch_add), the CuZK engine's codebase structure (the dispatcher loop, pacer state machine, and GPU worker lifecycle), and exponential moving averages for smoothing noisy measurements.

The edit creates new knowledge about the GPU's true processing characteristics. By measuring actual processing duration rather than inter-completion intervals, the system can now distinguish between "GPU is busy computing" and "GPU is idle waiting for work" — a distinction that was previously invisible. This enables more precise dispatch pacing and, ultimately, higher GPU utilization.

The Broader Significance

This single edit is part of a larger narrative about the difficulty of measuring system behavior in complex pipelines. The naive approach (inter-completion intervals) seemed correct until the two-worker scenario revealed its hidden assumption: that completions are sequential rather than parallel. The fix required not a tweak to the existing measurement but a complete reconceptualization of what should be measured. Instead of inferring GPU busyness from external observables (queue depth, completion timing), the new approach measures it directly at the source.

This pattern recurs throughout systems engineering: indirect measurements are fragile because they depend on unstated assumptions about the relationship between the observable and the target quantity. Direct measurement, when feasible, is almost always more robust. The GPU workers already had the processing duration available — the challenge was recognizing it as the right signal to feed into the controller.

The bootstrap timer branch edit, for all its apparent simplicity, represents the completion of this conceptual shift. Every call site that touches the pacer must now speak in terms of actual processing time, not inferred intervals. The system is now instrumented to see the GPU's true behavior, and that visibility is what enables the PI controller to make informed decisions.