The 47-Second Mirage: How Pipeline Fill Time Masqueraded as GPU Processing Rate

A Single User Message That Unraveled a Flawed Calibration

In the midst of a complex optimization effort to tune a PI-controlled dispatch pacer for GPU-accelerated zero-knowledge proof generation, a single user message landed with surgical precision, exposing a fundamental measurement error that had been silently corrupting the entire control system. The message, timestamped at the moment a freshly deployed binary began its first real workload, contained a log line and a sharp observation:

`` 2026-03-13T22:34:04.260673Z INFO cuzk_core::engine: pacer: GPU rate calibrated, switching to PI control ema_gpu_ms="47221" ema_waiting="0.8" 2026-03-13T22:34:34.794700Z INFO cuzk_core::engine: pacer: status total=10 waiting=0 ema_waiting="1.1" ema_gpu_ms="12731" ema_synth_ms="1000" interval_ms=6888 integral="20.00" rate_capped=false synth_rate_known=false bootstrap=false ``

>

"the 'calibrated' gpu rate seems painfully slow for a slow start, normally the gpu is processing in 1-2s (closer to 1s), If this number includes synthesis time it.. shouldnt't. That for the first burst should be measured from first warmup proof entering the gpu to the last one entering, then that should be used for the initial rate estimate"

This seemingly simple observation — that the GPU rate measurement of 47 seconds was implausible — would trigger a chain of debugging, code changes, and ultimately a complete rethinking of how the dispatch pacer measures and responds to GPU processing capacity. It is a textbook example of how a single measurement error, if left uncorrected, can cascade through a feedback control system and produce pathological behavior.

The Context: A PI Controller for GPU Dispatch

To understand why this message matters, one must understand what the dispatch pacer was trying to accomplish. The cuzk proving engine processes GPU-based zero-knowledge proofs in a pipelined architecture. Synthesis workers prepare proof partitions in CPU memory, then dispatch them to GPU workers for accelerated proving. The challenge is balancing two competing resources: CPU synthesis throughput and GPU processing throughput. If synthesis dispatches too quickly, the GPU queue grows and memory pressure builds. If synthesis dispatches too slowly, the GPU starves and utilization drops.

The PI (Proportional-Integral) controller was designed to solve this by maintaining a target number of partitions waiting in the GPU queue. It uses two key inputs: the current queue depth (how many items are waiting) and the GPU processing rate (how fast the GPU consumes items). The GPU rate serves as a feed-forward term — it tells the controller the base dispatch rate the GPU can sustain, so the PI controller only needs to make small corrections around that baseline.

The critical assumption baked into this design was that the GPU rate measurement accurately reflects how long the GPU takes to process a single proof partition. This assumption is what the user's message would demolish.

The 47-Second Illusion

The log line shows ema_gpu_ms="47221" — an exponential moving average of GPU processing time of approximately 47 seconds. The user immediately recognized this as wrong. Normally, GPU processing for a single partition takes 1-2 seconds. Why was the pacer seeing 47 seconds?

The answer lies in how the measurement was taken. The GPU rate was being measured as the time between when a proof partition was dispatched to the GPU and when the GPU completed processing it. But during the initial pipeline fill — the first burst of work when the system starts — the GPU workers are idle and waiting. When the first partition arrives, it starts processing immediately, but the completion time includes not just the GPU processing but also the time spent waiting for all the other partitions in the burst to be dispatched and start processing.

In other words, the measurement was contaminated by pipeline fill time. The first partition dispatched might complete 47 seconds later, but most of that time was spent waiting for the rest of the pipeline to fill, not in actual GPU computation. The GPU itself was only busy for about 1 second per partition.

This is a classic measurement pitfall in pipelined systems: the latency of the first item through an empty pipeline includes the pipeline fill time, not just the processing time. The controller was effectively measuring the wrong thing.

The User's Proposed Fix: First-to-Last Measurement

The user's suggestion was precise: "That for the first burst should be measured from first warmup proof entering the gpu to the last one entering, then that should be used for the initial rate estimate."

This insight recognizes that the relevant measurement for the GPU rate is not how long an individual item takes from dispatch to completion (which includes pipeline fill), but rather the throughput rate — how many items the GPU can process per unit time when the pipeline is full. By measuring from the first item entering the GPU to the last item entering (not completing), and dividing by the number of items, you get the true per-item GPU processing time, uncontaminated by pipeline fill.

This is the difference between measuring latency and measuring throughput. The controller needs throughput to set the dispatch rate, but the measurement was capturing latency.

The Cascade: How One Bad Measurement Breaks a Control System

The user's message was more than just a bug report — it was a diagnosis of a systemic problem that would later be identified as a "vicious cycle." When the GPU rate is measured as 47 seconds instead of 1 second, the PI controller's feed-forward term tells it to dispatch at a rate of one partition every 47 seconds. This slow dispatch rate means synthesis workers produce partitions slowly (because the dispatcher isn't consuming their output), which in turn means the synthesis throughput appears low. If a synthesis throughput cap exists (as it did in this version), that cap would tighten further based on the low synthesis rate, creating a self-reinforcing collapse loop: slow dispatch → fewer concurrent syntheses → slower synthesis throughput → tighter cap → even slower dispatch.

The user's observation was the first crack in the facade. Without this message, the team might have continued tuning PI parameters (kp, ki, integral limits) without ever addressing the root cause — the measurement itself was wrong. All the tuning in the world cannot fix a controller that is being fed fundamentally incorrect state estimates.

Assumptions Laid Bare

This message reveals several assumptions that had been made, consciously or unconsciously:

  1. The GPU completion time equals GPU processing time. This assumption conflates system latency with processing throughput. In a pipelined system with multiple workers, completion time includes queueing, scheduling, and pipeline fill effects.
  2. The first burst of work is representative. The initial pipeline fill is a transient state, not a steady-state condition. Using transient measurements to calibrate a steady-state controller introduces systematic bias.
  3. EMA smoothing will fix bad initial values. The exponential moving average was intended to smooth out noise, but it cannot correct a fundamentally wrong measurement. Worse, the EMA's inertia means that a bad initial calibration takes a long time to decay — the 47-second value would drag down the rate estimate for many subsequent updates.
  4. The measurement is self-consistent. The user implicitly assumed that if the GPU rate were truly 47 seconds, the system would behave differently than observed. The contradiction between the measured rate and the expected behavior was the clue that something was wrong.

The Knowledge Created

This message created several pieces of actionable knowledge:

The Broader Lesson

This message exemplifies a pattern that recurs across engineering disciplines: the most impactful debugging insight often comes not from diving deeper into the code, but from questioning the measurements. The user did not suggest a code change to the PI controller's parameters or the synthesis throughput cap. Instead, they questioned the fundamental measurement that everything else was built on.

In control theory, this is known as the observability problem — if you cannot accurately observe the state of your system, you cannot control it. The 47-second GPU rate was not just an inaccurate number; it was a failure of observability. The system was blind, and the controller was flying on bad instruments.

The message also demonstrates the value of domain expertise in interpreting system behavior. The user knew that GPU processing takes 1-2 seconds, not 47 seconds. This domain knowledge — the expected physical performance of the hardware — provided an independent sanity check that the measurement had to be wrong. Without that check, the 47-second value might have been accepted as reality, and the team would have spent days or weeks tuning a controller that was fundamentally misinformed.

Conclusion

The user's message at index 3537 is a masterclass in diagnostic thinking. In two log lines and a paragraph of analysis, it identified a measurement error, explained why it was wrong, proposed a concrete fix, and implicitly warned of the cascading consequences if the error were left uncorrected. It is the kind of message that changes the trajectory of a debugging effort — not by adding complexity, but by revealing that the foundation was built on sand. The subsequent work to skip the first GPU completion, then to measure GPU processing duration directly from GPU workers via an AtomicU64, and ultimately to remove the synthesis throughput cap entirely, all trace back to this single moment of clarity. The 47-second mirage was dispelled, and the real work of building a correctly calibrated control system could begin.