The Hidden Complexity of Measuring GPU Throughput with Pipelined Workers

In the middle of an intense debugging session focused on GPU underutilization in a zero-knowledge proof system, the assistant issued a deceptively simple message:

[edit] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

This is message [msg 3560] in the conversation, and on its surface it appears to be nothing more than a routine confirmation that a code edit was applied. But this single line is the culmination of a deep chain of reasoning spanning dozens of messages, representing a fundamental rethinking of how to measure GPU processing throughput in a system with multiple pipelined GPU workers. To understand why this edit matters, one must trace the intellectual journey that led to it.

The Problem: Why Simple Rate Measurement Fails

The story begins with the user's observation at [msg 3541]: "tracking gpu processing time isn't that simple because there are two lightly pipelined gpu workers, not just one." This seemingly simple remark exposed a critical flaw in the assistant's approach to measuring GPU consumption rate.

The assistant had been using a waiting > 0 heuristic to determine whether the GPU was busy — the idea being that if items were waiting in the GPU work queue, the GPU must be actively processing. But with two pipelined workers, this assumption collapses. As the assistant's own reasoning at [msg 3542] makes clear:

"Worker A pops item, starts processing (1s). Worker B pops item, starts processing (1s). Queue might now be empty (waiting=0). Worker A finishes → completion event. waiting=0 → we'd skip this measurement. Worker B finishes → completion event. waiting=0 → we'd skip this measurement."

The queue length reflects what is waiting, not what is active. Two workers can both be fully utilized with an empty queue, as long as items arrive at roughly the rate workers consume them. The waiting > 0 proxy is fundamentally broken.

The Pivot: Direct Measurement from GPU Workers

The assistant's reasoning at [msg 3542] shows a remarkable evolution. It considers multiple approaches — tracking an active worker counter, using bootstrap-phase burst measurement, measuring inter-completion intervals — before arriving at the cleanest solution:

"The cleanest approach is to have GPU workers publish their last processing duration to a shared atomic, then the pacer reads that value on each completion event and feeds it into an exponential moving average, dividing by the number of workers to get the effective dispatch interval."

This is the key insight. Instead of trying to infer GPU busyness from queue depth or completion timing (both contaminated by idle periods and pipeline-fill effects), measure the actual GPU processing duration directly from the workers that perform the work. The GPU workers already time themselves for logging purposes — the assistant discovered this at [msg 3544] when it found that gpu_result.gpu_duration was already available in the finalizer code. The data was there all along, just not being used for pacing.

The formula is elegant: effective_interval = avg_gpu_processing_s / num_gpu_workers. With two workers each taking one second per partition, completions happen every 0.5 seconds, so the pacer should dispatch every 0.5 seconds. This is immune to idle time, pipeline fill, and correctly handles any number of interleaved workers.

What This Edit Actually Does

By the time we reach message [msg 3560], the assistant has already made several preparatory edits. At [msg 3551] it added the gpu_processing_total_ns: Arc<AtomicU64> atomic accumulator. At [msg 3553] it rewrote the update() method in DispatchPacer to use processing time instead of inter-completion intervals. At [msg 3554] it updated the interval() method to compute ema_gpu_processing_s / num_gpu_workers.

Message [msg 3560] is the edit that wires it all together at the creation site — it creates the atomic, passes it into the DispatchPacer::new() call, and threads num_gpu_workers into the pacer's constructor. Without this edit, all the other changes would be disconnected pieces. This is the integration point that makes the new measurement approach operational.

The assistant's plan at [msg 3559] lists exactly what this edit accomplishes:

Add gpu_processing_total_ns atomic at creation site
Pass it to pacer update calls (all 4 sites)
Wire it into GPU workers (finalizer + sync path)
Update DispatchPacer::new() call to pass num_gpu_workers Message [msg 3560] handles items 1 and 4 — the creation and constructor wiring. The remaining items (2 and 3) are handled in subsequent messages [msg 3563] through [msg 3571].

The Thinking Process: From Flawed Heuristic to Robust Measurement

What makes this message interesting is not the edit itself but the reasoning that produced it. The assistant's thinking at [msg 3542] reveals a sophisticated debugging process. It starts by acknowledging the user's correction, then systematically works through the implications:

First, it identifies why waiting > 0 fails: with two workers, both can be busy with an empty queue. This is a classic concurrency pitfall — confusing queue depth with utilization.

Second, it considers tracking an active worker counter using atomics, but realizes the timing is wrong: "by the time the pacer wakes up from the completion notification, the active counter has already been decremented."

Third, it considers bootstrap-phase burst measurement, but recognizes this only works during initialization.

Fourth, it arrives at the direct measurement approach: have workers publish their processing duration to a shared atomic accumulator. The assistant considers the race condition problem — concurrent writes from multiple workers — and settles on fetch_add to accumulate total processing time, then computes the average by dividing by the delta in completion count.

This is a textbook example of choosing the right measurement for the right purpose. The assistant could have continued hacking on the waiting > 0 heuristic, adding more conditions and special cases. Instead, it stepped back and asked: what is the actual quantity we want to measure? The answer is "how long does the GPU take to process one partition?" — and the workers already know this. Measuring it directly eliminates an entire class of estimation errors.

Assumptions and Their Validity

The assistant makes several assumptions in this approach. It assumes that num_gpu_workers is a static value known at construction time — which is true in this codebase, as confirmed at [msg 3561] where num_workers = gpu_ordinals.len() * gpu_workers_per_device. It assumes that GPU processing duration is stable enough that an exponential moving average will converge to a useful value — reasonable given that proof generation for a fixed circuit size has predictable computational cost. It assumes that fetch_add from multiple workers will not cause significant contention — also reasonable since GPU completions are relatively infrequent (seconds apart) compared to CPU clock speeds.

One subtle assumption deserves scrutiny: the assistant assumes that dividing average processing time by the number of workers gives the correct dispatch interval. This holds when workers are truly independent and the GPU can process multiple partitions concurrently without significant resource contention. The assistant acknowledges this at [msg 3542]: "GPU workers have some pipelining where they share resources, so actual processing time increases slightly under contention — maybe 1.2 seconds per item when both are active instead of 1 second alone." The EMA naturally captures this effect, so the formula adapts automatically.

The Broader Context

This edit is part of a much larger effort spanning segments 21-26 of the conversation (see [segment 21] through [segment 26]). The team has been fighting GPU underutilization for days, trying various approaches: adding timing instrumentation ([segment 21]), designing a zero-copy pinned memory pool ([segment 22]), wiring it into the engine ([segment 23]), deploying and debugging it ([segment 24]), and iterating on a PI-controlled dispatch pacer ([segment 25]). The current segment ([segment 26]) is about tuning that pacer.

The specific problem this edit addresses — measuring GPU processing rate correctly with multiple workers — was a critical blocker. The PI controller's feed-forward term depends on knowing the GPU's consumption rate. If that measurement is wrong, the controller either dispatches too slowly (underutilizing the GPU) or too quickly (flooding the queue and causing memory pressure). Getting this measurement right is essential for the entire pacing system to work.

Conclusion

Message [msg 3560] is a turning point. It represents the moment when the assistant abandoned an increasingly tangled set of heuristics and adopted a clean, direct measurement approach. The edit itself is small — a few lines at the atomic creation site — but the reasoning behind it represents hours of debugging, analysis, and design iteration. It is a reminder that in systems programming, the hardest problems are often not about writing code but about knowing what to measure and how to measure it correctly.