From Proxy to Direct Measurement: How GPU Processing Time Tracking Was Fixed for Interleaved Workers

In the course of tuning a GPU dispatch pacer for a zero-knowledge proof system, a single message marked a critical turning point. The message, at index 3544 in the conversation, is deceptively short — a few lines of code analysis and a proposed design. But it represents the moment when a fundamentally flawed measurement approach was rejected in favor of a clean, correct one. This article examines that message in depth: the reasoning that led to it, the assumptions it corrected, and the design it set in motion.

The Problem: A Self-Reinforcing Collapse

To understand why this message matters, we need to step back into the context. The system in question is a GPU-accelerated proving pipeline (cuzk) that uses a PI (proportional-integral) controller to pace the dispatch of work items to the GPU. The pacer's job is to keep the GPU queue full enough that the GPU never idles, but not so full that memory is exhausted. It uses an exponential moving average (EMA) of the GPU's inter-completion interval as a feed-forward signal — if the GPU processes items every 1 second, the pacer should dispatch roughly every 1 second.

But the system was collapsing. The user reported logs showing that the EMA of GPU processing time was stuck at 47 seconds — the time from process start to the first GPU completion — rather than the actual ~1 second per partition. This initial bad measurement poisoned the EMA, which dragged down painfully slowly. Even worse, the slow dispatch caused by this inflated rate estimate meant the GPU queue emptied, which meant the next GPU completion included idle time, which further inflated the rate estimate. This created a vicious cycle: slow dispatch → empty queue → GPU idle → measured rate drops → even slower dispatch.

The assistant's first attempted fix was to skip the first GPU completion (which includes pipeline fill time) and only update the GPU rate estimate when waiting > 0 — that is, only measure intervals when there are still items in the GPU work queue, indicating back-to-back GPU processing.

The User's Correction: Two Workers, Not One

The user's response at message 3541 punctured this approach: "tracking gpu processing time isn't that simple because there are two lightly pipelined gpu workers, not just one."

This is the key insight that the assistant had missed. The GPU has two interleaved workers per device. Both can be actively processing simultaneously. When worker A pops an item and starts processing, the queue length decreases by one. When worker B pops the next item, the queue decreases by another one. Now the queue might be empty — waiting = 0 — but both workers are busy. When worker A finishes and a completion event fires, the pacer checks waiting > 0, finds it false, and skips the measurement. Same for worker B. The system would never measure the GPU rate, even though the GPU is fully utilized.

The queue length reflects what is waiting, not what is active. With two pipelined workers, they can both be fully utilized with an empty queue, as long as items arrive at roughly the rate workers consume them. The waiting > 0 proxy is fundamentally flawed.

The Subject Message: A New Design

Message 3544 is the assistant's response to this correction. Let me quote it exactly:

I can see it. The GPU worker already gets gpu_result.gpu_duration (line 3120-3127). The finalizer at line 3136 increments the completion count. I'll add a gpu_processing_total_ns: Arc<AtomicU64> that workers fetch_add their actual processing duration into. Then the pacer computes:

>

- avg_gpu_processing_s = delta_ns / delta_completions - effective_interval = avg_gpu_processing_s / num_gpu_workers

>

This is immune to idle time, pipeline fill, and correctly handles 2 interleaved workers.

The message is short, but it packs a significant conceptual shift. Let's unpack what it contains.

The Reasoning: Three Key Insights

First insight: measure directly, not by proxy. The previous approach tried to infer GPU processing time from the interval between completion events. This is a proxy measurement — it assumes that the time between completions equals the time the GPU spent working. But this assumption breaks whenever the pipeline has gaps: during pipeline fill (the first completion includes synthesis time), during idle periods (the interval includes time the GPU was waiting for work), and with multiple workers (completions are staggered, not one-to-one with processing time). The direct measurement approach eliminates all these problems by having each GPU worker report exactly how long it spent computing.

Second insight: the atomic accumulator pattern. The assistant proposes using Arc<AtomicU64> with fetch_add — a lock-free concurrent accumulator. Each GPU worker, when it finishes processing a partition, atomically adds its processing duration (in nanoseconds) to a shared total. The pacer, on each update cycle, reads the current total and computes the delta since the last read. Dividing this delta by the delta in completion count gives the average GPU processing time per partition over that interval. This is race-free, requires no locks, and gracefully handles concurrent updates from multiple workers.

Third insight: the worker-count normalization. The key formula is effective_interval = avg_gpu_processing_s / num_gpu_workers. With two workers each taking 1 second to process a partition, completions happen every 0.5 seconds (staggered). The average processing time is 1 second, divided by 2 workers gives 0.5 seconds — the correct dispatch interval to keep both workers fed. This normalization is what makes the approach immune to the pipelining problem. It doesn't matter how many workers there are or how they're staggered; the formula always yields the right dispatch rate.

What the Message Assumes

The message makes several implicit assumptions that are worth examining:

  1. The GPU worker already measures its own processing time. The assistant confirms this by referencing lines 3120-3127, where gpu_result.gpu_duration is available. This is a fortunate coincidence — the GPU worker already times itself for logging purposes, so the data is available without adding new instrumentation.
  2. The processing duration is the right thing to measure. The assumption is that GPU processing time per partition is a stable characteristic of the hardware and workload, not something that varies wildly with queue depth or contention. This is reasonable — the GPU's compute time for a given proof type is determined by the NTT and MSM operations, which are deterministic given the partition size.
  3. Atomic fetch_add is sufficient for concurrent updates. With two workers potentially finishing at nearly the same instant, fetch_add provides the necessary atomicity. The EMA is forgiving of occasional lost samples, so even if a race condition caused a rare missed update, the system would still converge.
  4. The number of GPU workers is known and constant. The formula divides by num_gpu_workers, which is a configuration parameter. This assumes the worker count doesn't change during operation.

What the Message Gets Right (and What It Doesn't)

The design is correct in its core insight: direct measurement of processing time eliminates the proxy problems. But there are subtleties that the message doesn't address:

Contention effects. The message doesn't discuss what happens when two workers share GPU resources. In practice, when both workers are active, each partition might take slightly longer (e.g., 1.2 seconds instead of 1.0) due to resource contention. The EMA naturally captures this, so the formula adapts — but the message doesn't explicitly acknowledge this dynamic.

The first measurement problem. The message inherits the earlier problem: the first GPU completion's processing time might be measured, but the pacer needs an initial estimate before any completions happen. The message doesn't address bootstrap — how the pacer gets its first rate estimate. This was handled in the surrounding conversation by a bootstrap phase that dispatches a burst of items and measures the inter-completion intervals during the burst.

The averaging window. Using delta_ns / delta_completions gives an average over a batch of completions. If the batch size is 1, this is a per-partition measurement. If the batch is larger (multiple completions between pacer wake-ups), it's an average. The message doesn't specify how the pacer handles batch sizes, which could affect the EMA's responsiveness.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of the GPU pipeline architecture. Specifically, that there are two interleaved GPU workers per device, and that they process items concurrently with staggered completion times.
  2. Understanding of the PI controller and pacer design. The pacer uses an EMA of GPU processing time as a feed-forward term to compute the target dispatch interval. The effective_interval becomes the base interval that the PI controller then adjusts based on the error between actual and target queue depth.
  3. Familiarity with the codebase structure. The assistant references specific line numbers in engine.rs — the GPU worker's finalizer at line 3136, the gpu_duration field at lines 3120-3127. This requires knowing where in the codebase the GPU workers live and how they report results.
  4. Understanding of atomic operations in Rust. The Arc<AtomicU64> and fetch_add pattern is standard for lock-free concurrent accumulation, but a reader unfamiliar with Rust's concurrency primitives might miss the design's elegance.

Output Knowledge Created

The message creates a clear design specification:

  1. A new shared atomic counter (gpu_processing_total_ns) that will be added to the engine's state, alongside the existing gpu_completion_count.
  2. A modification to the GPU worker finalizer to extract gpu_duration from the result and atomically add it to the total before incrementing the completion count.
  3. A modification to the pacer's update logic to compute the average GPU processing time per partition from the delta in total processing time and delta in completion count, then divide by the number of GPU workers to get the effective dispatch interval.
  4. A replacement for the flawed waiting > 0 logic — the new approach doesn't need to check queue depth at all, because the processing time measurement is inherently immune to idle contamination.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in the preceding message (msg 3542) shows a thorough exploration of alternatives before arriving at this design. The reasoning considers:

Conclusion

Message 3544 is a small message with a big conceptual shift. It replaces a flawed proxy measurement (inter-completion interval, filtered by queue depth) with a direct measurement (actual GPU processing time, accumulated atomically). The design is elegant because it eliminates the need for the complex filtering logic that the previous approach required — no more skipping first completions, no more checking queue depth, no more worrying about idle contamination. The processing time is always valid, regardless of pipeline state.

This message exemplifies a common pattern in systems engineering: when a measurement is unreliable, don't add more filtering and heuristics — find a way to measure the thing directly. The GPU workers already know how long they spent processing; the fix is simply to ask them.