The Quiet Wiring: How One Atomic Clone Fixed GPU Rate Measurement

In a high-performance GPU proving pipeline, the difference between a system that hums along at peak throughput and one that stutters and stalls often comes down to a single number: the dispatch interval. How often should the CPU dispatch new work to the GPU? Too fast, and the GPU queue overflows, memory pressure spikes, and the system collapses. Too slow, and the GPU idles, wasting its parallel processing power. Getting this interval right requires accurate measurement of how fast the GPU actually consumes work — and as the developers of the CuZK proving engine discovered, that measurement is surprisingly easy to get wrong.

Message [msg 3569] is a deceptively simple edit in a long chain of changes aimed at fixing this measurement. The assistant writes:

Now clone it into the finalizer task alongside fin_gpu_count: [edit] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

This one-liner is the culmination of a deep debugging journey spanning dozens of messages. To understand why this single clone operation matters, we must trace the reasoning that led to it.

The Pipeline Problem

The CuZK proving engine uses a dispatch pacer — a PI (proportional-integral) controller that decides when to send the next partition to the GPU. The pacer relies on a feed-forward term: an estimate of how fast the GPU processes work, used to compute a baseline dispatch interval. Originally, this estimate was derived from the inter-completion interval — the time between successive GPU completion events. The logic was straightforward: if completions arrive every 1 second, dispatch every 1 second.

But this broke badly when the system used two pipelined GPU workers per device. With two workers processing in parallel, completions arrive every ~0.5 seconds even though each individual partition takes ~1 second to process. The inter-completion interval reflected the staggered output of two workers, not the actual processing time of a single partition. Worse, the pacer had been using a waiting > 0 heuristic to decide whether the GPU was busy — skipping rate updates when the queue was empty. With two workers, both could be actively processing with an empty queue, causing the pacer to skip measurements during the very periods when the GPU was most saturated.

A Better Measurement Strategy

The assistant recognized that the root cause was measuring the wrong signal. Instead of inferring GPU busyness from queue depth and completion timing, the pacer should measure the actual processing duration that each GPU worker reports. Every GPU worker already times its own work — the gpu_duration value was computed internally for logging purposes. The insight was to publish this duration to a shared atomic counter, then compute the effective dispatch interval as:

effective_interval = avg_processing_time / num_gpu_workers

This formula is immune to both pipeline fill (the initial period before the first completion) and idle time (when no workers are busy). With two workers each taking 1 second, the effective interval becomes 0.5 seconds — exactly the rate needed to keep both workers saturated. The measurement is always valid because processing duration is a property of the hardware and the partition size, not of queue dynamics.

The Data Flow

Implementing this required threading a new Arc<AtomicU64> called gpu_processing_total_ns through the engine's complex concurrency architecture. The data flow works like this:

  1. Creation: The atomic is created alongside existing atomics like gpu_completion_count in the engine's setup code ([msg 3560]).
  2. Clone to dispatcher: A clone is passed into the dispatcher task so the pacer can read accumulated processing time ([msg 3563]).
  3. Clone to worker spawn: At the GPU worker spawn site, a clone is created for each worker to use ([msg 3568]).
  4. Clone to finalizer (message [msg 3569]): Inside each worker's spawned thread, another clone is created for the finalizer — the async task that runs after GPU computation completes. This is the step in our subject message.
  5. Accumulation: In the finalizer, the gpu_duration from the GPU result is extracted and added to the atomic via fetch_add ([msg 3572]).
  6. Reading: On each pacer update cycle, the dispatcher reads the atomic, computes the delta in total processing time divided by the delta in completion count, and feeds the result into an exponential moving average ([msg 3553]).

Why This Particular Clone Matters

Message [msg 3569] is the step where the atomic reference is cloned into the finalizer task's scope. Without this clone, the finalizer — which runs as a tokio::spawn async task on a separate thread — would have no access to the atomic counter. The GPU processing duration, already measured and available in the gpu_duration field of the GPU result, would be discarded after being logged, never reaching the pacer.

The clone is placed "alongside fin_gpu_count" — the existing atomic that tracks how many GPU completions have occurred. This pairing is deliberate: the pacer needs both the count (how many partitions finished) and the total processing time (how long they took) to compute the average. The two atomics are updated together in the finalizer, ensuring that the delta in time corresponds to the delta in completions.

Assumptions and Correctness

The design makes several assumptions that are worth examining:

Atomic accumulation is race-free: Multiple GPU workers may complete simultaneously and call fetch_add concurrently. This is safe because AtomicU64::fetch_add is atomic — the total will be the sum of all durations, regardless of interleaving. The pacer reads the total and the completion count together, so the average is consistent even if individual reads are slightly staggered.

EMA smooths out noise: The exponential moving average naturally handles the fact that individual partition processing times vary. A single long or short partition doesn't dramatically shift the estimate. This also means that occasional lost samples (if a worker somehow fails to accumulate) are harmless — the EMA converges back to the true value over time.

Processing time scales with contention: When two workers run simultaneously, they share GPU resources (memory bandwidth, compute units), potentially increasing per-partition processing time. The EMA captures this effect naturally: if contention makes each partition take 1.2 seconds instead of 1.0, the effective interval becomes 0.6 seconds instead of 0.5, slowing dispatch slightly. This is correct behavior — the pacer should dispatch at the rate the GPU can actually sustain, not at some idealized rate.

The Broader Context

This edit is one small piece of a much larger effort to stabilize the CuZK proving pipeline. The dispatch pacer had been through multiple iterations: a semaphore-based throttle, a P-controller, a PI controller, and now this refined measurement strategy. Each iteration revealed new failure modes — integral saturation, re-bootstrap cycles, synthesis throughput collapse loops. The processing-time measurement approach, deployed as /data/cuzk-synthcap2, was designed to be the most robust yet, immune to the pipeline-fill and idle-time contamination that plagued earlier attempts.

Message [msg 3569] is not dramatic. It does not introduce a new algorithm or restructure the codebase. It is a single line — a clone of an atomic reference into the right scope. But in a concurrent system, wiring is everything. A signal that never reaches its destination is as useless as no signal at all. This quiet edit ensures that the GPU's own measurement of its work reaches the controller that decides how fast to feed it, closing the feedback loop that keeps the pipeline running at peak throughput.