The Pivot to Direct Measurement: Deploying synthcap2 in the CuZK GPU Dispatch Pacer

Introduction

In the iterative development of a high-performance GPU proving pipeline for the CuZK zero-knowledge proving engine, few moments are as decisive as the one captured in message <msg id=3576>. On its surface, the message is unremarkable: a clean compile confirmation followed by a Docker build invocation. But this message represents a critical inflection point — the culmination of a fundamental redesign of how the GPU dispatch pacer measures processing rates, and the deployment of the "synthcap2" binary that embodied that redesign. To understand why this message matters, one must trace the reasoning chain that led to it, the flawed assumptions it corrected, and the architectural insight it encoded.

The Subject Message

The assistant wrote:

Clean compile — only pre-existing warnings. Now build, extract, and deploy:

`` DOCKER_BUILDKIT=1 docker build --no-cache -f Dockerfile.cuzk-rebuild -t cuzk-rebuild:synthcap2 . 2>&1 | tail -15 #15 107.3 Compiling cuzk-server v0.1.0 (/build/extern/cuzk/cuzk-server) #15 108.5 warning: cuzk-core (lib) generated 4 warnings (run cargo fix --lib -p cuzk-core to apply 1 suggestion) #15 108.5 Compiling cuzk-daemon v0.1.0 (/build/extern/cuzk/cuzk-daemon) #15 114.2 Finished release` profile [optimized] target(s) in 1m 52s #15 DONE 114.5s

#16 [stage-1 1/1] COPY --from=builder /build/extern/cuzk/target/release/cuzk-daemon /cuzk #16 DONE 0.1s

#17 exporting to image #17 exporting l... ```

The brevity is deceptive. The phrase "Clean compile — only pre-existing warnings" signals that a complex set of edits — spanning multiple files, touching atomics, pacer state, worker spawn logic, and finalizer paths — all landed without introducing new issues. The Docker build output shows the binary emerging from a multi-stage rebuild pipeline, ready for extraction and deployment to the production server at 141.0.85.211.

The Reasoning Behind the Message

This message was written because the previous approach to GPU rate calibration had failed. The story begins with the "synthcap1" binary, which introduced a synthesis throughput cap into the PI-controlled dispatch pacer. The pacer's job was to decide how often to dispatch new work items to the GPU, balancing two competing pressures: keeping the GPU busy (high throughput) versus avoiding memory exhaustion (backpressure). The pacer used a PI (proportional-integral) controller that tracked a smoothed error signal — the difference between a target queue depth and the current EMA (exponential moving average) of waiting work items.

The critical input to this controller was the GPU consumption rate: how fast the GPU was processing items. The pacer used this as a feed-forward term to set the baseline dispatch interval. In synthcap1, the GPU rate was inferred by measuring the time between GPU completions — the inter-completion interval. The assumption was that the interval between completions reflected how long the GPU took to process each item.

This assumption was wrong.

The Flawed Assumption and Its Discovery

When the user deployed synthcap1 and examined the logs, they noticed something alarming: the initial GPU rate calibration was measuring approximately 47 seconds per item, when the actual GPU processing time was closer to 1 second. The discrepancy arose because the inter-completion interval included the pipeline fill time — the time needed to dispatch enough work to saturate the GPU pipeline. During initial warmup, items were dispatched slowly (because the pacer hadn't converged yet), so the GPU spent most of its time waiting for work rather than processing it. The completion interval reflected this idle time, contaminating the EMA and causing it to drag downward painfully slowly.

The assistant's first attempted fix was to skip the first GPU completion and only update the GPU rate when the waiting queue was non-empty (waiting > 0). The reasoning was: if the queue is empty, the GPU is waiting for work, so the completion interval includes idle time. Only when items are queued does the completion interval reflect true GPU processing.

But the user identified a deeper flaw: with two interleaved GPU workers, both workers could be actively processing while the dispatch queue was empty. The queue depth reflected how many items were waiting to be dispatched, not how many were currently being processed. With two workers, the pipeline could have two items in flight with zero in the queue — the GPU was fully busy but waiting == 0. The heuristic was fundamentally broken for the multi-worker case.

The Pivot to Direct Measurement

At this point, the assistant made a critical architectural decision: instead of inferring GPU processing time from observable events (completions, queue depth), measure it directly from the source. The GPU workers already captured gpu_result.gpu_duration — the actual wall-clock time the GPU spent computing. The assistant's plan, articulated in message <msg id=3544>, was:

I'll add a gpu_processing_total_ns: Arc<AtomicU64> that workers fetch_add their actual processing duration into. Then the pacer computes: - avg_gpu_processing_s = delta_ns / delta_completions - effective_interval = avg_gpu_processing_s / num_gpu_workers

This approach is immune to both idle time and pipeline fill contamination. The GPU workers report only the time they spent actually computing. The pacer divides by the number of workers to get the effective dispatch interval — the rate at which the system as a whole can consume work. With two workers each taking 1 second, the effective interval is 0.5 seconds: the dispatcher should send a new item every 500ms to keep both workers saturated.

The Implementation

The implementation required changes across multiple locations in the engine:

Adding the atomic: A new Arc<AtomicU64> called gpu_processing_total_ns was created alongside the existing gpu_completion_count and gpu_done_notify.
Wiring into workers: The atomic was cloned for each GPU worker thread at spawn time (line ~2888 in engine.rs), then cloned again inside the finalizer tokio task.
Accumulating in the finalizer: After the GPU result was obtained but before the completion count was incremented, the assistant inserted a fetch_add of the GPU duration in nanoseconds. This ensured the measurement was recorded atomically and visible to the pacer.
Updating the pacer state: The DispatchPacer struct gained new fields (ema_gpu_processing_s, gpu_proc_known, prev_gpu_proc_ns) to track the smoothed processing time. The update() method was rewritten to compute the delta in processing nanoseconds between calls, divide by the delta in completions to get per-item processing time, and feed that into an EMA.
Computing the effective interval: The interval() method was updated to divide the EMA processing time by the number of GPU workers, producing a dispatch interval that naturally accounts for parallelism.
Passing num_gpu_workers: The pacer needed to know how many workers existed. The assistant extracted num_workers from the existing variable at the creation site and passed it into DispatchPacer::new().

The Significance of "Clean Compile"

The phrase "Clean compile — only pre-existing warnings" carries substantial weight. The edits touched shared mutable state across thread boundaries, introduced new atomic operations, and modified the pacer's core control loop. A single mistake — forgetting to clone the atomic into a worker, misordering the fetch_add relative to the completion count increment, or failing to handle the initial calibration phase — could cause data races, incorrect rate estimates, or outright crashes. The fact that the code compiled cleanly (with only the pre-existing warnings about JobTracker visibility, which were unrelated) validated that the structural changes were sound.

Moreover, the build completed in approximately 114 seconds using Docker BuildKit with a no-cache build. This is a full release-profile compilation of the entire CuZK workspace, including cuzk-core, cuzk-server, and cuzk-daemon. The multi-stage Dockerfile (Dockerfile.cuzk-rebuild) first compiles in a builder stage, then copies the resulting binary into a minimal runtime image. The COPY --from=builder step taking only 0.1 seconds confirms that the binary was successfully produced.

What This Message Creates

This message produces the cuzk-rebuild:synthcap2 Docker image, which contains the cuzk-daemon binary with the direct GPU measurement pacer. In the subsequent messages, we see this binary extracted from the container, copied to the remote server as /data/cuzk-synthcap2, and deployed to replace the running synthcap1 instance. The deployment flow — docker create, docker cp, docker rm, scp, and ssh — is a well-practiced pipeline for rapid iteration on remote infrastructure.

The message also creates knowledge: the confirmation that direct measurement is implementable without breaking the existing architecture. The atomic accumulation pattern is clean, the pacer integration is coherent, and the approach generalizes to any number of GPU workers. This pattern — having workers self-report their processing time rather than having an external observer infer it — is a durable architectural improvement that would persist through future iterations.

Context and Input Knowledge

To fully understand this message, one must know:

The PI-controlled dispatch pacer: A feedback control system that regulates how frequently work items are dispatched to the GPU. It uses a proportional-integral controller to maintain a target queue depth, with a feed-forward term based on GPU processing rate.
The two-GPU-worker architecture: The CuZK engine spawns multiple GPU worker threads (typically 2 per device) that pull work from a shared dispatch queue. Workers run finalizers asynchronously via tokio tasks.
The pipeline fill problem: When a pipeline first starts, items must be dispatched sequentially until the GPU is saturated. During this fill phase, completion intervals are dominated by dispatch latency, not GPU compute time.
The interleaved worker problem: With multiple workers, queue depth is not a reliable indicator of GPU busyness because workers can be actively processing while the dispatch queue is empty.
The existing atomic infrastructure: The engine already used gpu_completion_count (an Arc<AtomicU64>) for tracking completions and gpu_done_notify (a Condvar-like notification primitive) for waking the dispatcher.

Assumptions Made

The assistant made several assumptions in this message:

That the atomic accumulation is safe: The fetch_add on gpu_processing_total_ns is called from multiple tokio tasks (one per GPU worker finalizer). This is safe because AtomicU64 guarantees atomicity, and the ordering (Relaxed or Release) is sufficient for the pacer's sampling-based approach — the pacer reads the value periodically and computes deltas, so it doesn't need sequential consistency.
That the GPU duration is available in all paths: The assistant handled both the split-prove path (where gpu_result.gpu_duration is obtained from the finalizer) and the synchronous fallback path (used when CUZK_DISABLE_SPLIT_PROVE=1). The assumption was that both paths produce a Duration value.
That num_gpu_workers is constant: The pacer receives the worker count at construction time and assumes it doesn't change. This is valid because GPU workers are spawned once during engine initialization.
That the EMA on processing time converges faster than the old inter-completion EMA: The assistant assumed that direct measurement would eliminate the pipeline fill contamination, allowing the pacer to converge to the true GPU rate within a few completions rather than waiting for the pipeline to saturate.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption was embedded in the previous approach (synthcap1), not in this message itself. However, this message carries forward one subtle assumption that would later prove problematic: that measuring GPU processing time alone is sufficient for the pacer to maintain stable throughput. In the subsequent iteration (still within this chunk), the user would report that the system was "still collapsing" — the synthesis throughput cap introduced in synthcap1 was creating a vicious cycle where slow dispatch led to fewer concurrent syntheses, which reduced synthesis throughput, which tightened the cap further, which slowed dispatch even more. The direct GPU measurement was correct, but the synthesis cap was fundamentally destabilizing.

This message does not yet contain the fix for that collapse — that would come in the next iteration (synthcap3), where the assistant removed the synthesis throughput cap entirely and added re-bootstrap detection. But the direct measurement foundation laid here was essential: without accurate GPU rate information, the subsequent fixes (re-bootstrap, slow bootstrap timing, PI tuning) would have been operating on faulty data.

Conclusion

Message <msg id=3576> is a deployment milestone that represents the transition from indirect inference to direct measurement in the CuZK GPU dispatch pacer. The clean compile confirmed that a complex set of cross-cutting changes — new atomics, modified worker paths, restructured pacer state — were structurally sound. The Docker build produced the synthcap2 binary that would be deployed to production, carrying a fundamentally better approach to rate estimation: instead of guessing whether the GPU was busy from queue depth, the pacer now knew exactly how long each item took to process, directly from the workers that did the work.

This message exemplifies a pattern common in systems engineering: when a heuristic fails because its assumptions don't match reality, the correct response is not to refine the heuristic but to measure directly. The 47-second phantom GPU time was not a calibration issue — it was a measurement methodology issue. By sourcing the measurement from the GPU workers themselves, the assistant eliminated an entire class of inference errors and built a more robust foundation for the control system that would follow.