The Checkpoint That Preceded a Collapse: Wiring a Synthesis Throughput Cap Into a PI-Controlled Dispatch Pacer

Introduction

In the middle of a long and intricate optimization session for the CuZK GPU proving pipeline, there is a message that at first glance appears to be little more than a status update: a todowrite tool call marking four tasks as completed. The message, <msg id=3514>, reads in full:

[todowrite] {"todos":[{"content":"Add synth_completion_count Arc<AtomicU64> alongside gpu_completion_count","priority":"high","status":"completed"},{"content":"Wire synth_completion_count into synthesis workers (clone + increment)","priority":"high","status":"completed"},{"content":"Update all pacer.update() calls to pass synth_count","priority":"high","status":"completed"},{"content":"Update periodic status log to include synth rate in...

This is the moment when the assistant declared that the "synthesis throughput cap" — a mechanism designed to prevent the PI-controlled dispatch pacer from dispatching work faster than the CPU-based synthesis pipeline could produce it — was fully wired into the codebase. The code compiled cleanly. The binary was ready to build and deploy. And within hours of deployment, this carefully constructed mechanism would be revealed as fundamentally flawed, creating a vicious cycle that collapsed the pipeline and forced a complete redesign.

This article examines that message in depth: what it represents, the reasoning that led to it, the assumptions it rested on, and the lessons it contains about the difficulty of controlling deep, asynchronous pipelines with feedback loops.

The PI-Controlled Dispatch Pacer: A Brief History

To understand <msg id=3514>, one must first understand what it was completing. The CuZK proving engine operates as a two-stage pipeline: a CPU-based synthesis stage that prepares proof partitions, and a GPU-based proving stage that executes the computationally intensive zero-knowledge proof. Between them sits a dispatch mechanism that feeds synthesized partitions to the GPU workers.

Earlier in the session, the team had discovered that the GPU was severely underutilized — showing multi-second idle gaps between partition proves. The root cause was traced to slow host-to-device (H2D) PCIe transfers caused by unpinned host memory. A CUDA pinned memory pool was implemented, which reduced per-partition GPU time from thousands of milliseconds to under a second. But this fix revealed a new problem: the dispatch rate from synthesis to GPU was uncontrolled, causing bursty behavior that overwhelmed the pinned memory pool and created GPU timing jitter.

The solution was a PI-controlled dispatch pacer — a proportional-integral controller that regulates the dispatch interval to maintain a target number of synthesized partitions waiting in the GPU queue. The pacer uses an exponential moving average (EMA) of GPU processing time as a feed-forward term and adjusts the dispatch interval based on the error between the target queue depth and the actual waiting count.

This pacer worked well in steady state, but had a vulnerability: when the CPU-based synthesis pipeline couldn't keep up with the GPU, the PI controller would drive the dispatch interval below the GPU EMA, causing the system to dispatch work as fast as possible. This flooded the memory budget with concurrent syntheses that contended for CPU, making the synthesis throughput problem worse.

The Synthesis Throughput Cap: Design and Assumptions

The synthesis throughput cap was designed to address this vulnerability. The idea was straightforward: never dispatch work faster than the synthesis pipeline can produce it. The DispatchPacer struct was extended with new fields:

• ema_synth_interval_s — an exponential moving average of the time between synthesis completions
• synth_rate_known — a flag indicating whether enough synthesis completions have been observed for the rate estimate to be reliable
• synth_completions_seen — a counter to skip the first 8 completions (a warmup period to avoid startup skew from cudaHostAlloc calls)
• rate_capped — a flag indicating that the synthesis ceiling is currently active
• Anti-windup logic: when rate_capped is true, the PI controller's integral accumulation is frozen The interval() method was modified to apply a ceiling: if the PI-computed dispatch interval is less than ema_synth_interval / 1.1 (providing 10% headroom), the interval is clamped to the synthesis rate and rate_capped is set to true. These design decisions rested on several assumptions: 1. Synthesis throughput is a meaningful upper bound on dispatch rate. The assumption was that if synthesis produces N partitions per second, dispatching faster than N per second is pointless because the GPU queue will simply drain. This is true in steady state, but ignores the dynamics of a deep pipeline where synthesis and dispatch are coupled through memory pressure. 2. The EMA of synthesis completion intervals provides a stable, lag-free estimate of synthesis throughput. In practice, the EMA introduces lag, and during transient conditions (e.g., between batches of different proof types), the estimate can be significantly wrong. 3. The warmup period (skipping the first 8 completions) is sufficient to avoid startup skew. This assumed that after 8 completions, the pinned memory pool allocations would have stabilized and the synthesis rate would reflect steady-state behavior. 4. Anti-windup on the integral when capped prevents the PI controller from accumulating excessive error. This is a standard control technique, but it assumes that the capping condition is temporary and that the integral will naturally discharge when the cap is released.

The Wiring: What Was Actually Done

The tasks marked as completed in <msg id=3514> represent the data-flow wiring that connects the synthesis completion counter to the pacer's update() method. Specifically:

1. synth_completion_count: Arc<AtomicU64> was created at engine.rs:1399, alongside the existing gpu_completion_count. This atomic counter tracks the total number of synthesis completions across all synthesis workers.
2. The counter was cloned into the dispatcher task (as synth_count) and into each synthesis worker (as synth_done_count). This required careful scoping to ensure each worker had its own handle to the shared counter.
3. The counter was incremented in the synthesis workers immediately after gpu_work_queue.push(), ensuring that a synthesis completion is counted only after the partition has been successfully enqueued for GPU processing.
4. All four pacer.update() call sites in the dispatcher loop were updated to pass synth_count.load() as the third argument. These call sites are: - The top-of-loop update (the main pacing tick) - The bootstrap wait loop (during initial pipeline fill) - The bootstrap timer branch (periodic ticks during bootstrap) - The wait-for-work loop (when no work is available)
5. The periodic status log was updated to include ema_synth_ms, rate_capped, and synth_rate_known fields, providing visibility into the synthesis cap's behavior during runtime. The assistant verified the changes with cargo check, which passed with only pre-existing warnings. In <msg id=3518>, the assistant summarized: "All code changes compile cleanly."

The Collapse: What Went Wrong

The synthesis throughput cap was deployed as binary /data/cuzk-synthcap1. Almost immediately, the user reported that the system was collapsing. Analysis of the logs revealed three root problems, as documented in the chunk summary:

1. The synthesis throughput cap creates a self-reinforcing collapse loop. The mechanism works like this: slow dispatch (due to the cap) → fewer concurrent syntheses in flight → slower aggregate synthesis throughput → the EMA of synthesis interval increases → the cap tightens further → even slower dispatch. This is a positive feedback loop in a system that desperately needs negative feedback.
2. There is no re-bootstrap mechanism when the pipeline drains between batches. When a batch of proofs completes and a new batch starts, the pipeline can drain completely (zero partitions waiting in the GPU queue). The pacer has no way to detect this condition and re-enter the bootstrap mode that would rapidly fill the pipeline.
3. The 200ms bootstrap spacing floods the pinned memory pool. During bootstrap, the pacer dispatches every 200ms, which causes multiple concurrent cudaHostAlloc calls that serialize through the GPU driver and stall the GPU. This is the same problem the pinned memory pool was supposed to solve, but the bootstrap timing makes it worse. The assistant's response was to completely rewrite the DispatchPacer — removing the synthesis throughput cap entirely, adding re-bootstrap detection (re-entering bootstrap when ema_waiting < 1 with no active bootstrap), and slowing the bootstrap to 3 seconds for initial warmup or max(2s, gpu_eff) for re-bootstrap. The PI controller and memory budget backpressure were left to naturally balance GPU and synthesis rates, without an explicit synthesis cap.

The Deeper Lesson: Feedback Loops in Deep Pipelines

The synthesis throughput cap story is a cautionary tale about the difficulty of controlling systems with deep pipelines and coupled dynamics. The PI controller alone was sufficient for steady-state regulation, but the addition of a secondary feedback loop (the synthesis cap) created an unstable system.

The fundamental problem was that the synthesis throughput cap attempted to solve a symptom (excessive dispatch) by controlling a variable (synthesis rate) that was itself affected by the controller's output. When dispatch slowed down, fewer syntheses ran concurrently, which reduced synthesis throughput, which justified even slower dispatch. This is a classic example of a control system interacting with its own environment in unexpected ways.

The solution — removing the explicit cap and relying on the memory budget as a natural backpressure mechanism — is a return to a simpler, more robust design. The memory budget provides a hard constraint that doesn't create a feedback loop: when memory is exhausted, synthesis simply cannot proceed, regardless of what the pacer wants. This is a physical constraint rather than a control-theoretic one, and it is inherently stable.

Conclusion

Message <msg id=3514> appears to be a mundane status update, but it captures a pivotal moment in the development of a complex GPU proving pipeline. It marks the completion of a feature that, within hours, would be proven fundamentally flawed and removed. The message is a testament to the iterative nature of systems engineering: every deployment teaches something new, and every failed experiment provides the data needed for a better design.

The synthesis throughput cap was not a waste — it was a necessary experiment that revealed the true dynamics of the system. The re-bootstrap detection and slow bootstrap timing that replaced it were direct responses to the failure modes exposed by the cap. And the lesson about avoiding secondary feedback loops in coupled pipeline systems is one that will inform future designs.

In the end, the clean compile reported in <msg id=3517> and the completed tasks of <msg id=3514> were not the end of a feature — they were the beginning of a deeper understanding of the system they were trying to control.