The Synthesis Throughput Cap: Closing the Control Loop on GPU Pipeline Scheduling

Subject Message (msg 3493): [assistant] Now I'll replace the entire DispatchPacer with the version that includes synthesis rate cap + anti-windup: [edit] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

At first glance, message 3493 appears to be little more than a status update — a brief announcement that an edit has been applied to a file. But in the context of the broader conversation, this message represents the culmination of an intense reasoning session, the moment when a complex theoretical insight about control systems was translated into concrete code. The message is the delivery mechanism for a solution to one of the most subtle problems in GPU-accelerated proving: how to prevent a feedback controller from destroying its own throughput by flooding the CPU with more work than it can handle.

The Problem: When the Pacer Eats Its Own Tail

To understand why this message was written, we need to trace the problem back to its roots. The team had been iterating on a GPU pipeline scheduling system for the CuZK proving engine. The pipeline had three stages: synthesis (CPU-bound work that generates circuit constraints), dispatch (sending synthesized work to the GPU), and GPU proving (the actual computation on the GPU). The challenge was to keep the GPU fed with work without overwhelming the CPU with too many concurrent synthesis jobs.

The assistant had previously implemented a PI-controlled dispatch pacer (DispatchPacer) that used an exponential moving average (EMA) of the GPU inter-completion interval as a feed-forward rate, with a PI correction on the smoothed GPU queue depth error. This worked well in steady state, but the user identified a critical edge case in message 3483: when synthesis couldn't keep up with the GPU (i.e., the system was synthesis-constrained), the pacer would drive the dispatch interval below the GPU rate to try to fill the queue. This flooded the system with concurrent synthesis jobs, causing CPU contention that actually degraded overall throughput. Running 22 concurrent syntheses was worse than running 16, because the extra jobs starved each other for CPU time.

This is a classic control systems problem: the controller was optimizing for the wrong metric. It was trying to maintain a target queue depth of synthesized partitions waiting for the GPU, but when synthesis was the bottleneck, chasing queue depth only made things worse. The pacer needed to be constrained by the actual throughput of the synthesis stage.

The Reasoning Journey: From Queue Depth to Throughput

The assistant's reasoning in message 3484 reveals a remarkable depth of analysis. It began by framing the problem in terms of optimal concurrency: there exists a sweet spot where below it the GPU starves and above it syntheses contend with each other. The assistant explored multiple approaches, including TCP-style congestion control, exponential backoff based on synthesis duration, and Little's law (throughput = in-flight / latency).

A key insight emerged: queue depth alone is misleading. It can stay low even when the system is already saturated with synthesis work. The assistant recognized that it needed a "synthesis pressure" signal — when in-flight partitions are piling up and synthesis duration is climbing, the pacer should resist the urge to dispatch more even if the queue looks empty.

The assistant then went through a fascinating self-correction. It initially considered using the bandwidth-delay product (BDP) formula: max_in_flight = gpu_rate * ema_synth_duration * 1.2. But it realized this creates a positive feedback loop: if synthesis duration increases due to contention, the BDP formula tells us to push more in-flight requests, making the contention worse. This is unstable. The assistant correctly identified that the right approach is to measure synthesis throughput directly (completions per second) and use that as a ceiling on dispatch rate.

The Design: Synthesis Throughput Cap with Anti-Windup

The solution that the assistant implemented in message 3493 consists of several interconnected pieces:

  1. Synthesis completion tracking: An atomic counter (synth_completion_count) that synthesis workers increment after pushing their results to the GPU work queue. This provides a direct measurement of how fast synthesis can actually produce work.
  2. EMA of synthesis inter-completion interval: The pacer computes an exponential moving average of the time between synthesis completions, similar to how it already tracks GPU completions. This gives a smoothed estimate of the sustainable synthesis rate.
  3. Rate cap with headroom: The dispatch rate is clamped to not exceed the measured synthesis completion rate times a small headroom factor (around 1.1–1.2). This prevents the pacer from dispatching faster than synthesis can handle, while still allowing a small amount of overshoot to probe for additional capacity.
  4. Anti-windup: When the rate cap is active, the PI controller's integral term is frozen. This prevents the integral from accumulating error during the capped period, which would otherwise cause a burst of dispatches when the cap is lifted.
  5. Warmup threshold: The cap is gated by a minimum number of completion samples (5–10) to avoid being overly conservative during startup, when pinned buffer allocations skew measurements. The result is a self-regulating loop: if synthesis slows due to contention, the measured rate drops, automatically reducing the dispatch ceiling and relieving CPU pressure. If synthesis speeds up (e.g., after a proof type change), the measured rate rises and the cap loosens.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message produces a modified DispatchPacer that incorporates synthesis rate awareness. The key intellectual output is the recognition that in a multi-stage pipeline, a controller targeting queue depth can destabilize the system when the upstream stage (synthesis) is the bottleneck. The solution — capping dispatch rate to measured synthesis throughput with anti-windup — is a practical application of cascade control principles, where the outer loop (queue depth PI) is constrained by the inner loop's capacity measurement.

Assumptions and Potential Pitfalls

The design makes several assumptions worth examining. First, it assumes that synthesis completion rate is a reliable proxy for synthesis capacity. If synthesis completions are bursty or the EMA is too slow to adapt, the cap could be too conservative or too loose. Second, the headroom factor (1.1–1.2) is a heuristic — too high and it defeats the purpose, too low and it starves the GPU. Third, the warmup threshold assumes that the first few completions are unrepresentative, but if the system processes many short jobs, the warmup period could be a significant fraction of runtime.

The assistant also implicitly assumes that the synthesis bottleneck is CPU contention rather than memory bandwidth or I/O. On systems with different bottleneck profiles, the cap might not provide the same benefit.

Conclusion

Message 3493 is a deceptively simple message that masks a deep reasoning process. It represents the moment when a theoretical insight about control system design — that optimizing for the wrong metric can destabilize a pipeline — was encoded into production code. The synthesis throughput cap with anti-windup transforms the DispatchPacer from a single-variable controller (queue depth) into a multi-variable one that respects the actual capacity of the system. It is a small edit with a large intellectual footprint, and it exemplifies the kind of careful, iterative refinement that characterizes robust systems engineering.