The Synthesis Throughput Cap: A Turning Point in GPU Pipeline Control

Introduction

In the course of building a high-performance GPU proving pipeline for the CuZK zero-knowledge proving engine, a single message can encapsulate an entire engineering epiphany. Message <msg id=3492> is deceptively brief — just four bullet points and a file read — but it represents the culmination of an extended reasoning process that traversed control theory, congestion control algorithms, queuing theory, and systems engineering. This message marks the precise moment when a complex, multi-faceted problem was distilled into a concrete implementation plan, and the assistant committed to executing it.

The message reads in full:

Now I have all the sites. Let me make all the changes:

1. DispatchPacer struct — add synth rate fields 2. Shared state — add synth_completion_count 3. Synthesis workers — clone + increment 4. Dispatcher — pass synth count to update()

[read] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs

To an outside observer, these four items might appear as routine code modifications. But to understand why this message was written — and why it matters — we must reconstruct the engineering context, the reasoning trail that led here, and the assumptions embedded in each of those four bullet points.

The Problem: When the Pacer Overcorrects

The CuZK proving engine operates a pipelined architecture where CPU-bound synthesis workers produce circuit partitions that are consumed by a GPU for proving. Between these two stages sits a dispatch scheduler — a PI (proportional-integral) controller called the DispatchPacer that regulates how frequently new synthesis work is dispatched. The pacer's goal is to maintain a target number of synthesized partitions waiting in the GPU queue, ensuring the GPU never starves while avoiding excessive memory pressure.

The PI pacer had been working well in steady state, but the user identified a critical edge case in <msg id=3483>: when synthesis becomes compute-constrained (i.e., the CPU cannot keep up with the GPU), the pacer's PI controller drives the dispatch interval below the GPU consumption rate in an attempt to fill the queue. This floods the system with concurrent synthesis jobs, which then contend for CPU resources, slowing each individual synthesis and degrading overall throughput. The user observed that running 16 concurrent syntheses produced better throughput than 22 — a classic symptom of CPU contention collapse.

This is a fundamental limitation of a queue-depth-only controller: it cannot distinguish between "the queue is empty because the GPU is fast" and "the queue is empty because synthesis is stalled." Both conditions produce the same error signal (queue depth below target), but they demand opposite responses. The PI controller, lacking this distinction, pushes harder when synthesis is already struggling, creating a vicious cycle.

The Reasoning Journey

Message <msg id=3484> reveals the assistant's extensive reasoning process. The assistant explored multiple frameworks for solving this problem:

TCP congestion control analogy: The assistant recognized that the problem maps onto classic congestion control — the optimal number of in-flight requests equals the bandwidth-delay product (BDP). But it quickly identified a critical flaw: if synthesis duration increases due to contention, the BDP formula would recommend more in-flight requests, creating a positive feedback loop that makes contention worse. This is unstable.
Little's law: The assistant considered using Little's law (throughput = in-flight / latency) to find the concurrency level that maximizes throughput. As in-flight increases, throughput climbs linearly during spare capacity, plateaus at saturation, then drops due to contention overhead. The optimal operating point is the "knee" of this curve.
Direct concurrency limiting: A simpler approach — just cap the number of in-flight syntheses. But the user explicitly rejected fixed thresholds in <msg id=3483>, demanding an adaptive mechanism that works across different systems with different bottlenecks.
Synthesis throughput cap: The solution the assistant ultimately chose. Instead of managing queue depth or concurrency directly, cap the dispatch rate to never exceed the measured synthesis completion rate. This creates an implicit negative feedback loop: if synthesis slows due to contention, the measured rate drops, automatically reducing the dispatch ceiling and relieving CPU pressure. The key insight is that the synthesis completion rate is a direct measurement of the bottleneck's capacity. By capping dispatch to this rate (with a small headroom factor for probing), the controller naturally finds the sustainable throughput without needing to model contention explicitly.

The Four Bullet Points: What They Really Mean

Each of the four items in <msg id=3492> encodes a significant engineering decision:

1. DispatchPacer struct — add synth rate fields

This means adding exponential moving average (EMA) state for the synthesis inter-completion interval, following the same pattern already used for GPU rate tracking. The EMA provides smoothing against measurement noise while remaining adaptive enough to track changes in system behavior (e.g., proof type transitions that trigger pinned buffer reallocations). The assistant had previously debated the EMA alpha value — too fast causes jitter, too slow fails to adapt to proof type changes. The choice of alpha represents a tradeoff between stability and responsiveness.

2. Shared state — add `synth_completion_count`

This is an AtomicU64 counter shared between all synthesis workers and the dispatcher. Its placement is critical: it must be incremented after the synthesized partition is pushed to the GPU work queue, not before. This ensures the counter reflects completed synthesis work that is actually available for GPU consumption, not work that is still in the pipeline. The atomic ensures visibility across threads without requiring a mutex.

3. Synthesis workers — clone + increment

Each synthesis worker needs a clone of the atomic counter reference. The increment happens at line 1633 of engine.rs, immediately after the gpu_work_queue.push() call. This is the measurement point — the exact moment when synthesized work transitions from CPU-bound to GPU-bound. The assistant had to locate this site through grep searches in <msg id=3489> and <msg id=3490>, confirming the exact line.

4. Dispatcher — pass synth count to update()

The dispatcher's update() method, called on each GPU completion event, needs access to the current synthesis completion count to compute the synthesis rate. This is where the feedback loop closes: the pacer reads the counter, computes the EMA of the inter-completion interval, derives the synthesis rate, and caps the PI controller's output. If the cap is active, the integral term is frozen (anti-windup) to prevent accumulated error from causing a dispatch burst when the bottleneck clears.

Assumptions Embedded in the Design

The implementation plan rests on several key assumptions:

Synthesis throughput is the correct signal to cap against. The assistant considered alternatives (in-flight count, synthesis duration, direct concurrency limits) and concluded that the completion rate is the most direct measure of bottleneck capacity. This assumes that the synthesis completion rate is a stable, measurable signal that correlates with actual system throughput.
EMA smoothing will handle transients. Startup pinned allocations, proof type transitions, and other transient events will skew initial measurements. The assistant planned a warmup threshold (5-10 completions before the cap activates) to address this, but the EMA itself must be tuned to adapt quickly enough to real changes without overreacting to noise.
Anti-windup prevents integral buildup. When the cap is active, the PI controller's integral term continues to accumulate error, which would cause a burst of dispatches once the cap lifts. Freezing the integral during capping prevents this, but it also means the controller stops correcting for queue depth deficits during capped periods — queue depth may remain below target until synthesis capacity increases.
The headroom factor provides enough probing without causing drift. A headroom of 1.1 (10% above measured synthesis rate) allows the system to probe for additional capacity without growing in-flight buffers unboundedly. But as the assistant noted, even 10% headroom causes slow drift in in-flight count over time, bounded only by the memory budget hard cap.

What This Message Creates

Message <msg id=3492> is the pivot point between investigation and implementation. Before this message, the assistant had:

Identified the edge case (synthesis compute-constrained)
Explored multiple solution frameworks (TCP congestion control, Little's law, concurrency limiting)
Selected the synthesis throughput cap approach
Located the code sites (grep searches in <msg id=3489>)
Read the relevant code sections (<msg id=3490>, <msg id=3491>) After this message, the assistant executes the plan: <msg id=3493> applies the edit to engine.rs, replacing the entire DispatchPacer with the version including synthesis rate cap and anti-windup. The message thus creates the intent and structure for the implementation, serving as a bridge between analysis and action.

Conclusion

Message <msg id=3492> is a study in concise engineering communication. Four bullet points and a file read — yet each bullet condenses hours of reasoning about control theory, systems architecture, and performance optimization. The message demonstrates that the most impactful engineering communications are often the shortest, because they arrive at the moment when the problem is fully understood and the solution is clear. The assistant's journey through TCP congestion control, Little's law, BDP formulas, and anti-windup techniques all converges on this single moment of commitment to implementation. It is the calm before the edit — the last breath before diving into code.