The Synthesis Throughput Cap: Closing the Loop on GPU Pipeline Control

Introduction

In the iterative refinement of a complex GPU pipeline scheduler, the most consequential changes often arrive not as grand architectural rewrites but as targeted additions that close a single feedback loop. This article examines a pivotal sequence within an opencode coding session — spanning messages 3483 through 3494 — in which a PI-controlled dispatch pacer was extended with a synthesis throughput cap to solve a subtle but critical edge case. The journey from problem identification to implementation traversed control theory, TCP congestion control principles, Little's law, and the practical realities of concurrent systems programming in Rust. What emerged was a dispatch scheduler that could automatically adapt to whichever stage of the pipeline — GPU or CPU — was the bottleneck, without requiring fixed thresholds or system-specific tuning.

The Problem: When the Pacer Fights Itself

The story begins with a success. The assistant had deployed a PI-controlled dispatch pacer (DispatchPacer) that used an exponential moving average (EMA) of the GPU inter-completion interval as a feed-forward rate, with a PI correction on the smoothed GPU queue depth error [1]. This replaced an earlier burst-based P-controller that had caused pinned memory pool exhaustion and GPU driver serialization. The PI pacer worked "wonderfully compared to previous state," as the user noted in [msg 3483].

But the user immediately identified a critical edge case. On systems where the CPU-bound synthesis stage was the bottleneck — where the CPU could not produce synthesized partitions fast enough to keep the GPU fully fed — the pacer's behavior became pathological. The PI controller, seeing an empty GPU queue, drove the dispatch interval below the GPU consumption rate in an attempt to fill the queue. This flooded the system with concurrent synthesis jobs, which then contended for CPU resources. The user observed that running 22 concurrent syntheses was worse than running 16 — the extra jobs starved each other for CPU time, reducing individual synthesis throughput and degrading overall system performance [57].

This is a classic control systems failure mode: the controller was optimizing for the wrong metric. It was trying to maintain a target queue depth of synthesized partitions waiting for the GPU, but when synthesis was the bottleneck, chasing queue depth only made the problem worse. The pacer could not distinguish between "the queue is empty because the GPU is fast" and "the queue is empty because synthesis is stalled." Both conditions produced the same error signal — queue depth below target — but demanded opposite responses.

The user's diagnosis was precise and profound: "the control should optimize for overall sustained system throughput" [57]. This redefined the objective function. Queue depth was a proxy, not the true goal. The true goal was proofs completed per unit time, and that depended on both GPU and CPU resources interacting in complex ways.

The Reasoning Journey: From Queue Depth to Throughput

The assistant's response in [msg 3484] was arguably the most extended reasoning chain in the entire session — a deep exploration of how to make the pacer bottleneck-aware [58]. The assistant worked through multiple frameworks:

TCP congestion control analogy. The assistant recognized that the problem maps onto classic congestion control. The optimal number of in-flight requests equals the bandwidth-delay product (BDP). But the assistant quickly identified a critical flaw: if synthesis duration increases due to contention, the BDP formula would recommend more in-flight requests, creating a positive feedback loop that makes contention worse. This is unstable. The assistant caught this before it reached code — a moment of genuine self-correction.

Little's law. The assistant considered using Little's law (throughput = in-flight / latency) to find the concurrency level that maximizes throughput. As in-flight increases, throughput climbs linearly during spare capacity, plateaus at saturation, then drops due to contention overhead. The optimal operating point is the "knee" of this curve.

Direct concurrency limiting. A simpler approach — just cap the number of in-flight syntheses. But the user explicitly rejected fixed thresholds, demanding an adaptive mechanism that works across different systems with different bottlenecks.

The solution the assistant converged on was the synthesis throughput cap: measure the rate at which synthesis completes, and clamp the dispatch rate to not exceed that rate (with a small headroom factor). This creates an implicit negative feedback loop: if synthesis slows due to contention, the measured rate drops, automatically reducing the dispatch ceiling and relieving CPU pressure. The key insight is that the synthesis completion rate is a direct measurement of the bottleneck's capacity [58].

The User's Refinements: Adaptivity and Startup

Before the assistant could implement this design, the user added two critical constraints that shaped the final implementation.

In [msg 3485], the user noted that different proof types trigger pinned page re-allocations, so the system "needs to be adaptive" [59]. The CuZK engine handles multiple proof types — SnapDeals, PoRep, WinningPoSt, WindowPoSt — each requiring different pinned buffer sizes. A SnapDeals partition might use 2.4 GiB per buffer, while PoRep uses 4.17 GiB. When the proof type changes, the pinned memory pool must re-allocate buffers via cudaHostAlloc, a synchronous GPU driver operation that stalls the pipeline. The pacer's synthesis rate measurement would be temporarily skewed during these transitions. The user's message was a warning: the pacer must not assume a single steady state.

In [msg 3486], the user added a second observation: "Also we allocate pinned pages on startup a lot which skews initial measurements" [60]. During the first job after deployment, every pinned buffer is a fresh allocation via cudaHostAlloc. These allocations are orders of magnitude slower than reusing buffers from the pool during steady-state operation. If the pacer's synthesis rate measurement includes these startup samples, the EMA would be dragged downward, producing an artificially low estimate of synthesis throughput. The cap would then constrain the dispatch rate to match this pessimistic estimate, starving the GPU of work long after the system had actually reached steady state.

The assistant's response in [msg 3487] integrated both constraints into the design [61]. For startup skew, the assistant planned a warmup threshold — "hold off applying the synth rate cap until we've seen at least 5-10 completions" — to prevent early measurements from distorting steady-state control. For proof type transitions, the assistant relied on the EMA's alpha parameter (0.15) to converge within roughly 10 samples after a regime change. This was a deliberate trade-off: fast enough to adapt within a few completions, but slow enough to filter out noise from individual allocations.

Finding the Measurement Point

With the design settled, the assistant needed to find the exact code locations where the measurement infrastructure would be wired. This required a series of grep searches and file reads that, while mundane on the surface, represented the critical transition from abstract reasoning to concrete implementation.

The first grep, in [msg 3488], searched for gpu_work_queue.*push" — expecting a string literal argument — and found nothing [62]. The trailing double quote was a fossil of an incorrect assumption about how the push call was structured. The assistant corrected this in [msg 3489], dropping the trailing quote, and found two matches at lines 1633 and 2704 of engine.rs [63]:

gpu_work_queue.push(p_job_seq, p_idx, job);
gpu_work_queue.push(0, 0, job);

Line 1633 was identified as the primary synthesis handoff — the exact point where a synthesis worker finishes its work and pushes the result to the GPU queue. This is where the atomic counter must be incremented.

But knowing where to increment was not enough. The assistant also needed to know how the counter variable gets into the worker's scope. In [msg 3491], the assistant read the synthesis worker spawn loop [65]:

// Synthesis worker pool: receive dispatched work, synthesize, push to GPU queue
for sw_id in 0..synth_worker_count {
    let synth_dispatch_rx = synth_dispatch_rx.clone();
    let gpu_work_queue = gpu_work_queue.clone();
    ...

This revealed the cloning pattern — each worker clones the shared state from Arc-wrapped references. The assistant would follow this exact pattern for the new atomic counter.

The Implementation: Four Bullet Points

In [msg 3492], the assistant distilled the implementation into four bullet points [66]:

1. DispatchPacer struct — add synth rate fields (EMA state, warmup counter, cap flag)
2. Shared state — add synth_completion_count: Arc<AtomicU64>
3. Synthesis workers — clone the counter and increment after gpu_work_queue.push()
4. Dispatcher — pass synth count to pacer.update() on each GPU completion event Each bullet encoded significant engineering decisions. The EMA state would follow the same pattern already used for GPU rate tracking, ensuring consistency. The atomic counter would be incremented after the push, ensuring the measurement reflected completed work actually available for GPU consumption. The anti-windup logic would freeze the PI integral term when the cap was active, preventing accumulated error from causing a dispatch burst when the bottleneck cleared [67]. The edit itself, applied in [msg 3493], replaced the entire DispatchPacer with a version that included the synthesis rate cap, anti-windup, and warmup gating. The update() method now accepted a synth_count parameter and computed an EMA of the synthesis inter-completion interval. The interval() method clamped the PI-computed dispatch interval so that the dispatch rate never exceeded the measured synthesis rate times a headroom factor (initially 1.1). When the cap was active, the integral accumulation was skipped — a textbook anti-windup implementation.

The Silent Signal

The user's response to this edit, in [msg 3494], was empty — nothing more than blank XML tags [68]. But this silence spoke volumes. It signaled trust in the assistant's technical judgment, shared understanding of the remaining wiring steps, and forward momentum. The edit was complex — it touched the core control loop of the system — but the reasoning had been thoroughly discussed, the approach agreed upon. No further clarification was needed.

This silence is only possible when both parties share a deep mental model of the system. The user knew exactly what the remaining steps were: adding the atomic counter, wiring it into synthesis workers, updating all pacer.update() calls in the dispatcher loop, and updating the periodic status log. The assistant knew the user knew. The empty message was an implicit "proceed."

The Broader Significance

The synthesis throughput cap represents the culmination of five iterations of dispatch control, each building on lessons from the previous. The evolution — from semaphore to burst controller to damped controller to PI pacer to capped PI pacer — mirrors the development of real-world control systems from simple feedback loops to sophisticated multi-variable controllers that respect the actual capacity of the system.

The design is elegant because it creates a self-regulating loop without requiring explicit mode switching. The pacer automatically defers to whichever stage of the pipeline is the bottleneck. When the GPU is the bottleneck, the PI controller's queue-depth regulation dominates, keeping the GPU fed. When synthesis is the bottleneck, the throughput cap engages, preventing over-dispatch and CPU contention. The transition between these regimes is continuous and automatic, governed by the measured rates rather than hardcoded thresholds.

This pattern — measure the bottleneck's throughput, cap the dispatch rate, and add anti-windup — is broadly applicable beyond GPU proving pipelines. It arises in any multi-stage system where a controller optimizes for a local metric but the global optimum depends on a different metric. Web servers, data pipelines, manufacturing systems — all face the same fundamental challenge of balancing upstream and downstream capacity without explicit coordination.

Conclusion

The sequence from [msg 3483] to [msg 3494] captures a complete engineering cycle: problem identification, deep reasoning, design refinement through user feedback, code navigation, implementation, and silent acceptance. The synthesis throughput cap with anti-windup transformed the DispatchPacer from a single-variable controller (queue depth) into a multi-variable one that respects the actual capacity of the system. It is a small edit with a large intellectual footprint — a testament to the power of iterative refinement, cross-disciplinary thinking, and the careful application of control theory to real-world systems engineering.## References

[1] "The Pacer Emerges: Designing a PI-Controlled Dispatch Scheduler for GPU Pipeline Stabilization" — [chunk 25.1], msg 3427

[57] "When the Pacer Meets the Wall: A Systems Critique of GPU Dispatch Control" — [chunk 25.1], msg 3483

[58] "The Synthesis Throughput Cap: Taming a GPU Pipeline Pacer with Control Theory and Congestion Control" — [chunk 25.1], msg 3484

[59] "The Adaptability Requirement: How a Single User Remark Reshaped the GPU Pipeline Control Design" — [chunk 25.1], msg 3485

[60] "The Startup Skew Problem: Why Initial Measurements Mislead GPU Pipeline Pacers" — [chunk 25.1], msg 3486

[61] "Adapting to Reality: How a GPU Dispatch Pacer Learned to Handle Proof Type Transitions and Startup Transients" — [chunk 25.1], msg 3487

[62] "The Missing Quote: How a Single Character Derailed a Complex Control System Implementation" — [chunk 25.1], msg 3488

[63] "The Critical Grep: Finding the Synthesis Handoff Point in the CuZK GPU Pipeline" — [chunk 25.1], msg 3489

[65] "The Wiring Moment: How a Single Read Tool Call Anchored a Control System" — [chunk 25.1], msg 3491

[66] "The Synthesis Throughput Cap: A Turning Point in GPU Pipeline Control" — [chunk 25.1], msg 3492

[67] "The Synthesis Throughput Cap: Closing the Control Loop on GPU Pipeline Scheduling" — [chunk 25.1], msg 3493

[68] "The Silent Signal: How an Empty Message Advanced a GPU Pipeline Control System" — [chunk 25.1], msg 3494