The Synthesis Throughput Cap: Taming a GPU Pipeline Pacer with Control Theory and Congestion Control

Introduction

In the world of high-performance GPU proving pipelines, scheduling is everything. When you have a multi-stage system where CPU-bound synthesis work feeds into GPU-bound proving work, the scheduling discipline that governs how work flows from one stage to the next can make or break overall throughput. Too aggressive, and you flood the CPU with concurrent synthesis jobs that contend for cores, memory bandwidth, and cache, slowing each individual job down. Too conservative, and the GPU starves, wasting the most expensive resource in the system.

This article examines a single message from an opencode coding session — message index 3484 — in which an AI assistant grapples with precisely this problem. The assistant has just deployed a PI-controlled dispatch pacer (a proportional-integral controller) to regulate the flow of synthesis work into a GPU proving pipeline. The pacer works beautifully in the general case, but the user reports a critical edge case: when the CPU-bound synthesis stage becomes the bottleneck, the pacer's logic drives the dispatch rate up in an attempt to fill the GPU queue, which paradoxically makes the CPU contention worse, degrading overall throughput. The assistant's response — a detailed reasoning trace followed by an implementation plan — reveals a fascinating journey through control theory, congestion control principles, Little's law, and the practical engineering of self-adaptive systems.

This message is a masterclass in how to think about feedback control in complex systems. It shows the assistant reasoning through multiple approaches, discarding unstable ones, and converging on a solution that combines a synthesis throughput cap with anti-windup logic on the PI controller's integral term. The result is a dispatch pacer that adapts to whichever stage of the pipeline is the bottleneck — GPU or CPU — without requiring fixed thresholds or system-specific tuning.

Context: The GPU Pipeline Scheduling Saga

To understand message 3484, we need to understand the journey that led to it. The session (segment 25 of a larger conversation) is the culmination of an extended effort to optimize GPU utilization in a zero-knowledge proof system called CuZK. The pipeline has two main stages:

  1. Synthesis (CPU-bound): Constraint system synthesis, which generates the proving artifacts that need to be processed by the GPU.
  2. GPU proving (GPU-bound): The actual GPU computation that produces the final proof. Between these stages sits a pinned memory pool — a set of pre-allocated GPU-pinned buffers that avoid expensive cudaHostAlloc calls at runtime. The pinned pool was the subject of the previous several segments, where the team identified that cudaHostAlloc calls were causing GPU driver serialization, which introduced timing jitter that broke the dispatch pacing. The dispatch pacing itself had evolved through multiple iterations. It started as a simple semaphore-based throttle, then became a P-controller (proportional only) that dispatched work in bursts whenever a GPU slot freed up. The P-controller proved too aggressive, so a dampening factor was added. But the deep synthesis pipeline made the raw "waiting count" a noisy and delayed feedback signal, so the team moved to a full PI-controlled pacer with an exponential moving average (EMA) of the GPU inter-completion interval as a feed-forward rate. This PI pacer (deployed as pacer1 in the message just before 3484) was a significant improvement. It used a bootstrap phase to prime the pipeline, then switched to timer-based pacing at a PI-computed interval. The controller targeted a specific GPU queue depth (8 items waiting) and used the EMA of GPU completion intervals to estimate the GPU's consumption rate. The PI correction term adjusted the dispatch interval to maintain the target queue depth.

The Edge Case: When Synthesis Becomes the Bottleneck

The user's report (message 3483) describes the problem with precision:

"Generally works wonderfully compared to previous state. Edge-case: When synthesis doesn't keep up (we're synthesis compute constrained it seems on this specific system, though very close match to the GPU), the interval collapses to below gpu ema (which by itself must happen to get waiting partitions), which eventually causes us to max out memory budget with running synthesis. There is another consideration which is that running too many synthesis starves them against each other, so it might be better to run 16 vs 22."

This is a classic control systems problem. The PI controller has a single objective: maintain a target number of items waiting in the GPU queue. When the GPU is faster than synthesis, the queue drains, the error term grows, and the PI controller responds by reducing the dispatch interval — i.e., dispatching more frequently. But if the system is already synthesis-constrained, dispatching more frequently doesn't help: the new dispatch just adds another synthesis job to the pool of already-contending CPU tasks. Each individual synthesis takes longer because of contention, so the GPU queue stays empty, the error persists, and the PI controller drives the interval even lower.

This is a positive feedback loop in the wrong direction. The PI controller, designed to stabilize the GPU queue depth, actually destabilizes the system when the bottleneck shifts from GPU to CPU. The controller doesn't know about the CPU bottleneck — it only sees the GPU queue depth — so it keeps pushing harder, making the problem worse.

The user's key insight is that the control should optimize for "overall sustained system throughput" rather than just queue depth. The optimal number of concurrent synthesis jobs is not a fixed threshold (it varies across systems with different CPU/GPU balances), but there is a sweet spot where throughput is maximized. Below that spot, the GPU starves. Above it, CPU contention reduces individual synthesis throughput, and overall system throughput drops.

The Assistant's Reasoning: A Journey Through Control Theory

Message 3484 begins with an extensive "Agent Reasoning" block — the assistant's internal monologue as it processes the user's report and designs a solution. This reasoning trace is remarkable for its depth, its willingness to explore multiple approaches, and its self-correcting nature. Let us walk through it step by step.

Phase 1: Problem Decomposition

The assistant starts by enumerating the key points from the user's message:

  1. The PI pacer works well overall.
  2. Edge case: when synthesis can't keep up, the pacer drives the interval down, maxing out the memory budget.
  3. Running too many concurrent syntheses causes CPU contention (16 concurrent is better than 22).
  4. The solution must be adaptive, not a fixed threshold.
  5. The ideal is to optimize for maximum sustained throughput. This decomposition shows the assistant correctly identifying the multi-dimensional nature of the problem. It's not just about capping the dispatch rate — it's about finding the throughput-maximizing concurrency level, and doing so adaptively across different hardware configurations.

Phase 2: Exploring the Solution Space

The assistant then explores several approaches:

Approach 1: Concurrency limiter with TCP-style congestion control. The idea is to monitor synthesis completion rates and use a TCP-like algorithm to find the optimal in-flight synthesis count. This is appealing because TCP congestion control is a well-understood, proven approach to finding the optimal sending rate in a network where the bottleneck capacity is unknown and variable.

Approach 2: Exponential backoff based on synthesis duration. If synthesis times increase as concurrency rises, that signals contention, and the dispatcher should back off. This is simpler but potentially slower to converge.

Approach 3: Synthesis pressure signal. Track in-flight partitions and synthesis duration, then use Little's law to estimate sustainable throughput. This becomes the ceiling for the dispatch rate.

Approach 4: Bandwidth-delay product (BDP) formula. The optimal number of in-flight requests equals synthesis throughput multiplied by synthesis latency. This is the networking equivalent of the bandwidth-delay product, which determines how many packets can be in flight without causing congestion.

Phase 3: Self-Correction — The BDP Trap

The assistant's reasoning takes an interesting turn here. It initially gravitates toward the BDP approach, calculating max_in_flight = gpu_rate * ema_synth_duration * 1.2. But then it catches a critical flaw:

"But wait, there's a feedback problem here. If synthesis duration increases because of contention, the BDP formula tells us to push even more in-flight requests, which makes the contention worse. That's unstable."

This is a moment of genuine insight. The BDP formula assumes that latency is independent of load, but in a CPU-bound system, latency increases with load. Using a formula that increases the concurrency target when latency increases creates a positive feedback loop — exactly the opposite of what a controller should do. The assistant correctly identifies this as unstable and abandons the approach.

This self-correction is one of the most valuable aspects of the reasoning trace. It shows the assistant not just generating ideas, but actively stress-testing them against the dynamics of the system, catching potential failure modes before they reach code.

Phase 4: The Key Insight — Throughput, Not Queue Depth

The assistant pivots to a different framing:

"I need to flip my thinking and focus on what actually matters: synthesis throughput measured in completions per second. By Little's law, throughput equals in-flight divided by latency. As I increase in-flight, throughput climbs linearly at first when there's spare capacity, then plateaus once the CPU is saturated, and finally drops if I push past that point due to contention overhead."

This is the crucial conceptual shift. The PI controller was optimizing for queue depth (a proxy for GPU utilization), but the real objective should be system throughput. The assistant realizes that the relationship between in-flight count and throughput follows a classic shape: linear increase, plateau, then decline. The goal is to operate at the plateau — the maximum throughput point — without overshooting into the decline region.

Phase 5: The Synthesis Throughput Cap

The assistant converges on a solution: measure synthesis throughput directly and use it as a ceiling on the dispatch rate. The PI controller still drives the dispatch rate based on queue depth, but the output is capped so that the system never dispatches faster than synthesis can actually produce work.

This is elegant because it creates a natural, self-regulating loop. If synthesis slows due to contention, the measured throughput drops, which lowers the cap, which reduces dispatch pressure, which relieves contention. The EMA smoothing on the throughput measurement prevents overreaction to transient fluctuations.

But there's a subtlety: when the cap is active, the PI controller's integral term continues to accumulate error. The queue depth is below target (because we can't dispatch fast enough to fill it), so the integral term grows, storing up "demand" that will be released as a burst once the cap lifts. This is the classic problem of integrator windup in control systems.

Phase 6: Anti-Windup

The assistant identifies this problem and proposes a solution:

"The tricky part is that when the cap is active, the PI controller's integral term keeps accumulating error, which means once synthesis catches up again, we'll get a sudden burst of dispatches. I need anti-windup logic: freeze the integral accumulation whenever the actual dispatch rate is being capped."

Anti-windup is a well-known technique in control theory. When a controller's output is limited by a saturation constraint (in this case, the synthesis throughput cap), the integral term should be frozen or reset to prevent it from accumulating error that can't be acted upon. Without anti-windup, the controller would eventually "wind up" a large integral term that, when the saturation lifts, causes a massive overshoot — a burst of dispatches that could destabilize the system.

Phase 7: Implementation Details

The reasoning then moves to implementation specifics. The assistant plans to:

  1. Add an AtomicU64 counter for synthesis completions, incremented by synthesis workers after they push work to the GPU queue.
  2. In the DispatchPacer, compute an EMA of the synthesis inter-completion interval (following the same pattern as the GPU rate EMA).
  3. Derive the synthesis rate from the interval EMA.
  4. Clamp the dispatch interval so that the dispatch rate never exceeds the synthesis rate multiplied by a headroom factor (initially 1.2, later reconsidered to ~1.1).
  5. Freeze the PI integral term when the cap is active.
  6. Gate the cap with a warmup threshold to avoid being overly conservative during startup. The headroom factor is interesting. The assistant initially sets it to 1.2 (20% above measured synthesis rate) to probe for additional capacity and keep the pipeline pressurized. But then it reconsiders:
"Now I'm thinking through the headroom factor — dispatching 20% faster than synthesis produces would grow in-flight buffers too much, so I should dial it back to around 1.1."

This reflects a tension between exploration (probing for more capacity) and exploitation (stable operation). A higher headroom factor might discover that the system can actually sustain a higher throughput, but it risks over-committing. A lower factor is safer but might leave headroom on the table.

The assistant also notes that the memory budget provides a hard safety ceiling: even if the headroom factor causes a slow drift upward in in-flight count, the budget's acquire() call will block once memory is exhausted, providing a natural backstop.

The Todo List: From Reasoning to Action

The message concludes with a structured todo list that translates the reasoning into concrete implementation steps:

  1. Add synth_completion_count AtomicU64, wire into synthesis workers
  2. Add synthesis rate tracking to DispatchPacer (EMA of synth interval)
  3. Add synth rate cap + anti-windup to pacer interval()/update()
  4. Wire synth_count into dispatcher update() calls This todo list is notable for its clarity and its fidelity to the reasoning. Each item corresponds directly to a design decision made during the reasoning process. The first item implements the measurement infrastructure. The second implements the signal processing (EMA). The third implements the control logic (cap + anti-windup). The fourth wires everything together.

What Makes This Message Exceptional

Message 3484 is remarkable for several reasons.

First, the depth of the reasoning. The assistant doesn't just accept the first solution that comes to mind. It explores multiple approaches, identifies their weaknesses, and iterates toward a robust design. The BDP trap — where the assistant catches itself proposing an unstable solution — is a particularly striking example of rigorous thinking.

Second, the integration of multiple disciplines. The assistant draws on control theory (PI control, anti-windup), networking (congestion control, bandwidth-delay product, Little's law), and systems engineering (measurement, feedback loops, saturation constraints). This cross-disciplinary synthesis is exactly what complex systems engineering requires.

Third, the practical engineering judgment. The assistant doesn't just propose a theoretically elegant solution — it considers real-world factors like warmup thresholds, headroom factors, and the interaction between the rate cap and the memory budget. It recognizes that the solution will need refinement based on real behavior ("Rather than over-engineer this now, I'll implement the synth rate cap with anti-windup and refine it based on real behavior").

Fourth, the self-correcting nature of the reasoning. The assistant actively stress-tests its own ideas, catching the instability in the BDP approach before it reaches code. This is the hallmark of a mature engineer: not just generating solutions, but critically evaluating them.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message produces:

  1. A design for a synthesis throughput cap that prevents the dispatch pacer from over-committing the CPU.
  2. An anti-windup mechanism that freezes the PI integral term when the cap is active.
  3. A measurement infrastructure (atomic counter + EMA) for tracking synthesis completion rates.
  4. A warmup strategy that gates the cap during startup to avoid excessive conservatism.
  5. A todo list that translates the design into concrete implementation steps. The message also implicitly creates a framework for thinking about multi-stage pipeline scheduling: instead of optimizing for a single queue depth, the controller should optimize for overall system throughput by measuring and respecting the bottleneck's capacity.

Assumptions and Potential Pitfalls

The assistant makes several assumptions that deserve scrutiny:

  1. Synthesis throughput is a meaningful signal. The assumption is that the EMA of synthesis completion intervals accurately reflects the system's sustainable synthesis capacity. If the measurement is noisy or biased (e.g., because of variable partition sizes or proof types), the cap might be too conservative or too aggressive.
  2. The headroom factor is safe. The assistant assumes that a small headroom factor (1.1 or 1.2) allows probing for additional capacity without causing instability. This is a reasonable assumption, but it depends on the dynamics of the system. If the headroom causes a slow drift that eventually hits the memory budget, the system might oscillate between budget exhaustion and recovery.
  3. Anti-windup is sufficient. Freezing the integral term when the cap is active prevents windup, but it also means the controller stops learning during capped periods. If the system's characteristics change while the cap is active (e.g., the GPU becomes slower), the controller might not adapt until the cap lifts.
  4. The warmup threshold is adequate. The assistant gates the cap with a warmup threshold to avoid being overly conservative during startup. But if the warmup period is too short, the cap might activate before the EMA has stabilized, causing premature throttling. If too long, the system might overshoot before the cap engages.
  5. The synthesis completion counter is correctly placed. The assistant plans to increment the counter when synthesis workers push work to the GPU queue. This measures the rate at which synthesis produces work, not the rate at which the GPU consumes it. If there's a significant delay between pushing and GPU processing (e.g., because of queueing), the measured rate might not reflect the actual bottleneck.

Broader Implications

The problem addressed in this message is not specific to GPU proving pipelines. It arises in any multi-stage system where a controller optimizes for a local metric (queue depth) but the global optimum depends on a different metric (system throughput). Examples include:

Conclusion

Message 3484 captures a moment of genuine engineering insight. Presented with a subtle control systems problem — a PI pacer that works well in the common case but destabilizes in an edge case — the assistant reasons through multiple approaches, identifies a fundamental instability, and converges on a solution that combines measurement, control theory, and practical engineering judgment.

The synthesis throughput cap with anti-windup is not a hack; it's a principled extension of the control system that makes it robust to varying bottleneck conditions. By measuring the actual synthesis throughput and using it as a ceiling on the dispatch rate, the assistant creates a self-regulating loop that automatically adapts to whichever stage of the pipeline is the bottleneck. The anti-windup logic ensures that the PI controller doesn't build up excess correction that would cause a burst when the cap lifts.

This message is a testament to the value of rigorous reasoning in engineering. The assistant doesn't just implement the first solution that comes to mind. It explores the solution space, stress-tests each approach against the dynamics of the system, catches its own mistakes, and iterates toward a robust design. The result is a dispatch pacer that can handle both GPU-bound and CPU-bound scenarios, adapting to the system's actual bottleneck without requiring fixed thresholds or manual tuning.

In the broader context of the opencode session, this message represents the culmination of an extended effort to optimize GPU pipeline scheduling. The team has moved from a simple semaphore to a damped P-controller, to a PI-controlled pacer with GPU rate feed-forward, and finally to a PI pacer with a synthesis throughput cap and anti-windup. Each iteration has been informed by real deployment data and careful analysis of edge cases. The result is a sophisticated control system that balances GPU utilization, CPU contention, memory pressure, and overall system throughput — all without requiring system-specific configuration.

The story of this message is ultimately a story about the power of feedback control, the importance of bottleneck awareness, and the value of rigorous reasoning in complex systems engineering. It's a reminder that the best solutions often come not from applying a single technique, but from synthesizing insights from multiple disciplines — control theory, networking, systems engineering — and adapting them to the specific dynamics of the system at hand.