When the Pacer Meets the Wall: A Systems Critique of GPU Dispatch Control

The Message

Generally works wonderfully compared to previous state. Edge-case: When synthesis doesn't keep up (we're synthesis compute constrained it seems on this specific system, though very close match to the GPU), the inverval collapses to below gpu ema (which by itself must happen to get waiting partitions), which eventually causes us to max out memory budget with running synthesis. There is another consideration which is that running too many synthesis starves them against each other, so it might be better to run 16 vs 22. Pacer feels like the right tool for the job, but the control should optimize for overall sustained system throughput, which means either we need to clamp the pacer to avoid creating too many running synthesis jobs (not a fixed threshold, this should run on many systems with different bottlenecks), or consier different optimizer for the pacer which somehow lands on a stable maximum throughput.

This message, delivered by the user at index 3483 of a long-running coding session, is a deceptively dense piece of systems engineering feedback. On its surface, it reports an edge case in a newly deployed GPU dispatch pacer. But beneath that lies a profound critique of control-theoretic assumptions, a nuanced understanding of distributed bottlenecks, and a clear-eyed vision for what a production scheduler must actually optimize. To appreciate its depth, we must understand the journey that led to this moment.

The Context: From Burst Dispatch to PI Control

The session had been iterating for hours on a single, maddening problem: GPU underutilization in a zero-knowledge proof proving pipeline. The system had two major phases — synthesis (CPU-bound circuit construction) and proving (GPU-bound computation). The assistant had already solved the most glaring bottleneck by implementing a zero-copy pinned memory pool ([msg 3480]), which eliminated expensive H2D transfers. But the dispatch logic — the mechanism that decides when to send synthesized work to the GPU — remained unstable.

The first attempt was a semaphore-based reactive dispatch, which the user correctly identified as fundamentally flawed: it limited total in-flight partitions rather than targeting a specific queue depth of synthesized partitions waiting for the GPU. The assistant then implemented a P-controller (proportional control) that used GPU completion events as feedback, dispatching a burst of work on each event to converge on a steady state. But this proved too aggressive, instantly filling all allocation slots. A dampened version capped burst size at max(1, min(3, deficit * 0.75)), but the system remained unstable because the deep synthesis pipeline made the raw waiting count a noisy and delayed signal.

The breakthrough was the PI-controlled dispatch pacer ([msg 3480]). This was a sophisticated piece of control theory applied to systems engineering: an exponential moving average (EMA) of GPU inter-completion intervals served as a feed-forward rate estimate, while a PI (proportional-integral) correction acted on the smoothed GPU queue depth error. A bootstrap phase dispatched a fixed number of items at 200ms spacing before the first GPU completion, then switched to timer-based pacing at the PI-computed interval. The gains were deliberately conservative (Kp=0.1, Ki=0.008) to account for the 20-60 second synthesis delay. The assistant committed this as 0ba6cbca with the message "cuzk: PI-controlled dispatch pacer for GPU queue depth" and deployed it as pacer1.

What the User's Message Reveals

The user's first sentence — "Generally works wonderfully compared to previous state" — is significant. It confirms that the PI pacer is a genuine improvement over the burst-based P-controller. The steady dispatch rate has likely stabilized the pinned pool allocation pattern, breaking the vicious cycle of bursty dispatch → pool exhaustion → cudaHostAlloc → GPU driver serialization → timing jitter → more bursty dispatch. This is a genuine engineering win.

But the user immediately pivots to the edge case, and this is where the message becomes extraordinary. The user has identified that on this specific system, the bottleneck is not the GPU but the CPU synthesis pipeline. The system is "synthesis compute constrained" — the CPU cannot produce synthesized partitions fast enough to keep the GPU fully fed. This is a fundamentally different regime than what the pacer was designed for.

The pacer's control loop works as follows: if the GPU queue (the number of synthesized partitions waiting for the GPU) drops below the target, the PI controller reduces the dispatch interval, causing the dispatcher to send work to the synthesis workers more frequently. This increases the number of concurrent synthesis jobs, which should increase the synthesis throughput, which should refill the GPU queue. But this reasoning assumes that synthesis throughput scales with the number of concurrent synthesis jobs — an assumption that breaks down under CPU contention.

The Starvation Problem

The user's key insight is that "running too many synthesis starves them against each other." When the system is CPU-bound, adding more concurrent synthesis jobs does not increase aggregate throughput. Instead, each job gets less CPU time, context switching overhead increases, cache thrashing intensifies, and the total system throughput actually decreases. The user provides a concrete data point: "it might be better to run 16 vs 22" concurrent syntheses. This is not a guess — it's an observation from the running system. Somewhere between 16 and 22 concurrent syntheses, the system crosses a threshold where CPU contention dominates and throughput plateaus or declines.

This is a classic systems engineering phenomenon, but it's particularly insidious here because the pacer has no visibility into it. The pacer sees only two signals: the GPU queue depth (how many synthesized partitions are waiting) and the GPU completion rate (how fast the GPU is consuming them). When the GPU queue is empty, the pacer's natural response is to dispatch faster — but if the system is already at peak CPU throughput, dispatching faster just creates more concurrent syntheses that fight for CPU time without increasing the output rate. The queue remains empty, the pacer dispatches even faster, and the system spirals into a state of high CPU contention, high memory pressure (from all those in-flight syntheses), and low overall throughput.

The Control Problem Redefined

The user's most profound statement is: "the control should optimize for overall sustained system throughput." This is a redefinition of the control objective. The pacer was designed to optimize for GPU queue depth — keeping the GPU fed. But the user correctly identifies that queue depth is a proxy, not the true goal. The true goal is system throughput: proofs completed per unit time. And system throughput is a function of both GPU and CPU resources, with complex interactions between them.

The user proposes two directions for fixing this. The first is to "clamp the pacer to avoid creating too many running synthesis jobs" — but explicitly notes this cannot be a fixed threshold because "this should run on many systems with different bottlenecks." A hard limit of 16 concurrent syntheses might be optimal on this system but suboptimal on a system with more CPU cores or faster synthesis. The clamp must be adaptive.

The second direction is to "consider different optimizer for the pacer which somehow lands on a stable maximum throughput." This is a call for a higher-level control loop that measures system throughput directly and adjusts the pacer's parameters to maximize it. This could be a form of extremum-seeking control, where the controller perturbs the dispatch rate and observes whether throughput increases or decreases, then moves in the direction of improvement. Or it could be a model-based approach that estimates the system's throughput curve from observed behavior.

Assumptions and Their Failure Modes

The pacer's design made several assumptions that this edge case exposes:

That synthesis throughput is elastic. The pacer assumes that increasing the dispatch rate will increase the synthesis output rate. This holds when the CPU has spare capacity but fails when the CPU is saturated.
That GPU queue depth is a sufficient proxy for system throughput. The pacer treats a non-empty GPU queue as evidence that the system is healthy. But a non-empty queue can coexist with low system throughput if the CPU is thrashing.
That the bottleneck is stable. The pacer assumes the system operates in a single regime (GPU-bound). But the user observes that the system is "very close match" — the GPU and CPU are nearly balanced, meaning small perturbations can shift the bottleneck from one to the other.
That more concurrent work is always better. This is the most fundamental assumption, and the one most directly contradicted by the user's observation. In any system with finite resources, adding more concurrent work beyond a point reduces per-work efficiency due to contention.

The Thinking Process Visible in the Message

The user's message reveals a sophisticated mental model of the system. They have internalized the pacer's control logic well enough to predict its behavior under CPU-bound conditions. The phrase "which by itself must happen to get waiting partitions" shows that the user understands the pacer's mechanism: the interval must collapse below the GPU EMA to build a queue, because if the dispatch rate exactly matched the GPU consumption rate, the queue would never grow. But the user also sees the unintended consequence: this necessary collapse, when the CPU is saturated, creates a runaway effect.

The parenthetical "(not a fixed threshold, this should run on many systems with different bottlenecks)" is particularly telling. It shows that the user is thinking about generality — this is not a one-off hack but a system that will be deployed across diverse hardware configurations. The solution must be adaptive, not hardcoded.

The phrase "Pacer feels like the right tool for the job" is also significant. Despite identifying a fundamental flaw, the user does not reject the pacer architecture. They recognize that the PI control framework is sound — it just needs a different objective function and additional constraints.

Input and Output Knowledge

To understand this message, one needs knowledge of: control theory (PI controllers, EMA filtering, feed-forward vs. feedback), GPU pipeline architecture (synthesis, proving, H2D transfers), pinned memory management, CPU contention and context switching overhead, and the specific deployment context (the pacer1 binary, the GPU queue target of 8, the synthesis delay of 20-60 seconds).

The message creates new knowledge: the identification of the CPU-bound edge case, the observation that 16-22 concurrent syntheses is the contention threshold, the requirement that the pacer must optimize for system throughput rather than queue depth, and the design constraint that the solution must be adaptive across different hardware.

Conclusion

This message is a masterclass in systems engineering feedback. It praises what works, precisely identifies what doesn't, explains the mechanism of failure, and charts a clear path forward — all in a few dense sentences. The user demonstrates deep understanding of both the control theory underlying the pacer and the real-world behavior of the distributed system. The edge case they identified would likely have remained hidden in synthetic testing, only to emerge in production under CPU-bound workloads. By catching it now and articulating the required fix so clearly, the user saves the team from a painful debugging session later. The message is a reminder that in complex systems, the control loop's objective function is as important as its implementation — and that optimizing for a proxy (queue depth) without considering the true goal (system throughput) can lead to stable but suboptimal behavior.