The Feedback Loop That Broke P-Control: How Late GPU Allocations Corrupted Pipeline Pacing

"And the late allocations cause GPU hickups which break the naive P-control pacing"

This single sentence from the user at <msg id=3470> crystallizes a profound systems engineering insight that had been eluding the development team across multiple iterations of GPU pipeline scheduling. In just twelve words, the user identifies the root cause of instability in the dispatch controller, explains why three successive attempts at proportional control had failed to stabilize the system, and provides the essential diagnostic observation that motivates the next architectural shift. To understand the weight of this message, one must trace the tortured path that led to it.

The Context: Three Iterations of Failed Control

The cuzk proving engine operates a complex pipeline: CPU-bound synthesis jobs produce partitioned circuit data, which is then dispatched to GPUs for proving. The team had invested heavily in a pinned memory pool (PinnedPool) to eliminate costly host-to-device (H2D) memory transfers — a fix that worked brilliantly, achieving near-zero transfer times. But a second problem remained: how many synthesis jobs should be in flight at any given moment? Dispatch too few and the GPU starves; dispatch too many and memory pressure explodes.

The team's first attempt was a semaphore-based reactive dispatch (<msg id=3449> area), which limited total in-flight partitions. The user correctly identified that this approach failed because it constrained the wrong variable — it capped total concurrency rather than targeting a specific queue depth of synthesized partitions waiting for the GPU. The assistant then implemented a P-controller (cuzk-pctrl1) that used GPU completion events as a trigger to dispatch in bursts, intentionally overshooting to converge on a steady state. This proved too aggressive, instantly filling all allocation slots. A dampened variant (cuzk-pctrl2) capped burst size at max(1, min(3, deficit * 0.75)), but the system remained unstable.

The assistant's own reasoning in <msg id=3469> reveals the confusion: examining production logs from cuzk-pctrl2, the assistant found 348 pinned buffer allocations alongside 1,530 reuses — a 22% allocation rate — with allocations continuing even 27 minutes into the run. The pool had 87–89 free buffers at checkout time yet was still allocating new ones. The assistant puzzled over this contradiction, writing: "if there are 87 free buffers, why would any allocation fail?" and "the allocations are happening after the reuses with free_remaining=87, which is backwards from what I'd expect."

The User's Insight: Allocations as Noise

The user's message cuts through this confusion with a single, elegant observation. The "late allocations" — pinned buffer allocations that occur deep into a run when the pool should have stabilized — are not merely a performance nuisance. They are actively destructive to the control loop.

Here is the critical chain of causality that the user identified:

  1. Late allocations are slow. Each cudaHostAlloc call is a synchronous GPU driver operation that serializes at the driver level. A single allocation of a 2.4 GiB pinned buffer can take tens of milliseconds — an eternity compared to the sub-millisecond dispatch decisions the controller is trying to make.
  2. Slow allocations cause GPU hiccups. When a synthesis worker thread triggers a pinned buffer allocation, it blocks the GPU driver's internal command queue. Other GPU operations — including the completion of running proving jobs — are delayed. The GPU appears to "hiccup": a sudden pause in throughput that is unrelated to actual computational load.
  3. Hiccups corrupt the pacing signal. The P-controller uses GPU completion events as its sole feedback signal. When a hiccup occurs, the time between completions spikes artificially. The controller cannot distinguish between "the GPU is genuinely busy processing work" and "the GPU is stalled because some thread is allocating memory." It interprets the long gap as a change in GPU throughput and adjusts its dispatch rate accordingly — in the wrong direction.
  4. The controller reacts to noise, not signal. A naive P-controller has no mechanism to filter out transient artifacts. Every completion interval is treated as equally meaningful. When allocations create spurious long intervals, the controller oscillates: it backs off (seeing a long interval as "GPU is saturated"), then when the hiccup passes and completions resume rapidly, it over-corrects and dispatches too aggressively, triggering more allocations. The system enters a limit cycle driven by its own control actions.

The Deeper Systems Principle

The user's observation touches on a fundamental principle of control theory: a controller is only as good as its feedback signal. If the signal contains artifacts from processes outside the controlled system, the controller will chase those artifacts. The P-controller was designed to regulate GPU queue depth by reacting to GPU completion rate. But pinned buffer allocations are not part of the GPU's computational pipeline — they are a separate resource management process that happens to share the same driver infrastructure. The controller was unknowingly trying to regulate two coupled systems (GPU throughput and memory allocation) with a signal that conflated both.

This also explains why the dampened P-controller (cuzk-pctrl2) failed to stabilize. Dampening reduces the magnitude of response to any single signal, but it does not address the fundamental problem of signal corruption. A dampened controller reacting to noise is still a controller reacting to noise — it just does so more slowly, which can actually make the system harder to diagnose because oscillations occur at lower frequencies.

Input Knowledge Required

To fully appreciate this message, one must understand several layers of the system:

Output Knowledge Created

This message generates several critical insights that reshape the development trajectory:

  1. A diagnosis of the P-controller failure mode. The team now understands that the controller was not merely too aggressive or insufficiently dampened — it was fundamentally misdesigned for a system where the feedback signal is corrupted by an uncontrolled process.
  2. A requirement for signal filtering. Any replacement controller must incorporate mechanisms to distinguish genuine throughput signals from allocation-induced artifacts. This directly motivates the Exponential Moving Average (EMA) smoothing that the PI pacer uses, as well as the bootstrap phase that waits for stable measurements before engaging PI control.
  3. A constraint on the memory pool design. If late allocations are destabilizing the controller, then the pool must either (a) reach steady state faster so late allocations stop occurring, or (b) have a hard ceiling that prevents allocations beyond a certain point, forcing the system to cope with existing buffers even under transient spikes.
  4. A validation of the PI pacer approach. The PI controller with EMA feed-forward that the assistant was implementing in parallel (<msg id=3448> through <msg id=3457>) directly addresses this problem: the EMA smooths out transient hiccups, the feed-forward GPU rate provides a reference independent of allocation artifacts, and the integral term can absorb sustained deviations without overreacting to individual noisy samples.

Assumptions and Potential Mistakes

The user's message makes one implicit assumption that deserves scrutiny: that the P-controller's instability is primarily caused by allocation hiccups, rather than by other dynamics in the system. The deep synthesis pipeline — where multiple synthesis workers run concurrently on CPU threads with varying completion times — introduces its own noise into the dispatch timing. Even with zero allocation hiccups, a P-controller operating on raw completion intervals might still oscillate due to the stochastic nature of CPU synthesis times. The allocation hiccups are likely the dominant source of noise, but not the only one.

Additionally, the user assumes that the PI pacer with EMA will solve the problem. The EMA does filter high-frequency noise, but it also introduces phase lag — the controller responds more slowly to genuine changes in GPU throughput. If the GPU workload changes rapidly (e.g., switching between proof types with different computational requirements), the lag could cause transient starvation or overshoot. The assistant's subsequent implementation of a synthesis throughput cap with anti-windup (<msg id=3471> area) addresses one such edge case, suggesting that the user's diagnosis was correct but not exhaustive.

The Thinking Process Visible

The assistant's extensive reasoning in <msg id=3469> shows a mind grappling with contradictory data. The assistant correctly identifies that 348 allocations against 1,530 reuses represents a 22% allocation rate, that allocations continue 27 minutes into the run, and that the pool has 87–89 free buffers. But the assistant cannot resolve the paradox: why does checkout allocate new buffers when free buffers are available? The assistant speculates about burst patterns exceeding previous peaks, about the dispatch throttle not being aggressive enough — all plausible but ultimately secondary hypotheses.

The user's message provides the missing piece. The assistant was looking at the effect (allocations happening despite free buffers) without connecting it to the cause (those allocations are triggered by transient spikes that the controller itself creates). More importantly, the assistant was treating allocations as a performance metric to be optimized, while the user recognized them as a control system pathology — a signal corruption that actively undermines the controller's ability to function.

This is a classic pattern in complex systems debugging: the person closest to the implementation sees individual data points and tries to optimize each one, while the person with broader system perspective sees the feedback loops that connect those data points into a self-destabilizing cycle. The user's message is valuable precisely because it reframes the problem from "how do we reduce allocations?" to "how do we build a controller that is robust to allocation artifacts?" — a fundamentally different question with a fundamentally different solution space.

Conclusion

The user's twelve-word message at <msg id=3470> is a masterclass in diagnostic insight. It identifies the coupling between two subsystems (memory allocation and GPU dispatch) that were being treated as independent, explains why that coupling breaks the control loop, and implicitly defines the requirements for a working solution: the controller must either eliminate late allocations entirely or become robust to the noise they create. The PI pacer with EMA smoothing that the team was already building turns out to be the right answer — not because it was designed to handle this problem, but because its signal-filtering properties happen to address the pathology that the user so precisely identified. In systems engineering, the difference between a working controller and a broken one is often not the control law itself, but the quality of the signal it receives.