The Pacer Emerges: From Semaphore to PI-Controlled Dispatch with Synthesis Throughput Cap

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, every millisecond of GPU idle time represents wasted computational capacity. The CuZK proving engine — a specialized system for generating Groth16 proofs at scale for the Filecoin network — had already achieved a remarkable breakthrough: a CUDA pinned memory pool that slashed host-to-device transfer times from 14 seconds to near-zero milliseconds. But as the team would discover, fixing the memory bottleneck was only the beginning. A subtler, more insidious problem lurked in the dispatch logic that fed synthesized partitions to the GPU — a problem that would drive the team through five increasingly sophisticated control system designs in rapid succession.

This article chronicles that journey. Over the course of a single intense segment of development, the team moved from a simple reactive semaphore, through a proportional controller (P-controller), then a dampened P-controller, then a PI-controlled dispatch pacer with exponential moving average (EMA) smoothing, and finally added a synthesis throughput cap with anti-windup to handle CPU-bound edge cases. Each iteration revealed new insights about the system's dynamics, and each failure brought the team closer to a principled understanding of what it takes to keep a GPU pipeline stable under varying workloads.

The Foundation: Consolidating the Pinned Memory Pool

The segment opens with a moment of consolidation. The pinned memory pool work — the breakthrough that had eliminated the GPU underutilization crisis — existed as an extensive uncommitted diff spanning multiple files and components. The assistant's first task was to transform this sprawling change into two clean, focused commits: one for the bellperson pinned backing and one for the CuZK pinned pool and reactive dispatch throttle ([msg 3346], [msg 3352]).

This consolidation was itself a significant engineering achievement. The diff touched the Rust-level pipeline dispatcher, the C++ CUDA GPU code, the memory budget system, and the build infrastructure. Separating it into coherent commits required understanding which changes were logically independent and which were coupled. The bellperson commit captured the pinned backing infrastructure — the mechanism that allows Rust's Vec<Fr> arrays to be allocated on pinned (page-locked) host memory. The CuZK commit captured the PinnedPool allocator, the reactive dispatch throttle that prevented buffer thrashing, and the PCE caching fix that restored memory budget health.

The result of these fixes was dramatic. Before the pinned pool, the GPU was active only ~1.2 seconds per partition but the GPU mutex was held for 1.6–14 seconds because cudaMemcpyAsync had to stage data through a tiny internal bounce buffer at 1–4 GB/s instead of the PCIe Gen5 line rate of ~50 GB/s. After the fix, ntt_kernels time dropped from 2,000–14,000ms to 0ms, total per-partition GPU time dropped from 8–19 seconds to 934–994ms, and the pinned pool reuse ratio went from 474:12 (thrashing) to 24:48 (2:1 reuse). The memory budget, previously stuck at 5 GiB, now had 288 GiB available. PCE caching, previously blocked by budget exhaustion, was now working.

These numbers told a story of success. The pinned memory pool had fixed the GPU. But as the team was about to discover, the dispatch logic that fed work to the GPU was fundamentally flawed — and fixing it would require a journey into control theory.

The Critique: Why Permits Were the Wrong Abstraction

The user's message at [msg 3356] is a masterclass in precise technical critique. The reactive dispatch throttle that had been committed alongside the pinned pool used a tokio::sync::Semaphore with 8 permits. The dispatcher would acquire a permit before starting synthesis, and the GPU finalizer would return a permit after completing a proof. This limited the total number of in-flight partitions (synthesis + waiting + on-GPU) to 8.

The user rejected this design in no uncertain terms: "Didn't want 8 basic 'permits', that's actually wrong way to modulate scheduling."

The reasoning was subtle but critical. A permit-based system limits the total number of things in flight, but it does not distinguish between synthesizing (which consumes CPU and memory) and waiting-for-GPU (which consumes only memory). When the GPU finishes a job and releases a permit, the semaphore allows exactly one new synthesis to start. But if the GPU just finished a job and there are zero waiting partitions, the GPU will go idle while that one synthesis runs — a classic pipeline stall.

The correct control variable, the user argued, is not the number of in-flight permits but the number of partitions waiting for the GPU — the queue depth of synthesized, ready-to-prove work sitting in pinned memory. By targeting a specific queue depth, the system can ensure the GPU always has work to consume without overprovisioning synthesis capacity.

The user's proposed mechanism was elegant: "When GPU finishes a job, we should look at how many synthesis are Waiting, then if that number is greater than target (N, default 8) don't start new synths, if it's anything below the target start this many synths as we are below the target."

This is a textbook proportional controller operating on queue depth. The error signal is target_waiting - actual_waiting. If the error is positive, dispatch that many new syntheses. If the error is negative or zero, do nothing. The system self-regulates: when the GPU consumes a waiting partition, the queue depth drops below target, triggering new syntheses to fill the gap.

The P-Controller Epiphany: Intentional Overshoot

The assistant's response at [msg 3361] is one of the most remarkable messages in the entire conversation — a 4,300-word extended reasoning session that traces the full arc from understanding the user's critique to designing a complete replacement for the dispatch mechanism.

The assistant begins by articulating the problem: the semaphore model treats all in-flight work as equivalent, limiting total pipeline depth rather than GPU queue depth. The correct approach is a proportional controller where the deficit (target - waiting) determines how many new syntheses to start.

But the assistant quickly identifies a timing issue: when synthesis jobs are dispatched, they don't appear in the GPU queue until synthesis completes (which takes 20–60 seconds). If the dispatcher checks the deficit against only completed items, it could end up dispatching way more synthesis tasks than intended, with all of them landing in the GPU queue at once and overshooting the target.

The assistant's initial instinct is to track an in_flight counter to account for items being synthesized but not yet in the queue. But then comes the critical insight — a re-reading of the user's specification reveals that the user wants overshoot:

"We will overshoot, but we must overshoot, to then undershoot little less hopefully and so on. Essentially like a Proportional (not quite PI/PID) controller"

This is the P-controller epiphany. The overshoot is not a bug — it is a feature. By dispatching N items when the queue is empty, the system intentionally overshoots, then GPU consumption naturally brings the queue back down, and the next dispatch cycle undershoots slightly less, converging to a steady state. The assistant formalizes this: the error signal is deficit = target - waiting, the control action is to start deficit new syntheses, and the gain is 1 — a pure proportional controller.

The assistant designs a two-phase loop: Phase 1 waits for a GPU completion event (or startup), Phase 2 dispatches the full deficit in a burst, then returns to Phase 1. This is fundamentally different from the continuous-loop approach that tries to dispatch one item at a time and blocks on budget between each one. The burst approach creates intentional overshoot: if the deficit is 8, dispatch 8 items even if that temporarily exceeds the budget. The budget system will naturally pace the dispatcher, but the intent is to fill the pipeline aggressively.

The assistant implements this design, replacing the Semaphore with a tokio::sync::Notify that the GPU finalizer signals on completion. The dispatcher waits on the notify, calculates the deficit, dispatches the burst, and loops back to waiting. The implementation is deployed as cuzk-pctrl1.

The First Deployment: Too Aggressive

The deployment of cuzk-pctrl1 revealed an immediate problem. The P-controller with a gain of 1.0 was too aggressive. When the GPU completed one partition and the waiting queue was still empty (because the burst of 8 hadn't landed yet), the deficit would again be 8, triggering another burst of 8. This positive feedback loop would instantly fill all available allocation slots, preventing the controller from ever converging to a steady state.

The user's diagnosis at [msg 3408] was precise: "Currently the problem is that we nearly instantly bump into allocating all allocatable slots so with P=1 the p-controller can't stabilize."

The root cause was that the dispatcher's burst dispatch, combined with the long synthesis pipeline, created a situation where the controller was operating on stale information. By the time the dispatched syntheses completed and appeared in the waiting queue, the controller had already dispatched more work based on the same stale deficit. The system was oscillating because the feedback delay was longer than the controller's reaction time.

The Dampened P-Controller: Reducing the Gain

The user's proposed fix at [msg 3410] was a dampening factor: instead of dispatching the full deficit, dispatch floor(max(1, min(3, deficit * 0.75))). This formula applied a 0.75 gain to the deficit, floored it to an integer, ensured at least 1 dispatch per GPU event (to prevent starvation), and capped the burst at 3 (to prevent runaway growth).

The assistant's reasoning at [msg 3411] shows the careful interpretation of this formula. The user's initial expression floor(min(1, deficit * 0.75)) didn't parse correctly — min(1, X) for any X >= 1.34 would always return 1, making the controller always dispatch exactly 1 item regardless of deficit. The assistant correctly inferred that the user meant a clamping structure combining two separate comments: a lower bound of 1, an upper bound of 3, and a 0.75 gain.

The assistant verified the formula against the user's examples:

The Deeper Diagnosis: Pipeline Depth and Signal Coarseness

But the dampened P-controller also proved insufficient. At [msg 3426], the user delivered the diagnosis that would reshape the entire approach: "Still not stable, this is a pretty deep pipeline. The Waiting count is still the best signal we have from scheduling theory, but the input is too coarse."

This message contains two crucial insights. First, the pipeline depth is the root cause of instability. In a deep pipeline, there is a significant delay between when a synthesis job is dispatched and when it completes and appears in the waiting count. During that delay, the dispatcher operates on stale information — it sees a low waiting count and dispatches more work, but those earlier dispatches haven't landed yet. By the time they do land, the system has already overshot. This is a classic problem in control theory: delay in the feedback loop causes oscillation.

Second, the waiting count is conceptually the right signal — it directly measures how much work is queued for the GPU — but the raw instantaneous value is too noisy and too delayed to serve as a stable feedback signal. A P-controller reacts to the current error, but when the error signal itself is a lagging indicator, the controller inevitably overcorrects.

The user's proposed solution was a "pacer" approach with PI control (Proportional-Integral) operating on a smoothed input signal. A PI controller has two terms: the proportional term reacts to the current error (like the existing P-controller), while the integral term accumulates error over time. The integral term is crucial for a system with delay: even if the instantaneous error is small (because the waiting count hasn't updated yet), the integral of past errors provides a memory of the system state that prevents oscillation.

The user suggested three options for the smoothed input signal:

  1. A short EMA of the Waiting count — Exponential Moving Average filters out high-frequency noise from the raw waiting count, providing a smoother feedback signal.
  2. GPU consumption rate — Instead of measuring queue depth, measure the rate at which the GPU consumes work.
  3. A combined signal — Some weighted combination of queue depth and consumption rate, providing both position and velocity information. The phrase "pacer" is evocative: rather than reacting to events (GPU completions triggering bursts of dispatches), a pacer would work like a metronome, dispatching work at a computed interval that matches the system's sustainable throughput. This is a fundamental shift from reactive to predictive scheduling. The user added a critical data point at [msg 3428]: "Note that the system needs quite some time to stabilize from under/overshoot, multiple minutes (single synth is 20-60s depending on contention so from dispatch to gpu)." This confirmed that the PI gains needed to be extremely conservative — with a 20-60 second feedback delay, even small corrections could cause oscillation if applied too aggressively.## The PI Pacer: A Continuous Dispatch Metronome The assistant's response at [msg 3427] designed the PI-controlled dispatch pacer in extraordinary detail. The core insight was that the dispatch rate should match the GPU's consumption rate as a feed-forward term, with PI correction on the smoothed queue depth error providing fine-grained adjustment around that baseline. The design had three key components: Feed-forward rate. The pacer maintains an exponential moving average (EMA) of the GPU inter-completion interval — the time between successive GPU proof completions. This provides a direct measurement of how fast the GPU is consuming work. The base dispatch rate is set to match this rate: if the GPU completes one partition per second, the pacer dispatches one synthesis job per second. This feed-forward term does the heavy lifting; it ensures the dispatch rate is always in the right ballpark regardless of queue depth. PI feedback on smoothed queue depth. The pacer maintains a second EMA of the GPU queue depth (the number of synthesized partitions waiting for the GPU). The error signal is target - ema_waiting. The proportional term (Kp) reacts to the current error, while the integral term (Ki) accumulates error over time. The PI correction adjusts the dispatch interval relative to the feed-forward rate: if the queue is below target, the interval shortens (dispatch faster); if above target, the interval lengthens (dispatch slower). Bootstrap phase. Before any GPU completions have occurred, the pacer has no rate data. During this phase, it dispatches target items at a fixed 200ms spacing to prime the pipeline, then waits for the first GPU completion to calibrate the GPU rate EMA. After calibration, it switches to PI-controlled pacing. The assistant implemented this design with conservative gains: Kp = 0.1, Ki = 0.008, and EMA alpha = 0.3 for both the waiting count and GPU interval smoothing. These tiny gains reflected the 20-60 second feedback delay — the controller needed to be extremely cautious to avoid oscillation. The dispatch interval was clamped to [50ms, 60s] to prevent extreme values. The deployment at [msg 3480] captured the commit message: "Replace the burst-based P-controller with a continuous pacer that regulates synthesis dispatch rate using PI control with GPU rate feed-forward." The commit touched 2 files and 291 insertions, replacing 56 lines of the old burst dispatch logic.

The First Pacer Deployment: Success with an Edge Case

The PI pacer deployed as cuzk-pacer1 and the user's verdict at [msg 3483] was positive: "Generally works wonderfully compared to previous state."

But the user immediately identified a critical edge case. On systems where the CPU-bound synthesis stage was the bottleneck — where the CPU could not produce synthesized partitions fast enough to keep the GPU fully fed — the pacer's behavior became pathological. The PI controller, seeing an empty GPU queue, drove the dispatch interval below the GPU consumption rate in an attempt to fill the queue. This flooded the system with concurrent synthesis jobs, which then contended for CPU resources. The user observed that running 22 concurrent syntheses was worse than running 16 — the extra jobs starved each other for CPU time, reducing individual synthesis throughput and degrading overall system performance.

This is a classic control systems failure mode: the controller was optimizing for the wrong metric. It was trying to maintain a target queue depth of synthesized partitions waiting for the GPU, but when synthesis was the bottleneck, chasing queue depth only made the problem worse. The pacer could not distinguish between "the queue is empty because the GPU is fast" and "the queue is empty because synthesis is stalled." Both conditions produced the same error signal — queue depth below target — but demanded opposite responses.

The user's diagnosis was precise and profound: "the control should optimize for overall sustained system throughput." This redefined the objective function. Queue depth was a proxy, not the true goal. The true goal was proofs completed per unit time, and that depended on both GPU and CPU resources interacting in complex ways.

The Synthesis Throughput Cap: Closing the Loop

The assistant's response at [msg 3484] was arguably the most extended reasoning chain in the entire segment — a deep exploration of how to make the pacer bottleneck-aware. The assistant worked through multiple frameworks:

TCP congestion control analogy. The assistant recognized that the problem maps onto classic congestion control. The optimal number of in-flight requests equals the bandwidth-delay product (BDP). But the assistant quickly identified a critical flaw: if synthesis duration increases due to contention, the BDP formula would recommend more in-flight requests, creating a positive feedback loop that makes contention worse. This is unstable. The assistant caught this before it reached code — a moment of genuine self-correction.

Little's law. The assistant considered using Little's law (throughput = in-flight / latency) to find the concurrency level that maximizes throughput. As in-flight increases, throughput climbs linearly during spare capacity, plateaus at saturation, then drops due to contention overhead. The optimal operating point is the "knee" of this curve.

Direct concurrency limiting. A simpler approach — just cap the number of in-flight syntheses. But the user explicitly rejected fixed thresholds, demanding an adaptive mechanism that works across different systems with different bottlenecks.

The solution the assistant converged on was the synthesis throughput cap: measure the rate at which synthesis completes, and clamp the dispatch rate to not exceed that rate (with a small headroom factor). This creates an implicit negative feedback loop: if synthesis slows due to contention, the measured rate drops, automatically reducing the dispatch ceiling and relieving CPU pressure. The key insight is that the synthesis completion rate is a direct measurement of the bottleneck's capacity.

Before the assistant could implement this design, the user added two critical constraints at [msg 3485] and [msg 3486]:

  1. Proof type transitions. Different proof types (SnapDeals, PoRep, WinningPoSt, WindowPoSt) require different pinned buffer sizes. A SnapDeals partition might use 2.4 GiB per buffer, while PoRep uses 4.17 GiB. When the proof type changes, the pinned memory pool must re-allocate buffers via cudaHostAlloc, a synchronous GPU driver operation that stalls the pipeline. The pacer's synthesis rate measurement would be temporarily skewed during these transitions.
  2. Startup skew. During the first job after deployment, every pinned buffer is a fresh allocation via cudaHostAlloc. These allocations are orders of magnitude slower than reusing buffers from the pool during steady-state operation. If the pacer's synthesis rate measurement includes these startup samples, the EMA would be dragged downward, producing an artificially low estimate of synthesis throughput. The cap would then constrain the dispatch rate to match this pessimistic estimate, starving the GPU of work long after the system had actually reached steady state. The assistant's response at [msg 3487] integrated both constraints into the design. For startup skew, the assistant planned a warmup threshold — "hold off applying the synth rate cap until we've seen at least 5-10 completions" — to prevent early measurements from distorting steady-state control. For proof type transitions, the assistant relied on the EMA's alpha parameter (0.15) to converge within roughly 10 samples after a regime change. This was a deliberate trade-off: fast enough to adapt within a few completions, but slow enough to filter out noise from individual allocations.

Finding the Measurement Point

With the design settled, the assistant needed to find the exact code locations where the measurement infrastructure would be wired. This required a series of grep searches and file reads that, while mundane on the surface, represented the critical transition from abstract reasoning to concrete implementation.

The first grep, at [msg 3488], searched for gpu_work_queue.*push" — expecting a string literal argument — and found nothing. The trailing double quote was a fossil of an incorrect assumption about how the push call was structured. The assistant corrected this at [msg 3489], dropping the trailing quote, and found two matches at lines 1633 and 2704 of engine.rs:

gpu_work_queue.push(p_job_seq, p_idx, job);
gpu_work_queue.push(0, 0, job);

Line 1633 was identified as the primary synthesis handoff — the exact point where a synthesis worker finishes its work and pushes the result to the GPU queue. This is where the atomic counter must be incremented.

But knowing where to increment was not enough. The assistant also needed to know how the counter variable gets into the worker's scope. At [msg 3491], the assistant read the synthesis worker spawn loop:

// Synthesis worker pool: receive dispatched work, synthesize, push to GPU queue
for sw_id in 0..synth_worker_count {
    let synth_dispatch_rx = synth_dispatch_rx.clone();
    let gpu_work_queue = gpu_work_queue.clone();
    ...

This revealed the cloning pattern — each worker clones the shared state from Arc-wrapped references. The assistant would follow this exact pattern for the new atomic counter.

The Implementation: Four Bullet Points

At [msg 3492], the assistant distilled the implementation into four bullet points:

  1. DispatchPacer struct — add synth rate fields (EMA state, warmup counter, cap flag)
  2. Shared state — add synth_completion_count: Arc<AtomicU64>
  3. Synthesis workers — clone the counter and increment after gpu_work_queue.push()
  4. Dispatcher — pass synth count to pacer.update() on each GPU completion event Each bullet encoded significant engineering decisions. The EMA state would follow the same pattern already used for GPU rate tracking, ensuring consistency. The atomic counter would be incremented after the push, ensuring the measurement reflected completed work actually available for GPU consumption. The anti-windup logic would freeze the PI integral term when the cap was active, preventing accumulated error from causing a dispatch burst when the bottleneck cleared. The edit itself, applied at [msg 3493], replaced the entire DispatchPacer with a version that included the synthesis rate cap, anti-windup, and warmup gating. The update() method now accepted a synth_count parameter and computed an EMA of the synthesis inter-completion interval. The interval() method clamped the PI-computed dispatch interval so that the dispatch rate never exceeded the measured synthesis rate times a headroom factor (initially 1.1). When the cap was active, the integral accumulation was skipped — a textbook anti-windup implementation.

The Silent Signal

The user's response to this edit, at [msg 3494], was empty — nothing more than blank XML tags. But this silence spoke volumes. It signaled trust in the assistant's technical judgment, shared understanding of the remaining wiring steps, and forward momentum. The edit was complex — it touched the core control loop of the system — but the reasoning had been thoroughly discussed, the approach agreed upon. No further clarification was needed.

This silence is only possible when both parties share a deep mental model of the system. The user knew exactly what the remaining steps were: adding the atomic counter, wiring it into synthesis workers, updating all pacer.update() calls in the dispatcher loop, and updating the periodic status log. The assistant knew the user knew. The empty message was an implicit "proceed."

The Broader Significance

The synthesis throughput cap represents the culmination of five iterations of dispatch control, each building on lessons from the previous. The evolution — from semaphore to burst controller to damped controller to PI pacer to capped PI pacer — mirrors the development of real-world control systems from simple feedback loops to sophisticated multi-variable controllers that respect the actual capacity of the system.

The design is elegant because it creates a self-regulating loop without requiring explicit mode switching. The pacer automatically defers to whichever stage of the pipeline is the bottleneck. When the GPU is the bottleneck, the PI controller's queue-depth regulation dominates, keeping the GPU fed. When synthesis is the bottleneck, the throughput cap engages, preventing over-dispatch and CPU contention. The transition between these regimes is continuous and automatic, governed by the measured rates rather than hardcoded thresholds.

This pattern — measure the bottleneck's throughput, cap the dispatch rate, and add anti-windup — is broadly applicable beyond GPU proving pipelines. It arises in any multi-stage system where a controller optimizes for a local metric but the global optimum depends on a different metric. Web servers, data pipelines, manufacturing systems — all face the same fundamental challenge of balancing upstream and downstream capacity without explicit coordination.

The Conceptual Arc: What the Journey Reveals

The progression from semaphore to capped PI pacer is not just a story of five increasingly sophisticated control systems. It is a story about how understanding deepens through failure. Each iteration revealed new dynamics that the previous design had ignored:

  1. The semaphore treated all in-flight work as equivalent, ignoring the distinction between synthesizing and waiting-for-GPU. This failed because it couldn't maintain a GPU queue depth.
  2. The P-controller correctly targeted queue depth but assumed instantaneous feedback. It failed because the synthesis pipeline introduced a delay that caused oscillation.
  3. The dampened P-controller reduced the gain to prevent runaway growth but still operated on a noisy, delayed signal. It failed because dampening alone couldn't compensate for the phase lag.
  4. The PI pacer addressed the delay by accumulating error over time (integral term) and smoothing the input signal (EMA), providing both memory of past errors and noise filtering. It succeeded in stabilizing the pipeline but failed when synthesis became the bottleneck, because it was still optimizing for queue depth rather than throughput.
  5. The capped PI pacer added a synthesis throughput measurement that automatically limited the dispatch rate to what the CPU could sustain, with anti-windup to prevent integral buildup when capped. This created a multi-variable controller that respected both GPU and CPU capacity. This arc mirrors the classic engineering journey of control system design. Each iteration reveals new dynamics, and the control law evolves to handle them. The team moved from a discrete, event-driven model (dispatch N items per GPU completion) to a continuous, time-domain model (dispatch at a computed rate). This is the difference between thinking in terms of counts and thinking in terms of rates — a conceptual leap that separates naive scheduling from principled control.

Conclusion

The segment that began with consolidating the pinned memory pool commits and ended with the synthesis throughput cap represents a remarkable arc of engineering discovery. In the span of a few dozen messages, the team moved from a flawed semaphore-based dispatch model to a sophisticated multi-variable controller that automatically adapts to whichever stage of the pipeline is the bottleneck.

The pinned memory pool fix remains deployed and effective — the GPU is no longer bottlenecked on H2D transfers. The dispatch scheduling logic, the mechanism that decides when to feed work to the GPU, evolved through five iterations, each building on the failures of the previous. The final design — a PI-controlled pacer with EMA feed-forward and a synthesis throughput cap with anti-windup — represents a principled, adaptive solution that can maintain stable, high-utilization scheduling across varying workloads, proof types, and system configurations.

In the end, this segment is a testament to the power of iterative refinement guided by precise observation and control-theoretic thinking. The team didn't just fix bugs — they developed an increasingly accurate mental model of their system's dynamics, and they encoded that understanding into progressively more sophisticated control laws. The result is not just a working GPU pipeline, but a deep understanding of what it takes to keep it working under real-world conditions.

References

[1] "The Control System Odyssey: From Semaphore to PI Pacer in GPU Pipeline Dispatch" — Analysis of the P-controller and dampened P-controller iterations, messages 3346–3426.

[2] "The Synthesis Throughput Cap: Closing the Loop on GPU Pipeline Control" — Analysis of the synthesis throughput cap implementation, messages 3483–3494.