The 47-Second Phantom: Debugging GPU Rate Calibration in a PI-Controlled Dispatch Pacer

Introduction

In the course of building a high-performance GPU proving pipeline for Filecoin's proof system (cuzk), a team encountered a subtle and pernicious bug: the dispatch pacer—a PI controller responsible for regulating how often synthesized proof partitions are sent to the GPU—was calibrating its GPU rate estimate at 47 seconds per partition, when the actual GPU processing time was closer to 1 second. This 47× error did not merely produce suboptimal throughput; it triggered a self-reinforcing collapse loop that starved the GPU of work, dragged synthesis throughput down with it, and left the entire proving pipeline operating at a fraction of its potential.

The message at the center of this story ([msg 3538]) is the assistant's response to the user reporting this calibration failure. It is a fascinating document because it captures a moment of genuine debugging insight—the assistant correctly identifies the root cause of the bad calibration—but also contains the seeds of an incorrect fix that would later prove insufficient. This article examines that message in depth: the reasoning, the decisions, the assumptions, the mistakes, and the broader lessons about measurement in feedback control systems.

The System: A PI Pacer for GPU Dispatch

To understand the message, one must first understand what the dispatch pacer is and why it exists. The cuzk proving engine processes Filecoin proofs through a multi-stage pipeline. Synthesis (the CPU-intensive construction of proof partitions) runs in parallel across many worker threads. Once synthesized, partitions are dispatched to GPU workers for the computationally intensive NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) operations that produce the final proof.

The challenge is that synthesis and GPU proving have very different time scales. A single partition might take 20–60 seconds to synthesize on CPU but only ~1 second to prove on GPU. If the system dispatches partitions to the GPU too slowly, the GPU sits idle, wasting the expensive hardware resource. If it dispatches too aggressively, the GPU queue grows without bound, consuming memory and potentially causing out-of-memory crashes.

The dispatch pacer is a PI controller (Proportional-Integral) that solves this regulation problem. It maintains a target number of partitions waiting in the GPU queue (the "waiting" count), and adjusts the dispatch interval—how often a new partition is sent to the GPU—based on the error between the target and the current waiting count. The P term (proportional) reacts to current error; the I term (integral) accumulates persistent error over time to eliminate steady-state偏差. A feed-forward term uses the measured GPU processing rate to predict how fast the GPU can consume work, allowing the pacer to preemptively match the dispatch rate to the consumption rate.

The pacer also has a synthesis throughput cap—a ceiling on the dispatch rate derived from the measured synthesis completion rate. This prevents the pacer from dispatching faster than the CPU synthesis pipeline can produce work, which would cause the GPU queue to drain and the GPU to go idle waiting for the next partition.

The pacer operates in two phases:

1. Bootstrap phase: When no GPU rate is known yet, the pacer dispatches 8 partitions at rapid 200ms intervals to fill the pipeline and gather initial measurements.
2. PI control phase: Once the GPU rate is calibrated, the pacer switches to PI control with feed-forward, continuously adjusting the dispatch interval to maintain the target queue depth. This was the system deployed as /data/cuzk-synthcap1 in the message immediately preceding our subject ([msg 3518]).

The User's Report: A Calibration Disaster

The user's message ([msg 3537]) contained two log lines that revealed the problem:

2026-03-13T22:34:04.260673Z  INFO cuzk_core::engine: pacer: GPU rate calibrated, 
  switching to PI control ema_gpu_ms="47221" ema_waiting="0.8"
2026-03-13T22:34:34.794700Z  INFO cuzk_core::engine: pacer: status total=10 
  waiting=0 ema_waiting="1.1" ema_gpu_ms="12731" ema_synth_ms="1000" 
  interval_ms=6888 integral="20.00" rate_capped=false synth_rate_known=false 
  bootstrap=false

The first log line shows the pacer exiting bootstrap with ema_gpu_ms="47221"—an estimated GPU processing time of 47 seconds per partition. The user knew this was wrong: "normally the gpu is processing in 1-2s (closer to 1s)." The second log line, 30 seconds later, shows the estimate had only dropped to 12.7 seconds despite several more completions—the exponential moving average (EMA) was dragging down painfully slowly from the bad initial measurement.

The user's diagnosis was prescient: "That for the first burst should be measured from first warmup proof entering the gpu to the last one entering, then that should be used for the initial rate estimate." This observation would prove to be the key insight, though implementing it correctly would require navigating several subtle pitfalls.

The Assistant's Reasoning: Root Cause Analysis

The assistant's response ([msg 3538]) begins with an "Agent Reasoning" section that reveals the internal thought process. This is the most valuable part of the message for understanding how the debugging unfolded.

Step 1: Identifying the contamination source. The assistant correctly identifies that ema_gpu_ms="47221" is measuring the time from pacer creation to the first GPU completion. During this interval, the first partition was being synthesized (20–60s on CPU) and then processed by the GPU (~1s). The 47-second measurement therefore includes the entire synthesis pipeline fill time, not just the GPU processing time. The EMA is "slowly recovering from that bad initial measurement."

Step 2: Tracing the EMA dynamics. The assistant performs a mental simulation of the EMA update: with an alpha of 0.2 (the smoothing factor), each new measurement pulls the estimate 20% of the way from the current value toward the new measurement. Starting at 47,000ms, after 9 measurements of ~1,000ms each, the estimate would be approximately:

EMA_0 = 47000
EMA_1 = 47000 * 0.8 + 1000 * 0.2 = 37800
EMA_2 = 37800 * 0.8 + 1000 * 0.2 = 30440
...
EMA_9 ≈ 12700

This matches the observed ema_gpu_ms="12731" at total=10. The assistant realizes that "even after 9 corrections with alpha=0.2 pulling it toward the ~1s measurements, it still settles around 12-13 seconds, which shows how much that first outlier is weighing on the estimate."

Step 3: Designing the fix. The assistant considers several approaches:

• Skip the first GPU completion entirely—don't use it for rate measurement, just record the timestamp and count. Start measuring from the second completion onward.
• Only measure when waiting > 0—if the GPU queue is empty when a completion occurs, the GPU might have been idle, so that interval measurement is unreliable.
• Track GPU processing time directly from workers—have GPU workers report their actual processing duration rather than inferring it from inter-completion intervals. The assistant initially favors the "skip first completion" approach, reasoning that during the bootstrap burst, 8 items are dispatched nearly simultaneously. The first completion arrives after synthesis + GPU processing (~47s), but subsequent completions arrive at ~1s intervals as the GPU processes the queued partitions. Measuring from the second completion onward captures the true inter-completion spacing. Step 4: Edge case analysis. The assistant considers what happens if there's only one partition—"we'd get just one GPU completion during bootstrap and hang forever." But the assistant correctly notes that bootstrap dispatches 8 items from the work queue, so if there aren't enough items, the dispatcher blocks waiting for more work, naturally delaying when bootstrap_remaining reaches zero. The assistant also considers the case where multiple GPU completions arrive in a single update batch (due to the Notify mechanism only storing one permit). The reasoning is that during the bootstrap wait loop, the dispatcher calls update on every GPU completion notification, so completions arrive individually or in small batches. Even if a batch arrives, the first batch includes pipeline fill time, so it's skipped; subsequent batches provide valid measurements.

The Decision: Skip First, Then Measure

The assistant settles on a two-part fix:

1. Skip the first GPU completion for rate measurement—it includes pipeline fill time. On the first completion, just update tracking variables (last_gpu_event, prev_gpu_count) without setting gpu_rate_known = true.
2. Calibrate from the second completion onward—the inter-completion interval between the second and subsequent completions measures actual GPU processing time. The assistant implements this by modifying the update() method of DispatchPacer. The key logic is: if prev_gpu_count == 0 (no previous completions seen), skip the EMA update but still record the timestamp and count. On subsequent completions, compute the interval and update the EMA normally. The assistant also adds a first_gpu_seen flag to make the logic cleaner, though the reasoning shows some back-and-forth about whether to reuse existing fields like gpu_rate_known or add a dedicated flag.

The Implementation

After the reasoning section, the assistant executes the fix:

1. Reads the current update() method in engine.rs to understand the existing logic.
2. Edits the method to skip the first GPU completion.
3. Runs cargo check to verify compilation.
4. Builds and deploys the binary as /data/cuzk-synthcap2. The code change is concise: in the GPU rate measurement section of update(), the assistant adds a check for self.prev_gpu_count == 0 that skips the EMA update on the first completion, while still updating last_gpu_event and prev_gpu_count so the next measurement starts from the correct reference point.

Assumptions and Blind Spots

The assistant's reasoning contains several assumptions, some correct and some that would prove problematic.

Correct assumption: The first completion includes pipeline fill. This is accurate. The 47-second interval from pacer creation to first GPU completion is dominated by synthesis time, not GPU time. Skipping it is necessary.

Correct assumption: Bootstrap dispatches 8 items. The assistant correctly relies on the bootstrap phase dispatching 8 partitions at 200ms intervals, ensuring multiple GPU completions will occur in rapid succession after the initial pipeline fill.

Incorrect assumption: waiting > 0 is a good proxy for GPU busyness. The assistant considers adding a waiting > 0 guard to only measure GPU rate when the queue has work, but doesn't implement it in this message. However, the reasoning reveals an assumption that queue depth reflects GPU utilization. This assumption is flawed with multiple GPU workers, as the user would later point out ([msg 3541]).

Incorrect assumption: The fix is sufficient. The assistant believes that skipping the first completion will solve the calibration problem. While this addresses the initial 47-second measurement, it does not prevent the deeper collapse loop where the GPU rate estimate drifts upward over time as the pipeline drains and the GPU sits idle between items.

Unstated assumption: The EMA will converge quickly enough. The assistant assumes that after the initial skip, the remaining bootstrap completions (7 more items at ~1s intervals) will produce a good initial estimate, and the EMA will maintain that estimate during steady-state operation. This ignores the possibility that the estimate could drift if subsequent measurements are contaminated by idle time.

The Deeper Problem: The Collapse Loop

What the assistant does not fully appreciate in this message is the self-reinforcing feedback loop that the inaccurate GPU rate measurement creates. The user would report this collapse in the very next message ([msg 3539]):

2026-03-13T22:39:58.647325Z  INFO cuzk_core::engine: pacer: status total=15 
  waiting=0 ema_gpu_ms="43414" ema_synth_ms="53470" interval_ms=48609 
  integral="20.00" rate_capped=true synth_rate_known=true bootstrap=false

The GPU rate estimate had gone back up to 43 seconds, and the synthesis rate estimate had ballooned to 53 seconds. The system was in a death spiral:

1. The pacer dispatches slowly because it thinks the GPU is slow (43s per item).
2. With slow dispatch, few synthesis workers are concurrently active.
3. With few concurrent syntheses, the synthesis completion rate drops (53s per item).
4. The synthesis throughput cap kicks in (rate_capped=true), further clamping the dispatch rate.
5. The GPU queue drains, the GPU sits idle, and the measured inter-completion interval grows.
6. This confirms the pacer's belief that the GPU is slow, reinforcing step 1. This is a classic positive feedback loop in a control system, where the measurement instrument (the GPU rate estimator) is contaminated by the very phenomenon it is trying to measure (the dispatch rate). The pacer's dispatch decisions affect the GPU completion pattern, which in turn affects the pacer's rate estimate, which affects future dispatch decisions. When the estimate is wrong, the system can converge to a stable but pathological equilibrium where the GPU is permanently starved. The assistant's fix—skipping the first completion—addresses the initial calibration but does not prevent this drift. The deeper issue is that the inter-completion interval is not a pure measure of GPU processing speed; it is a measure of GPU throughput under the current dispatch regime. When dispatch is slow, completions are spaced out, and the measured "GPU rate" drops to match the dispatch rate, creating a self-consistent but incorrect estimate.

The Aftermath: A Pivot to Direct Measurement

The user's response to the assistant's fix ([msg 3541]) reveals the next layer of complexity: "tracking gpu processing time isn't that simple because there are two lightly pipelined gpu workers, not just one."

This observation demolishes the waiting > 0 approach that the assistant was considering. With two GPU workers, both can be actively processing even when the queue is empty—each worker pops an item, processes it for ~1s, and during that time the queue might drain to zero. The queue length reflects what's waiting, not what's active. So waiting > 0 is a terrible proxy for GPU busyness.

The assistant pivots to a fundamentally different approach in the subsequent messages ([msg 3542], [msg 3544]): measure GPU processing time directly from the workers. Each GPU worker already measures its own processing duration for logging purposes. By publishing this duration to a shared AtomicU64 accumulator, the pacer can compute the average per-partition GPU processing time directly, without inferring it from inter-completion intervals. The feed-forward interval then becomes ema_gpu_processing / num_workers, which is immune to both pipeline fill contamination and idle-time drift.

This is the correct solution because it measures the quantity of interest—how long the GPU takes to process one partition—at the source, rather than inferring it from a downstream observable (completion timestamps) that is confounded by the dispatch rate itself.

Input Knowledge Required

To fully understand this message, one needs:

1. PI controller theory: Understanding of proportional and integral terms, error correction, feed-forward, and how they interact in a control system.
2. Exponential moving averages: How EMA smoothing works, the effect of alpha on convergence speed, and how outliers persist in the estimate.
3. GPU pipeline architecture: The concept of multiple GPU workers operating in parallel, the distinction between queue depth and active processing, and the time scales of GPU operations (NTT, MSM).
4. Filecoin proof system basics: Understanding that proofs involve CPU-intensive synthesis followed by GPU-intensive proving, and that partitions are independent units of work.
5. The cuzk codebase: Familiarity with engine.rs, the DispatchPacer struct, the update() method, and the bootstrap/PI control phases.
6. The deployment context: The system runs on a machine with ~755 GiB RAM, a GPU, and uses pinned memory pools for zero-copy transfers.

Output Knowledge Created

This message creates several pieces of knowledge:

1. Root cause identification: The 47-second GPU rate calibration is caused by measuring from pacer creation to first GPU completion, which includes synthesis pipeline fill time.
2. EMA contamination analysis: A single bad initial measurement can dominate the EMA for many subsequent updates, taking ~9+ iterations to recover even with alpha=0.2.
3. The skip-first-completion fix: A specific code change to the update() method that ignores the first GPU completion for rate measurement.
4. Awareness of the collapse loop: The assistant recognizes (though does not fully solve) the feedback loop where slow dispatch → empty queue → idle GPU → inflated rate estimate → even slower dispatch.
5. A deployed binary: /data/cuzk-synthcap2 containing the fix, ready for testing.

The Thinking Process: A Window into Debugging

The "Agent Reasoning" section of this message is particularly valuable because it shows the assistant's thought process in real-time, including dead ends, reconsiderations, and incremental refinement.

The reasoning begins with a clear problem statement: "The user is pointing out that the initial GPU rate calibration is way off." The assistant then walks through the chain of causality: the 47-second measurement includes synthesis time, not just GPU time. The EMA is slowly recovering. The bootstrap burst should provide clean measurements after the first completion.

The assistant considers multiple implementation strategies:

• "Option 1 seems cleanest—on the first GPU completion, I'll just update the tracking variables without setting gpu_rate_known = true"
• "I think using a dedicated first_gpu_seen flag would be cleaner than relying on the initial zero value"
• "I should track the time span from the first GPU completion to the Nth one to compute the initial rate estimate" The reasoning also shows the assistant grappling with edge cases:
• "What if there's only one partition—we'd get just one GPU completion during bootstrap and hang forever."
• "What if all the bootstrap completions arrive at once? Then we'd skip all of them and have no data to work with."
• "There's a subtle issue with how Notify handles rapid successive calls—if two GPU completions fire in quick succession before the loop checks again, the second notification might get lost." This kind of edge-case thinking is essential for robust systems programming, and the message captures it in detail.

Lessons Learned

This message, together with its aftermath, teaches several important lessons about building feedback control systems for real hardware:

1. Measure the right thing directly. Inferring GPU processing time from inter-completion timestamps introduces confounding factors (synthesis time, dispatch rate, queue depth). Measuring processing time directly from the GPU workers eliminates these confounds. The principle generalizes: whenever possible, measure the quantity of interest at its source rather than deriving it from downstream observables.

2. Beware of measurement feedback loops. When the controller's output affects the measurement that the controller uses as input, the system can converge to incorrect equilibria. This is the control theory equivalent of the observer effect. The fix is to ensure the measurement is independent of the control action, or to use a separate calibration mechanism that is immune to the feedback.

3. EMA initialization matters. A single bad initial value can dominate an exponential moving average for many iterations. The assistant's simulation shows that even after 9 corrections with alpha=0.2, the estimate is still 12× the true value. This is a reminder that EMA warmup periods should be handled carefully, and that the first few measurements should be validated before being incorporated into the estimate.

4. Multiple workers break simple heuristics. The waiting > 0 heuristic for detecting GPU busyness works for a single worker but fails with two pipelined workers. This is a common pattern in systems programming: assumptions that hold for simple cases break down under parallelism. The solution is to reason about the system at the level of individual workers rather than aggregate observables.

5. Deploy early, measure often. The team deployed the synthcap1 binary, observed the bad calibration in production logs, and iterated rapidly. This deployment-first approach surfaces real-world issues (like the 47-second calibration) that might never appear in unit tests or synthetic benchmarks.

Conclusion

Message [msg 3538] captures a pivotal moment in the development of a high-performance GPU proving pipeline. The assistant correctly identifies that the 47-second GPU rate calibration is contaminated by synthesis pipeline fill time, and implements a fix to skip the first GPU completion. However, the fix addresses only the initial calibration, not the deeper feedback loop that can cause the estimate to drift upward over time.

The message is a testament to the difficulty of measuring hardware performance in production systems. The GPU processing time—a seemingly simple quantity—turns out to be surprisingly hard to measure correctly when the measurement itself is mediated by a feedback controller that affects the system's behavior. The eventual solution—direct measurement from GPU workers—is elegant precisely because it breaks the feedback loop by measuring the quantity of interest at its source.

For anyone building feedback control systems for real hardware, this message and its surrounding context offer a masterclass in the pitfalls of measurement, the importance of understanding feedback dynamics, and the value of iterative deployment and observation. The 47-second phantom was not a bug in the GPU or the synthesis pipeline; it was a bug in how the system observed itself.