The Throttle That Saved the Pipeline: Adding GPU Queue Depth Control to the CuZK Proving Engine

In the midst of a deep optimization campaign targeting GPU underutilization in the CuZK proving pipeline, a single brief message appears: "Now I'll add the max_gpu_queue_depth config field" followed by a successful edit to config.rs. At first glance, this is a trivial change — a new configuration parameter, a default value, a serde annotation. But this message is the culmination of a multi-hour debugging session that had already uncovered and fixed a pinned memory pool budget double-counting bug, diagnosed a PCE caching deadlock, and traced the root cause of GPU idle time to a thundering herd of concurrent synthesis dispatches. The max_gpu_queue_depth field is the architectural keystone of a reactive backpressure mechanism that would transform the pipeline from a chaotic burst dispatcher into a smoothly modulated 1:1 synthesis-to-GPU system.

The Crisis: Budget Starvation and PCE Cache Failure

To understand why this message matters, we must reconstruct the crisis that precipitated it. The team had deployed a pinned memory pool (the "pinned2" build) designed to eliminate costly host-to-device (H2D) transfers by reusing pinned GPU buffers. The logs told a confusing story: PCE (Pre-Compiled Constraint Evaluator) extraction was completing successfully — each extraction producing a 15.8 GiB circuit — but no partition ever used the fast PCE synthesis path. Every single partition fell back to the slow enforce() path, taking 40–50 seconds per partition and consuming massive memory bandwidth.

The assistant's reasoning in [msg 3273] reveals the detective work. By cross-referencing the memory budget logs, the assistant calculated the arithmetic of failure:

The User's Insight: Reactive Backpressure

The user's observation in [msg 3272] — accompanied by a screenshot of the vast-manager UI showing many partitions in the purple "post-synthesis, waiting for GPU" state — crystallized the solution: "Seems like to reduce memory pressure we could implement a mechanism which stops adding new synth jobs once more than N (configurable, let's say 8) partitions are post synth waiting for a gpu (purple state)."

This is a textbook application of reactive backpressure: instead of polling or threshold-based throttling, the system should respond directly to the downstream consumer's capacity. The assistant immediately recognized the elegance of this approach, noting in its reasoning that it "kills multiple birds with one stone: frees budget for PCE, reduces memory pressure, and reduces memory bandwidth contention."

The Implementation Plan

The assistant's reasoning in [msg 3283] lays out a three-step plan:

  1. Add len() to PriorityWorkQueue — The GPU work queue was a priority queue backed by a BTreeMap under a Mutex. It had push() and try_pop() but no way to query its depth. Adding len() required locking the mutex and reading the map's size, a trivial but necessary operation.
  2. Add max_gpu_queue_depth to the pipeline config — This is the message we are examining. The field needed a serde default (the assistant chose 0 to mean "disabled" for backward compatibility) and a doc comment explaining its purpose.
  3. Add the throttle check in the dispatcher loop — The critical architectural decision was where to place the check. The assistant deliberately placed it before the budget acquisition step: "the dispatcher should check GPU queue depth before acquiring budget, so that budget stays free for PCE caching." This ordering is essential — if the check happened after budget acquisition, the budget would already be consumed by the time the throttle engaged, defeating the purpose.

The Config Field Itself

The subject message ([msg 3286]) is deceptively simple:

Now I'll add the max_gpu_queue_depth config field: [edit] /tmp/czk/extern/cuzk/cuzk-core/src/config.rs Edit applied successfully.

We don't see the exact diff, but from the surrounding context we can infer its structure. The PipelineConfig struct (visible in [msg 3285]) already had fields like enabled, synthesis_lookahead, and synthesis_concurrency. The new field would be something like:

/// Maximum number of pre-synthesized partitions waiting for GPU.
/// When the GPU work queue exceeds this depth, synthesis dispatch
/// pauses until GPU workers consume more items. Set to 0 to disable.
#[serde(default = "PipelineConfig::default_max_gpu_queue_depth")]
pub max_gpu_queue_depth: u32,

with a corresponding default function returning 0 (disabled) or 8 (as the user suggested).

Assumptions and Trade-offs

The implementation makes several assumptions worth examining:

Assumption 1: GPU queue depth is a sufficient proxy for memory pressure. The reasoning assumes that by limiting the number of synthesized-but-unproven partitions, the system naturally constrains memory usage. This is reasonable because each synthesized partition holds a SynthesizedJob containing the a/b/c vectors and proof data. However, it doesn't account for memory used during synthesis itself (intermediate allocations, temporary buffers), which could still cause pressure even with a small queue.

Assumption 2: A queue depth of 8 is sufficient to keep the GPU fed. With 2 GPU workers each processing a partition, a queue of 8 provides 4× headroom. This seems generous, but the assistant's reasoning notes that the original system had 78 partitions queued, which was wildly excessive. The actual optimal depth depends on the variance of GPU processing time — if some partitions take much longer than others, a deeper queue provides smoothing.

Assumption 3: The throttle should be poll-based (check on each dispatch loop iteration). The initial implementation used a polling approach: the dispatcher loop checks gpu_work_queue.len() and sleeps if the threshold is exceeded. This was later replaced in [chunk 24.1] with a semaphore-based reactive mechanism that provides exactly 1:1 modulation — one synthesis dispatched per GPU completion. The polling approach was a reasonable first step but exhibited burst behavior when the queue dropped below the threshold.

What This Message Enabled

This single config field unlocked a cascade of improvements. Once deployed as the "pinned3" build, the throttle allowed PCE to be cached (385 GiB budget used, 15 GiB for PCE). The initial batch of 80 partitions was already dispatched before PCE was available, so those slow unpinned partitions had to drain first, but subsequent jobs could use the fast PCE path.

More importantly, this message set the stage for the semaphore-based reactive dispatch in the "pinned4" build ([chunk 24.1]), which replaced the polling throttle with a completion-driven semaphore. The results were dramatic: H2D transfer time dropped from 1,300–12,000 ms to 0 ms, total per-partition GPU time reduced to ~935 ms (pure compute), and the pinned pool reuse ratio improved from 12:474 to 48:24.

Conclusion

The max_gpu_queue_depth config field is a small piece of code that embodies a large architectural insight: that memory pressure in a GPU proving pipeline is not a resource allocation problem but a flow control problem. By connecting synthesis dispatch to GPU consumption through a simple depth threshold, the team transformed a system that was drowning in its own parallelism into one that hummed along at near-constant GPU utilization. The message itself is just two lines of output, but the reasoning behind it — the budget arithmetic, the PCE cache analysis, the recognition of the thundering herd problem — represents hours of careful diagnosis and the kind of systems thinking that separates a working system from an optimal one.