The Throttle That Saved the Pipeline: Deploying GPU Queue Depth Control in CuZK

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proving, memory pressure is the silent killer of performance. When a proving pipeline synthesizes partitions faster than the GPU can consume them, the system doesn't just slow down—it can catastrophically fail to cache critical pre-compiled circuits, locking the entire pipeline into a slow path that compounds the original problem. This article examines a pivotal moment in the optimization of the CuZK proving engine: message <msg id=3294>, where the assistant confirmed that a newly implemented GPU queue depth throttle was ready for deployment and announced the plan to build, deploy, and configure it with a threshold of 8.

This message, though brief, represents the culmination of a multi-step debugging and implementation effort that transformed the system's memory management from a firehose of uncontrolled synthesis dispatch into a disciplined, backpressure-regulated pipeline. To understand its significance, we must trace the chain of reasoning that led to this point.

The Crisis: PCE Caching Starved by Budget Exhaustion

The story begins with the pinned memory pool deployment (segments 22–24 of the conversation). The team had identified that GPU underutilization was caused by excessive host-to-device (H2D) transfer times for the a/b/c vectors used in NTT kernels. The solution was a zero-copy pinned memory pool (PinnedPool) that would allow the GPU to directly access host memory without explicit transfers.

However, the initial deployment (pinned2) revealed a critical failure: the Pre-Compiled Constraint Evaluator (PCE) cache was never populated. The insert_blocking method in the PCE cache calls budget.try_acquire(15.8 GiB) in a retry loop, but the budget was completely saturated by concurrent synthesis partitions. The assistant's log analysis in <msg id=3270> confirmed: "No PCE cached message ever appears." The insert_blocking loop was sleeping for one second, retrying, and failing forever.

The budget arithmetic was brutal. With 5 jobs × 16 partitions each, the system had 80 partition reservations consuming roughly 362 GiB of the 400 GiB budget. After SRS loading consumed 32 GiB, only 5 GiB remained—far short of the 15.8 GiB needed for a single PCE entry. Every synthesis completed via the slow enforce() path instead of the fast PCE path, and the pinned pool itself was silently falling back to heap allocations because PinnedAbcBuffers::checkout() called budget.try_acquire() for memory that was already accounted for in per-partition budget reservations.

The User's Insight: Throttle by GPU Queue Depth

The breakthrough came from the user in <msg id=3272>. Observing a vast-manager UI screenshot showing many partitions in the "purple" state (post-synthesis, waiting for GPU), they proposed a simple but powerful mechanism: stop adding new synthesis jobs once more than N partitions are waiting for GPU. The user suggested a configurable threshold, defaulting to 8.

This observation was astute because it identified the root cause not as a lack of memory per se, but as a scheduling problem. The synthesis dispatcher was producing work far faster than the two GPU workers could consume it, creating a backlog of synthesized partitions that consumed budget without contributing to throughput. By throttling synthesis dispatch based on GPU queue depth, the system would naturally constrain memory usage and free budget for PCE caching.

The Assistant's Implementation Sprint

The assistant immediately recognized the brilliance of this suggestion. In <msg id=3273>, the reasoning laid out four benefits:

  1. Reduced memory pressure by not having 60–80 partitions' worth of synthesized data sitting in memory
  2. Free budget for PCE caching, which was currently constrained at 5 GiB available
  3. Reduced CPU memory bandwidth contention with fewer concurrent synthesis threads
  4. Sufficient GPU feeding since 2 GPU workers only needed a queue of 8 to stay busy The implementation required three coordinated changes, executed across messages <msg id=3283> through <msg id=3289>:
  5. Add len() to PriorityWorkQueue — The GPU work queue was a custom PriorityWorkQueue wrapping a BTreeMap behind a Mutex. It had push() and try_pop() but no way to query its depth. The assistant added a len() method that simply locked the mutex and returned the map's size.
  6. Add max_gpu_queue_depth to PipelineConfig — The configuration struct needed a new field with a default of 0 (disabled), making the throttle opt-in. The assistant added this with serde defaults and appropriate documentation.
  7. Add throttle check in the dispatcher loop — The critical architectural decision was where to place the throttle. The dispatcher loop (around line 1209 in engine.rs) pops from the synthesis work queue, acquires budget, then sends to workers. The assistant placed the throttle check before budget acquisition, so that when the GPU queue is full, the dispatcher sleeps without consuming budget. This is the key insight: by blocking before budget acquisition, the throttle naturally frees budget for PCE caching. The implementation used a simple polling approach: when gpu_work_queue.len() >= max_gpu_queue_depth, the dispatcher calls tokio::time::sleep(Duration::from_millis(100)) and retries. This is not elegant—it introduces a 100ms polling delay—but it is correct and simple.

The Subject Message: Ready for Deployment

Message <msg id=3294> is the transition point from implementation to deployment. The assistant writes:

No max_gpu_queue_depth set yet — it will default to 0 (disabled). Let me build, deploy, then update the config with max_gpu_queue_depth = 8:

This message serves several functions:

Confirmation of status. The assistant had just checked the remote config (cat /tmp/cuzk-memtest-config.toml in <msg id=3293>) and confirmed that the [pipeline] section only had enabled = true and synthesis_concurrency = 4. The new max_gpu_queue_depth field was absent, meaning it would default to 0—effectively disabled. This was expected and correct; the code change had not yet been deployed.

Declaration of intent. The assistant announces the next steps: build the binary, deploy it to the remote machine, and update the config file to set max_gpu_queue_depth = 8. This is a clear action plan that the user can review and approve.

Implicit request for confirmation. By stating the plan openly, the assistant invites the user to object or suggest modifications before proceeding. The todo list at the bottom shows that the implementation tasks are marked complete, and the remaining work is operational.

Decisions and Assumptions in This Message

Several important decisions and assumptions are embedded in this brief message:

The choice of default value 0 (disabled). The assistant chose to make the throttle opt-in rather than always-on. This is a conservative design choice: existing deployments that upgrade to the new binary will not change behavior unless they explicitly add max_gpu_queue_depth to their config. This avoids surprises but means the throttle must be manually enabled.

The choice of threshold 8. The user suggested 8 as a starting point. With 2 GPU workers and 2 GPUs (4 total workers), a queue depth of 8 means each GPU worker would have at most 2 partitions waiting. This is sufficient to absorb scheduling jitter without creating excessive memory pressure. The assumption is that 8 is a safe value that provides headroom without overwhelming the budget.

The assumption that polling is acceptable. The throttle uses tokio::time::sleep(Duration::from_millis(100)) in a polling loop. This means the dispatcher thread wakes up every 100ms to check the GPU queue depth. In the worst case, a GPU worker could finish and the dispatcher could take up to 100ms to notice. This is probably acceptable given that synthesis takes seconds per partition, but it introduces a small latency bubble.

The assumption that the dispatcher is the right place for the throttle. The assistant considered placing the throttle at the synthesis work queue pop or at the budget acquisition point. Choosing to place it before budget acquisition was a deliberate decision to ensure that budget remains free for PCE caching even when synthesis is paused. This is architecturally sound.

Input Knowledge Required

To understand this message, one needs:

Output Knowledge Created

This message creates:

The Broader Significance

This message sits at a critical inflection point in the optimization effort. The pinned memory pool had solved the H2D transfer problem in principle, but budget double-counting and uncontrolled synthesis dispatch prevented it from working in practice. The GPU queue depth throttle is the second half of the solution: it constrains memory usage so that the pinned pool and PCE cache can function correctly.

The assistant's reasoning in <msg id=3273> correctly identified that "throttling synthesis dispatch based on GPU queue depth kills multiple birds with one stone." The throttle:

  1. Prevents budget exhaustion by limiting concurrent synthesized partitions
  2. Enables PCE caching by freeing budget headroom
  3. Reduces memory bandwidth contention by limiting concurrent synthesis threads
  4. Creates natural backpressure between synthesis and GPU proving What makes this approach elegant is that it addresses the symptom (too many partitions waiting for GPU) by fixing the cause (uncontrolled synthesis dispatch), and in doing so, it also fixes the secondary problem (PCE cache starvation). The throttle is a single knob that controls a cascade of positive effects.

Conclusion

Message <msg id=3294> is a quiet moment of transition in a high-intensity debugging session. The hard work of diagnosis and implementation is done; what remains is the operational work of deployment and validation. The assistant's brief announcement—"Let me build, deploy, then update the config"—belies the depth of reasoning that led to this point: the budget arithmetic that revealed PCE starvation, the architectural analysis that identified the dispatcher as the right control point, and the careful placement of the throttle before budget acquisition to maximize its effect.

The GPU queue depth throttle would prove to be a transformative change. In the subsequent chunk (chunk 1 of segment 24), the assistant would refine the polling-based throttle into a semaphore-based reactive dispatch mechanism that eliminated burst dispatches entirely, reducing H2D transfer times from 1,300–12,000 ms to 0 ms and achieving near-constant GPU utilization. But that refinement built on the foundation laid in this message: the recognition that controlling the flow of work between synthesis and GPU is the key to memory stability.

In the end, the throttle wasn't just about limiting queue depth—it was about creating the conditions for the pinned memory pool and PCE cache to work as designed. Sometimes the most important optimization isn't adding more resources, but controlling how they flow.