The Critical Error Path: A Single Edit in the GPU Dispatch Refactoring

Message 3373 in this coding session is deceptively simple. Its entire content reads:

[assistant] [edit] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs
Edit applied successfully.

On its face, this is a mundane confirmation — a tool returning a success status for a file edit. But this message represents a pivotal moment in a larger architectural transformation: the replacement of a semaphore-based GPU pipeline throttle with a notification-driven proportional controller. The edit it confirms touches the error paths of the GPU finalizer, and understanding why this specific edit matters requires reconstructing the reasoning, assumptions, and design decisions that led to it.

The Context: Why the Semaphore Had to Go

The session preceding this message was consumed with a fundamental problem in the GPU proving pipeline. The system used a tokio::sync::Semaphore with N permits to limit how many partitions could be in-flight through the synthesis→GPU pipeline at once. The dispatcher would acquire a permit before starting synthesis, and the GPU finalizer would return a permit when a GPU job completed. This created a simple but flawed control loop: it limited the total number of partitions in the pipeline (synthesis + waiting + on-GPU), but it did not maintain a stable queue of already-synthesized partitions waiting for the GPU.

The user identified a critical flaw: when synthesis is fast and GPU proving is slow, the semaphore model allows all N partitions to be synthesized and dumped into the GPU queue at once, creating a burst that overwhelms memory. Conversely, when synthesis is slow, the GPU starves. The desired behavior was a queue-depth target: keep exactly N partitions sitting in the GPU work queue, synthesized and ready, so the GPU always has work without excessive memory pressure from too many concurrent synthesis jobs.

The assistant's reasoning in [msg 3361] reveals an extensive thought process about how to implement this. The assistant considered and rejected several approaches:

Tracking an explicit in-flight counter — increment when dispatching synthesis, decrement when synthesis completes. Rejected because the user specified counting only items already in the GPU queue, not items still being synthesized.
Dispatching the full deficit in one burst — when the GPU finishes, compute N - gpu_work_queue.len() and dispatch that many items at once. Rejected because it could block if insufficient items are available.
Dispatching one item per loop iteration — the chosen approach, where the dispatcher continuously checks the deficit and dispatches one item at a time, naturally self-regulating as items complete synthesis and enter the GPU queue. The core insight, articulated in the assistant's reasoning, was that the dispatcher should be event-driven: "when GPU finishes a job, check how many synthesis tasks are waiting and decide how many to start, with the main goal of keeping exactly N items in the gpu_work_queue at all times."

The Edit That Message 3373 Confirms

Message 3373 is the edit that updates the error paths in the GPU finalizer. To understand why this is significant, we need to trace the full chain of edits that preceded it:

[msg 3364]: Replaced the semaphore creation with a tokio::sync::Notify creation. Instead of Semaphore::new(N), the code now creates a Notify::new() and stores it as gpu_done_notify.
[msg 3365]: Updated the dispatcher loop. Instead of semaphore.acquire().await, the dispatcher now checks gpu_work_queue.len() against the target depth, and if at or above target, waits on gpu_done_notify.notified().
[msg 3368]: Updated the GPU worker clone path, replacing gpu_pipeline_sem_for_worker with gpu_done_notify_for_worker.
[msg 3370]: Updated the finalizer's happy path — the clone of the notification handle passed into the finalizer task.
[msg 3371]: Updated the permit release in the happy path. Instead of fin_pipeline_sem.add_permits(1), the finalizer now calls fin_done_notify.notify_one().
[msg 3373]: The error paths. When a GPU job fails (either gpu_prove_start itself failing before GPU kernels, or a later error), the code previously called add_permits(1) to return the permit to the semaphore. Now it must call notify_one() to wake the dispatcher.

Why Error Paths Are Not Trivial

The error paths in the GPU finalizer are a correctness-critical boundary. If a GPU job fails and the dispatcher is not notified, the system can deadlock: the dispatcher is waiting for a completion signal that will never arrive, while the GPU queue has fewer items than the target depth. The system would hang, unable to dispatch new work.

The assistant's reasoning in [msg 3361] shows careful consideration of the notification semantics:

"if notify_one() fires before the dispatcher calls notified(), that notification gets stored and consumed properly."

This is a key assumption about tokio::sync::Notify's behavior: it stores one pending notification, so even if the dispatcher is currently busy dispatching items (and thus not waiting on the notification), the signal is not lost. However, the assistant also recognized a limitation:

"If two GPUs complete while the dispatcher sleeps, only one permit gets stored, but that's fine because the dispatcher will check the deficit and dispatch both items in subsequent loop iterations."

This is a deliberate trade-off. The Notify mechanism only guarantees that at least one wakeup is delivered, not one per completion. The assistant's reasoning accepts this because the dispatcher re-evaluates the deficit on each loop iteration — even if it misses one notification, it will still dispatch enough work to fill the deficit, just potentially one iteration later.

Assumptions and Potential Pitfalls

The implementation makes several assumptions worth examining:

The deficit calculation is race-free. The dispatcher reads gpu_work_queue.len() and compares it to the target. But between reading the length and dispatching an item, the GPU could consume another item from the queue, changing the deficit. The assistant's loop structure handles this by re-checking on each iteration, but the dispatch count is still based on a stale snapshot.
Synthesis completion is instantaneous relative to the loop. When the dispatcher decides to dispatch one item (sending it to a synthesis worker), that item will not appear in the GPU queue until synthesis finishes. The assistant explicitly considered this: "when the dispatcher checks if gpu_work_queue.len() is below target and dispatches an item, that item won't appear in the queue until synthesis finishes. So the dispatcher keeps looping and dispatching until it runs out of work, budget, or available workers."
The budget and worker pool provide a natural backstop. The assistant reasoned that even if the dispatcher overshoots the target during ramp-up, the budget semaphore and worker pool capacity will eventually throttle it. This is a soft assumption — if synthesis is very fast and the budget is large, the overshoot could be significant.
Error paths are rare. The notification mechanism is lossy (only one pending notification is stored). The assistant's reasoning accepts this because in normal operation, completions are frequent enough that the dispatcher will wake up anyway. But if errors cluster — multiple GPU jobs failing in quick succession — only one notification is stored, and the dispatcher might not wake up enough times to refill the queue promptly.

The Broader Design Trajectory

This edit is part of a P-controller implementation — a proportional controller where the "control signal" is the number of items dispatched per GPU completion event, and the "setpoint" is the target queue depth. The assistant's reasoning reveals an awareness that this is an iterative process:

"The overarching theme of this chunk is the iterative refinement of a complex control system for GPU pipeline scheduling. The team moved from a simple reactive semaphore to a proportional controller, then to a dampened proportional controller."

Indeed, the chunk summary tells us that this P-controller deployment (cuzk-pctrl1) proved too aggressive, instantly filling all allocation slots. The user then requested a dampening factor, capping the burst size. Later, the system evolved to a PI-controlled pacer with EMA feed-forward and a synthesis throughput cap.

Message 3373, for all its apparent simplicity, is a critical piece of that evolution. It ensures that when things go wrong — when a GPU job fails — the control system still gets the feedback it needs to continue operating. Without correct error path signaling, the entire dispatch mechanism would deadlock on the first failure. The edit is small, but the correctness burden it carries is large.