The Borrow That Broke the Build: When Rust's Ownership Model Caught a Priority Queue Refactoring

"error[E0382]: borrow of moved value: gpu_work_queue"

In the middle of a substantial refactoring of the cuzk proving engine's partition scheduling system, the assistant ran a cargo check and was met with a single, sharp compilation error. Message [msg 2907] is that error output — a brief, almost mundane snippet of terminal text that nonetheless captures a critical moment in the development of a high-performance CUDA-based zero-knowledge proving daemon. This article unpacks that message in depth: why it was produced, what it reveals about the refactoring underway, the assumptions that led to it, and the Rust ownership semantics that caught it.

The Message: A Compilation Error in Context

The subject message is the output of a cargo check command:

[assistant] [bash] cd /tmp/czk/extern/cuzk && cargo check --features cuda-supraseal 2>&1 | tail -30
    Checking cuzk-core v0.1.0 (/tmp/czk/extern/cuzk/cuzk-core)
warning: unused import: `filecoin_proofs_api::RegisteredSealProof`
  --> cuzk-core/src/prover.rs:24:5
   |
24 | use filecoin_proofs_api::RegisteredSealProof;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |     = note: `#[warn(unused_imports)]` on by default

error[E0382]: borrow of moved value: `gpu_work_queue`
    --> cuzk-core/src/engine.rs:2404:38
     |
1135 |             let gpu_work_queue: Arc<PriorityWorkQueue<Synthesize...

The message is terse — a single warning about an unused import, followed by the error that matters. The error code E0382 is one of Rust's most familiar: a borrow of a moved value. But the location tells a deeper story. Line 1135 is where gpu_work_queue is declared, and line 2404 is where it is used after being moved. The compiler helpfully points out the declaration site, but the real issue lies in the hundreds of lines between them — the assistant's large-scale refactoring of the engine's partition dispatch and GPU worker infrastructure.

The Refactoring That Led Here

To understand message [msg 2907], we must understand what came before it. The assistant had been implementing a priority queue-based ordering system to replace the existing channel-based dispatch for partition synthesis work. The old system used tokio::sync::mpsc channels (synth_tx, synth_rx, partition_work_tx, partition_work_rx) to move synthesized jobs from the synthesis workers to the GPU workers. But this architecture had a fundamental problem: all partitions from all jobs were dispatched as independent tokio tasks racing on a Notify-based budget acquire, causing thundering herd wakeups and random partition selection across pipelines. This meant that instead of completing jobs sequentially (finishing job 1's partitions before starting job 2's), all pipelines stalled together in a chaotic free-for-all.

The fix was to introduce a PriorityWorkQueue — a custom data structure backed by a BTreeMap that orders work items by a (job_seq, partition_idx) tuple, ensuring FIFO ordering across jobs. Each PartitionWorkItem and SynthesizedJob was given a job_seq field, an atomic counter next_job_seq was added to sequence jobs, and the old channels were replaced by two priority queues: synth_work_queue (for work flowing from the dispatcher to synthesis workers) and gpu_work_queue (for work flowing from synthesis workers to GPU workers).

This was a deep, structural change touching dozens of locations across a ~2500-line file. The assistant made edit after edit — adding the PriorityWorkQueue struct, updating the dispatcher, modifying the synthesis worker loop, changing the GPU worker loop, updating dispatch_batch and process_batch signatures, and replacing every synth_tx.send() and synth_rx.recv() call. By message [msg 2905], the assistant believed all references to the old channels were gone. The next logical step was to verify the code compiles.

The Error: A Classic Ownership Violation

The error itself is straightforward in Rust terms:

error[E0382]: borrow of moved value: `gpu_work_queue`
    --> cuzk-core/src/engine.rs:2404:38
     |
1135 |             let gpu_work_queue: Arc<PriorityWorkQueue<Synthesize...

At line 1135, gpu_work_queue is declared as an Arc<PriorityWorkQueue<SynthesizedJob>>. An Arc (atomic reference count) is Rust's thread-safe shared ownership primitive — it allows multiple threads or tasks to hold clones of the same pointer, with the reference count ensuring the value is freed only when all references are dropped. The key word is clone: to share an Arc, you must explicitly clone it (which increments the reference count), not move it.

What happened is that the dispatcher closure — defined as an async move block — captured gpu_work_queue by value. In Rust, a move closure takes ownership of everything it captures. When the closure is spawned as a tokio task, the Arc<PriorityWorkQueue<...>> is moved into the closure's environment. After that move, the original variable gpu_work_queue is no longer valid — the ownership has been transferred. But the GPU workers, which are spawned later in the same function, also need access to the same queue. They try to use gpu_work_queue after it has been moved, and the compiler correctly rejects this.

The Assumption That Failed

The assistant's mistake was a subtle but common one when working with Arc in Rust. The assistant knew that gpu_work_queue was an Arc, and Arc is clonable. But the assistant assumed that the async move block would somehow "share" the Arc rather than consume it. In many languages with garbage collection or reference semantics, this would work fine — the variable would remain usable after being passed to a closure. But Rust's ownership model is stricter: moving a value into a closure means that value is gone from the original scope.

The assistant had even verified earlier (in [msg 2903]) that synth_work_queue, gpu_work_queue, and next_job_seq were "used in the dispatcher closure body" and "captured automatically by the async move block." This verification was correct — they were captured — but the assistant didn't consider the consequence: once captured, they were moved and no longer available for the GPU worker section that follows.

Why the Error Was Inevitable (and Valuable)

This error is a perfect example of why Rust's ownership model is so valuable for concurrent systems. The assistant was refactoring a highly concurrent proving engine where multiple tokio tasks (dispatcher, synthesis workers, GPU workers) all need access to shared work queues. In a language without ownership tracking, this kind of mistake could easily slip through and manifest as a use-after-move, a data race, or a silent logical error at runtime. Rust caught it at compile time, with a clear error message pointing to both the declaration and the problematic use.

The error also reveals the architecture of the refactoring. The gpu_work_queue is shared between:

1. The dispatcher task (which pushes synthesized jobs into the queue)
2. The GPU worker tasks (which pop jobs from the queue for proving) Both need concurrent access. The Arc provides the shared ownership, but each concurrent user needs its own clone of the Arc. The dispatcher closure should have received a clone, leaving the original Arc for the GPU workers.

Input Knowledge Required

To fully understand this message, one needs:

• Rust ownership and borrowing: The distinction between moving and cloning, how Arc provides shared ownership, and how move closures capture variables.
• The cuzk proving engine architecture: Understanding that there are synthesis workers (CPU-bound circuit synthesis) and GPU workers (CUDA proving), connected by work queues.
• The refactoring context: That the assistant was replacing unordered channel-based dispatch with ordered priority queue dispatch to fix a thundering-herd scheduling problem.
• The PriorityWorkQueue data structure: A custom BTreeMap-backed queue that orders work by (job_seq, partition_idx) for FIFO job completion.

Output Knowledge Created

This message produces several pieces of knowledge:

• A concrete compilation error that must be fixed before the refactoring can proceed.
• Evidence that the refactoring is structurally sound — only one error emerged from a large set of changes, suggesting the rest of the ownership and type relationships are correct.
• A diagnostic signal pointing to the exact location where the ownership model was violated, enabling a targeted fix.

The Thinking Process Visible in the Reasoning

The assistant's reasoning leading up to this message shows a methodical, incremental approach to a complex refactoring. In [msg 2866], the assistant laid out a detailed todo list with priorities. Each subsequent message applied a single focused edit: add the struct, add the fields, replace channel creation, update the synthesis worker, update dispatch functions, update GPU worker, fix shutdown handling, and so on.

In [msg 2891], the assistant demonstrated careful reasoning about shutdown handling in the GPU worker loop, realizing that a continue in an inner loop wouldn't break the outer loop as intended. This shows the assistant thinking through control flow implications of the new priority queue API vs. the old channel API.

In [msg 2903], the assistant explicitly checked whether the queues were captured by the dispatcher closure, reading the relevant code and confirming they were. This was the moment where the error could have been prevented — if the assistant had also considered that the capture would move the Arcs, the clone could have been added preemptively. But the assistant's focus was on whether the variables were accessible to the closure, not on the ownership consequences for later code.

The Resolution

The fix was straightforward once the error was understood. In message [msg 2908], the assistant immediately recognized the problem: "The gpu_work_queue is moved into the dispatcher closure, but the GPU workers (spawned later) also need it. I need to clone it before the dispatcher closure captures it." The assistant read the relevant code section and, in message [msg 2909], applied the fix: cloning both synth_work_queue and gpu_work_queue before the dispatcher closure, so the originals remain available for the GPU workers.

Broader Significance

This single error message encapsulates several important lessons for systems programming in Rust:

1. Arc does not automatically share. An Arc is still a value that can be moved. You must explicitly clone it to create additional references. This is a common pitfall even for experienced Rust developers.
2. move closures are greedy. An async move block captures everything it touches by value. When working with shared resources in concurrent Rust, it's often better to clone into the closure explicitly rather than relying on automatic capture.
3. Compile-time checking catches concurrency bugs. In a language like C++, this mistake would compile fine and likely cause a use-after-move at runtime (or worse, silently work because the moved-from Arc still points to valid memory until the reference count drops). Rust's ownership model transforms potential runtime bugs into compile-time errors.
4. Large refactorings benefit from frequent compilation checks. The assistant ran cargo check after completing the bulk of the changes, catching the error early. Had the assistant continued making changes without checking, the error could have been buried under more complex issues. Message [msg 2907] is, on its surface, just a compiler error. But in the context of the cuzk proving engine's development, it represents a critical quality gate — Rust's ownership model enforcing the correct sharing pattern for a concurrent work queue, ensuring that the priority queue refactoring would be memory-safe and race-free before it ever ran on a GPU.