The Pivot Point: Wiring the Dual-Worker GPU Interlock into the Engine

In the long arc of optimizing the cuzk SNARK proving engine for Filecoin's Curio node, there are messages that implement, messages that debug, and messages that design. Then there are messages like msg 2183 — short, almost mundane on the surface, but structurally pivotal. This message is the moment when the assistant transitions from modifying the low-level C++ and FFI layers to integrating those changes into the engine's async task orchestration. It is the bridge between "we have the tools" and "we use the tools."

The Message Itself

The assistant writes:

Now I need to use SendableGpuMutex from bellperson in the engine. The engine needs to: 1. Create one C++ mutex per GPU using supraseal_c2::alloc_gpu_mutex() 2. Wrap it in SendableGpuMutex for Send/Sync 3. Spawn gpu_workers_per_device workers per GPU, each receiving the same mutex pointer 4. Pass the mutex pointer to gpu_prove()

>

Let me update the engine. First, the GPU worker spawn section — I need to create per-GPU mutexes and spawn multiple workers: [read] /home/theuser/curio/extern/cuzk/cuzk-core/src/engine.rs

Four bullet points and a file read. Yet this message encapsulates the entire Phase 8 architecture in microcosm.

Why This Message Was Written

To understand why this message exists, we must trace the chain of reasoning that led to it. The assistant had just completed two critical steps in messages [msg 2181] and [msg 2182]:

  1. C++ helper functions: Added create_gpu_mutex() and destroy_gpu_mutex() to groth16_cuda.cu — tiny C++ functions that allocate and deallocate a std::mutex on the heap, returning a raw pointer. This was necessary because Rust's std::sync::Mutex has a completely different memory layout from C++'s std::mutex; you cannot cast one to the other. The assistant correctly recognized this incompatibility and chose the cleanest solution: allocate the C++ mutex from C++ code.
  2. FFI declarations: Added the extern declarations and Rust wrapper functions in supraseal-c2/src/lib.rs to expose alloc_gpu_mutex() and destroy_gpu_mutex() to the Rust side. With those pieces in place, the assistant now faces the question: how do we actually use this mutex in the engine? The engine is an async Tokio-based system that spawns GPU worker tasks. Those workers need to share a per-GPU mutex so that only one worker at a time runs CUDA kernels on that GPU. But the mutex is a raw *mut c_void pointer — not Send, not Sync, not safe to share across async task boundaries. This is the problem that msg 2183 exists to solve. The assistant must design the Rust-side integration that takes the raw C++ mutex pointer and makes it usable in the async engine.

The Four-Step Plan: A Micro-Architecture

The four bullet points in this message are not a random to-do list. They form a precise architectural sequence, each step building on the previous:

Step 1: Create one C++ mutex per GPU. The assistant plans to call supraseal_c2::alloc_gpu_mutex() once per physical GPU device. This is a deliberate design choice: each GPU gets its own mutex, so multi-GPU configurations (if supported in the future) would allow independent GPU kernel execution per device. In the current single-GPU setup (RTX 5070 Ti), this means exactly one mutex.

Step 2: Wrap it in SendableGpuMutex. This is the crucial safety layer. The SendableGpuMutex type (defined in bellperson's supraseal.rs) is a newtype wrapper around *mut c_void that implements Send and Sync. Without this wrapper, the raw pointer cannot be moved across the .await boundary in Tokio tasks — the compiler would reject it. The assistant had already created this type and re-exported it from bellperson's public API in earlier messages ([msg 2166]).

Step 3: Spawn gpu_workers_per_device workers per GPU. This is the heart of Phase 8's dual-worker interlock. Instead of one worker per GPU (the Phase 7 model), the engine will spawn multiple workers (default 2) that share the same GPU mutex. These workers both pull from the same synth_rx channel, competing for partitions. The mutex ensures that only one worker runs CUDA kernels at a time, while the other does CPU preprocessing — the fundamental overlap that eliminates GPU idle gaps.

Step 4: Pass the mutex pointer to gpu_prove(). Each worker passes the shared mutex pointer through the call chain: gpu_prove() → prove_from_assignments() → C++ FFI → generate_groth16_proofs_c(). The C++ function acquires the mutex only around the CUDA kernel region, not the entire function.

Assumptions Embedded in This Plan

The assistant makes several assumptions in this message, some explicit and some implicit:

That SendableGpuMutex is properly accessible from the engine. The assistant assumes that the re-export chain works: bellperson::groth16::SendableGpuMutex is public, and the engine can import it. This was set up in [msg 2166] but hasn't been verified by a compilation check yet.

That supraseal_c2::alloc_gpu_mutex() is the right API. The assistant assumes that allocating a C++ std::mutex on the heap and passing its raw pointer through FFI is safe. This is correct in principle — the C++ mutex's lifetime is managed manually (create before worker spawn, destroy after worker shutdown) — but any leak or double-free would be catastrophic.

That multiple workers sharing the same mutex will interleave correctly. The assistant assumes that the Tokio runtime will schedule the two workers such that one is doing CPU work while the other holds the GPU mutex. If both workers happen to reach the mutex at the same time (e.g., both finishing CPU work simultaneously), one will block — but this is the intended behavior. The assumption is that CPU work duration (~1.3s) is less than GPU kernel duration (~2.1s), so the natural timing creates overlap.

That the current GPU worker spawn section in engine.rs is the right place to modify. The assistant reads the file to understand the existing structure before making changes. This is a reasonable assumption — the engine's worker spawn loop is the logical place to add per-GPU mutex creation and multi-worker spawning.

Input Knowledge Required

To understand this message, one must already know:

Output Knowledge Created

This message produces a clear plan of action for the engine changes. It establishes:

  1. The integration pattern: How the raw C++ mutex pointer flows from creation in the engine, through the async task boundary (via SendableGpuMutex), into the GPU worker, and down to gpu_prove().
  2. The multi-worker topology: The engine will spawn gpu_workers_per_device workers per GPU, all sharing the same mutex and partition channel.
  3. The next concrete step: Read the current engine.rs to understand the existing worker spawn loop before making modifications. The message also implicitly creates a verification point: after implementing these changes, the assistant will need to compile and test to validate that the mutex sharing works correctly across async task boundaries.

The Thinking Process

The assistant's reasoning in this message is visible in the structure of the four bullet points. Notice the progression:

  1. "Create one C++ mutex per GPU" — the assistant immediately thinks in terms of per-device resources, not a global lock. This reflects the multi-GPU awareness from the Phase 8 design doc.
  2. "Wrap it in SendableGpuMutex for Send/Sync" — the assistant anticipates the Rust compiler's safety requirements. The raw pointer from alloc_gpu_mutex() is *mut c_void, which is neither Send nor Sync. Tokio tasks require Send because the task future may be moved between threads. The assistant knows this and plans to use the wrapper type.
  3. "Spawn gpu_workers_per_device workers per GPU, each receiving the same mutex pointer" — the key insight of Phase 8: multiple workers share one mutex, not one worker per mutex. The word "same" is critical.
  4. "Pass the mutex pointer to gpu_prove()" — the final step in the chain, connecting the engine's resource to the C++ function that actually uses it. The assistant then reads the engine file to understand the current structure. This is a deliberate, methodical approach: understand before modifying. The message ends with a file read, not an edit — the assistant is gathering information before acting.

Conclusion

Msg 2183 is a planning message that bridges two worlds: the low-level C++/FFI changes that created the mutex mechanism, and the high-level engine orchestration that will use it. It is short — four bullet points and a file read — but it encodes the entire Phase 8 integration strategy. The assistant's thinking is precise and architectural: per-GPU resource allocation, Rust safety constraints, async task topology, and the call chain from engine to CUDA kernel. This message is the pivot point where a mechanism becomes a system.