The Keystone of Phase 8: Wiring Dual GPU Workers into the cuzk Proving Engine

In the span of a single, deceptively short message, an AI assistant executing a complex optimization pipeline for Filecoin's SNARK proving engine performed the final integration that brought an entire architectural redesign to life. The message, message index 2184 in a long-running coding session, reads:

[assistant] I need to update the worker spawn loop to: 1. Create one C++ mutex per GPU 2. Spawn gpu_workers_per_device workers per GPU (default 2), each getting the same mutex pointer 3. Update the worker IDs to be unique across all workers [edit] /home/theuser/curio/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

This is the moment where Phase 8: Dual-Worker GPU Interlock — a design that had been specified, debated, and meticulously plumbed through five layers of code — finally took its operational form. The edit to engine.rs was the keystone: the single point where all the preceding work coalesced into a running system capable of keeping a GPU continuously busy by overlapping two workers' CPU and GPU phases.

The Long Road to This Moment

To understand why this message exists, one must trace the path that led to it. The session began with a fundamental problem: the cuzk SNARK proving engine for Filecoin's Curio node was leaving its GPU idle for significant stretches of time. Phase 7 had introduced a per-partition pipeline that dispatched individual proof partitions to the GPU, but benchmarks revealed GPU efficiency of only 64.3%. Between each partition's GPU work, the single worker thread spent 10–200ms on CPU bookkeeping — proof assembly, mutex acquisition, malloc_trim(0) calls, and channel operations — while the GPU sat idle.

The Phase 8 design document (c2-optimization-proposal-8.md) proposed a radical solution: instead of one worker per GPU, spawn two workers per GPU that interleave their execution. While Worker A runs CUDA kernels on the GPU, Worker B performs CPU preprocessing for the next partition. When A finishes GPU work, B immediately takes over the GPU while A does its CPU post-processing. The key enabler: narrowing the C++ static mutex that previously covered the entire generate_groth16_proofs_c function to cover only the CUDA kernel region, allowing CPU work to proceed in parallel.

The assistant had spent the preceding messages implementing this vision layer by layer. The C++ CUDA kernel in groth16_cuda.cu was refactored to remove the static mutex and accept a passed-in mutex pointer with narrowed scope. FFI plumbing threaded the mutex through supraseal-c2/src/lib.rs. The bellperson library's prove_from_assignments function was updated to accept and forward the mutex pointer. The gpu_prove function in pipeline.rs was modified to carry the mutex through to the FFI call. By message 2183, all the plumbing was in place — but nothing would actually use it until the engine spawned multiple workers.

The Three Decisions in the Message

The message's three numbered items encapsulate the critical design choices for the dual-worker architecture.

"Create one C++ mutex per GPU" — This decision reflects a deep understanding of the C++/Rust FFI boundary. The assistant had earlier considered using a Rust std::sync::Mutex<()> and passing its raw pointer to C++, but caught a critical mistake: Rust's std::sync::Mutex has a completely different memory layout than C++'s std::mutex. Passing a Rust mutex pointer to C++ code expecting a std::mutex* would cause undefined behavior. The correct approach, which the assistant implemented in messages 2181–2182, was to add two C++ helper functions — create_gpu_mutex and destroy_gpu_mutex — that allocate and free a genuine C++ std::mutex on the heap. The Rust side calls these via FFI and wraps the resulting pointer in a SendableGpuMutex type that implements Send and Sync for safe sharing across threads. This decision demonstrates careful cross-language memory safety reasoning.

"Spawn gpu_workers_per_device workers per GPU (default 2), each getting the same mutex pointer" — This is the heart of the dual-worker interlock. Both workers share the same C++ mutex, which acts as a hardware-level GPU scheduler: only one worker can hold the mutex at a time, and the mutex is only held during the CUDA kernel region (NTT+MSM, batch additions, tail MSMs). CPU preprocessing and b_g2_msm run outside the lock, so Worker B can prepare its partition while Worker A occupies the GPU. The default of 2 workers is a deliberate choice based on the design document's analysis: the CPU work per partition (~1.3s of prep + b_g2_msm + epilogue) is comfortably less than the CUDA kernel time (~2.1s), so two workers suffice to keep the GPU fully utilized. Adding a third worker would increase CPU contention without improving GPU throughput.

"Update the worker IDs to be unique across all workers" — This seemingly minor detail reveals an important systems-thinking consideration. Previously, each GPU had exactly one worker, so worker_id could correspond directly to gpu_ordinal. With multiple workers per GPU, worker IDs must be globally unique for logging, tracing, and error reporting. The assistant recognized that the existing worker_id field, used in timeline events and diagnostic output, would become ambiguous if two workers shared the same ID. This change ensures that each spawned task has a distinct identity, enabling accurate performance analysis of the dual-worker pattern.

Assumptions and Their Risks

The message makes several implicit assumptions. First, it assumes that the C++ mutex helper functions have been correctly implemented and that the mutex pointer is safe to share across multiple Rust threads. This is reasonable given the SendableGpuMutex wrapper, but the actual safety depends on the C++ std::mutex implementation being compatible with the pthreads or C++ runtime on the target system (AMD Ryzen Threadripper PRO 7995WX running Linux).

Second, it assumes that two workers consuming from the same synth_rx channel will naturally interleave — that the channel's fairness properties won't cause one worker to starve while the other hogs partitions. The design document's analysis suggests this is safe because the mutex serializes GPU access, and the CPU work is short enough that whichever worker finishes its CPU phase first will be waiting for the mutex when the other worker releases it.

Third, it assumes that gpu_workers_per_device = 2 is the optimal default. The design document's timeline analysis shows that CPU work per partition (~1.3s) is less than CUDA time (~2.1s), so two workers should achieve near-100% GPU utilization. But this ratio depends on the specific hardware (RTX 5070 Ti GPU, 96-core Zen4 CPU) and may differ on other configurations.

The Broader Architecture

This message represents the final integration point of a multi-layered change spanning seven files and approximately 195 lines of code. The architecture follows a clean dependency chain: the C++ CUDA kernel accepts a mutex pointer and narrows its lock scope; the FFI layer in supraseal-c2 passes the pointer through; bellperson's prove_from_assignments forwards it; pipeline.rs's gpu_prove accepts it; and finally, the engine creates the mutexes and spawns the workers that share them. Each layer is a thin wrapper that adds no complexity beyond passing the pointer through.

The beauty of this design is that the engine itself needs no knowledge of the C++ mutex internals. It calls supraseal_c2::alloc_gpu_mutex() to obtain an opaque pointer, wraps it in SendableGpuMutex for thread safety, and passes it to gpu_prove(). The C++ code does the actual synchronization. This clean separation of concerns — the engine manages worker topology, the C++ code manages GPU serialization — is a hallmark of well-designed FFI boundaries.

What This Message Achieved

The edit to engine.rs was the final step before benchmarking. After this message, the assistant would proceed to build the daemon, run benchmarks, and discover that the Phase 8 dual-worker interlock achieved 100.0% GPU efficiency in single-proof mode and 13–17% throughput improvement in multi-proof mode (44.0s/proof vs 50.7s for Phase 7). The message itself is unassuming — a three-item planning statement and a single edit — but it represents the culmination of hours of careful design, cross-language plumbing, and systems-level reasoning. It is the moment when architecture becomes execution.