The Final Connection: Plumbing the GPU Mutex Pointer Through the cuzk Proving Engine

Message: [assistant] Now update the gpu_prove call to pass the mutex pointer: [edit] /home/theuser/curio/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

On its surface, this message is unremarkable — a single edit in a single file, accompanied by a brief one-line comment. But in the architecture of the Phase 8 Dual-Worker GPU Interlock for the cuzk SNARK proving engine, this edit represents the final, critical connection in a chain of changes spanning seven files across C++, Rust FFI, the bellperson library, the pipeline abstraction layer, and the engine itself. It is the moment when a carefully designed performance optimization becomes operational.

Context: The Problem Phase 8 Solves

To understand why this message was written, one must first understand the problem it solves. The cuzk engine is a high-performance Groth16 proof generator for Filecoin's Proof-of-Replication (PoRep) protocol. It synthesizes circuits on the CPU, then proves them on the GPU using CUDA kernels for Number Theoretic Transforms (NTT) and Multi-Scalar Multiplication (MSM). Prior to Phase 8, the C++ function generate_groth16_proofs_c held a static std::mutex that locked the entire function body. This meant that while one GPU worker held the lock — running CUDA kernels — a second worker targeting the same GPU could not even begin its CPU-side preprocessing. The result was measurable GPU idle time: the GPU would finish its kernel work and sit idle while the next worker finished its CPU preprocessing before it could acquire the mutex and launch new kernels.

Phase 7 had already improved throughput by dispatching partitions independently, but the static mutex remained a bottleneck. Phase 8's insight was to narrow the mutex scope: instead of locking the entire function, the lock would cover only the CUDA kernel region (NTT+MSM, batch additions, tail MSMs). CPU preprocessing and the b_g2_msm computation would run outside the lock. This allowed two GPU workers per device to interleave — one performing CPU work while the other held the mutex for kernel execution.

The Plumbing Chain

Implementing this required threading a mutex pointer through the entire call stack. The C++ CUDA kernel was refactored to accept a passed-in std::mutex* parameter with narrowed lock scope. Two helper functions — create_gpu_mutex and destroy_gpu_mutex — were added to allocate and deallocate C++ std::mutex objects from the Rust side. The FFI layer in supraseal-c2/src/lib.rs was updated to expose these helpers and to thread the mutex pointer through the extern declaration and wrapper functions. The bellperson library's prove_from_assignments function was extended with a gpu_mutex parameter. The pipeline's gpu_prove function received the same parameter, with non-engine callers passing std::ptr::null_mut() for backward compatibility.

By the time we reach message 2186, all of these pieces are in place. The engine's worker spawn loop (message 2184) has been updated to create one C++ mutex per GPU using supraseal_c2::alloc_gpu_mutex(), wrap it in a SendableGpuMutex for thread-safe sharing, and spawn gpu_workers_per_device workers (default 2) per GPU, each receiving the same mutex pointer. The tokio::spawn block (message 2185) has been updated to include a worker_sub_id for logging and to carry the mutex into the closure.

What This Message Actually Does

The edit in message 2186 is deceptively simple: it modifies the gpu_prove call inside the spawned worker closure to pass the mutex pointer. In Rust terms, this is a single argument added to a function call. But this is the moment when the mutex, allocated per-GPU and shared across workers, finally reaches the function that will actually use it. Without this edit, all the preceding work — the C++ refactoring, the FFI declarations, the bellperson parameter, the pipeline signature change, the worker spawn loop — would be dead code. The mutex would be created, wrapped, and carried into the closure, but never actually passed to the proving function. The narrowed lock scope would never be exercised.

This pattern is common in large-scale refactoring: the most critical change is often the most mundane. The assistant's reasoning, visible in the preceding messages, shows a careful step-by-step approach. Message 2180 identifies the need: "The engine calls crate::pipeline::gpu_prove() — I need to pass the mutex pointer there." Message 2181 reveals a key design decision: the assistant initially considered using Rust's std::sync::Mutex<()> but realized "the C++ side expects a std::mutex* (C++ type) — Rust's std::sync::Mutex is NOT the same thing." This cross-language awareness is crucial; using a Rust mutex would have produced a pointer to a completely different data structure, causing undefined behavior or crashes. The assistant correctly chose to add C++ helper functions (create_gpu_mutex/destroy_gpu_mutex) to allocate the mutex on the C++ side where its layout is guaranteed.

Assumptions and Risks

The edit makes several implicit assumptions. It assumes that the mutex pointer created by create_gpu_mutex is valid and correctly sized. It assumes that the SendableGpuMutex wrapper correctly implements Send and Sync for the raw pointer. It assumes that two workers sharing the same mutex will correctly interleave — that the narrowed lock scope is sufficient to prevent races while allowing overlap. And it assumes that the C++ std::mutex allocated via new (inside the helper) will outlive all workers that hold references to it, which is guaranteed by the engine's lifecycle management.

There is also an assumption about thread safety at the FFI boundary. The Rust side holds a raw pointer to a C++ object. The SendableGpuMutex wrapper must ensure that the pointer is not used after the mutex is destroyed, and that the mutex is destroyed exactly once after all workers have finished. The engine's shutdown sequence must respect this lifecycle.

What This Message Produces

The output of this message is not just a successful edit — it is the completion of the Phase 8 plumbing. With this change, the engine can now spawn multiple GPU workers per device that interleave CPU preprocessing and GPU kernel execution, eliminating the idle gaps that plagued earlier phases. The benchmark results that follow in the session confirm the impact: single-proof GPU efficiency reaches 100.0% (zero idle gaps), and multi-proof throughput improves by 13–17% compared to Phase 7.

In the broader narrative of the cuzk optimization project, this message is the final stitch in a fabric of performance engineering. It connects the high-level architectural insight — narrow the mutex, interleave the workers — to the low-level reality of a CUDA kernel accepting a mutex pointer. It is a reminder that in systems programming, the distance between an idea and its execution is measured not in lines of code, but in the careful, deliberate threading of state through every layer of the stack.