The Architecture of a Single Edit: Routing Mutex Pointers Through Bellperson in Phase 8's Dual-Worker GPU Interlock

Introduction

In the course of implementing Phase 8 of the cuzk SNARK proving engine—a dual-worker GPU interlock designed to eliminate GPU idle gaps in Filecoin's PoRep proof generation—a single, seemingly trivial edit reveals the hidden architecture of a complex multi-language, multi-library codebase. The message in question, message [msg 2205], reads in its entirety:

Now update the engine to use the bellperson re-export instead of supraseal_c2: [edit] /home/theuser/curio/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

This is not a glamorous message. It contains no benchmark results, no algorithmic insight, no clever optimization. It is a simple import path change. Yet the reasoning, context, and assumptions that produced this edit illuminate the fundamental structure of the entire proving pipeline and the careful layering of its dependencies. Understanding why this edit was necessary—and what it reveals about the system—is essential to grasping the architecture of the cuzk proving engine.

The Problem: A Missing Dependency Edge

To understand message [msg 2205], we must trace the chain of reasoning that led to it. The Phase 8 design, documented in c2-optimization-proposal-8.md, called for narrowing the scope of a C++ static std::mutex in groth16_cuda.cu so that only the CUDA kernel region (NTT+MSM, batch additions, tail MSMs) holds the lock, while CPU preprocessing and the b_g2_msm computation run outside it. This allows two GPU workers per device to interleave: one worker performs CPU work while the other holds the lock and runs CUDA kernels, effectively doubling GPU utilization.

The implementation required threading a C++ std::mutex* pointer from the Rust engine layer, through the FFI boundary into supraseal-c2, through the bellperson abstraction layer, and finally into the CUDA kernel. In message [msg 2181], the assistant designed the approach: add two C helper functions (create_gpu_mutex and destroy_gpu_mutex) in the .cu file that allocate and free a C++ std::mutex on the heap, returning a raw void* pointer. These were exposed through supraseal-c2/src/lib.rs as FFI functions.

The engine code in cuzk-core/src/engine.rs then needed to call these functions. In message [msg 2184], the assistant wrote code that directly invoked supraseal_c2::alloc_gpu_mutex() to create one mutex per GPU device. This seemed natural: the engine needed to allocate a C++ mutex, and supraseal-c2 provided that function.

But when the assistant attempted to build in message [msg 2201], the compilation revealed a fundamental architectural constraint: the engine (cuzk-core) does not have a direct dependency on supraseal-c2. The dependency graph is layered as follows:

The Decision: Routing Through Bellperson

In message [msg 2202], the assistant recognized this constraint and formulated the solution:

The engine can't directly access supraseal_c2 — it needs to go through bellperson or pipeline. Let me re-export the mutex creation functions through bellperson and then through the pipeline module.

The assistant then executed a two-step plan. First, in message [msg 2203], it added wrapper functions in bellperson's supraseal.rs module that re-export alloc_gpu_mutex and destroy_gpu_mutex from supraseal-c2, along with the SendableGpuMutex wrapper type that makes the raw *mut c_void pointer Send and Sync for use in async Rust contexts. Second, in message [msg 2204], it re-exported these from bellperson's public groth16::mod.rs so they are accessible as bellperson::groth16::SendableGpuMutex and bellperson::groth16::alloc_gpu_mutex.

Message [msg 2205] is the third and final step: updating the engine to use these bellperson re-exports instead of the nonexistent direct path to supraseal-c2. The edit itself is trivial—likely changing supraseal_c2::alloc_gpu_mutex to bellperson::groth16::alloc_gpu_mutex and adjusting the import accordingly—but it represents a crucial architectural alignment.

Assumptions and Their Consequences

Several assumptions underpin this edit, and examining them reveals the thinking process:

Assumption 1: Bellperson is the correct abstraction layer. The assistant assumed that bellperson, not the pipeline module, should expose the mutex creation functions. This was a deliberate choice over an alternative: routing through cuzk-core/src/pipeline.rs. The pipeline module already re-exports gpu_prove() and other proving functions, so it could have been a natural intermediary. However, bellperson is the library that already depends on supraseal-c2, making it the most direct path. The pipeline module, being part of cuzk-core, would have required adding supraseal-c2 as a dependency of cuzk-core—a more invasive change that would break the existing abstraction layering.

Assumption 2: The SendableGpuMutex wrapper is sufficient for async safety. The raw *mut c_void pointer from alloc_gpu_mutex is neither Send nor Sync in Rust's type system, meaning it cannot be safely transferred across async task boundaries. The assistant had already created a SendableGpuMutex wrapper type (a newtype around the pointer) that implements Send and Sync. However, as message [msg 2207] reveals, the build still failed because extracting the raw pointer from SendableGpuMutex.0 inside the spawn_blocking closure reintroduced a non-Send value. The wrapper was not enough—the code needed to pass the wrapper itself into the closure and only dereference it inside.

Assumption 3: The dependency graph is fixed. The assistant accepted the existing dependency layering as immutable and worked within it. An alternative approach would have been to add supraseal-c2 as a direct dependency of cuzk-core, which would have been simpler but would violate the project's architectural boundaries. The assistant chose to respect the existing structure, even at the cost of additional plumbing.

Input and Output Knowledge

To understand message [msg 2205], one needs specific input knowledge:

  1. The Rust dependency graph: cuzk-core → bellperson → supraseal-c2, with no direct link between cuzk-core and supraseal-c2.
  2. The Phase 8 design: The dual-worker GPU interlock requires a per-GPU C++ mutex that must be allocated on the C++ side and passed as a raw pointer through FFI.
  3. The FFI plumbing already in place: alloc_gpu_mutex and destroy_gpu_mutex already exist in supraseal-c2/src/lib.rs and are declared as extern "C" functions in the .cu file.
  4. The bellperson re-export work: Messages [msg 2203] and [msg 2204] have already added the wrappers and re-exports. The output knowledge created by this message is:
  5. A corrected import path: The engine now uses bellperson::groth16::alloc_gpu_mutex instead of the inaccessible supraseal_c2::alloc_gpu_mutex.
  6. Validation of the architecture: The edit confirms that the bellperson re-export strategy is the correct approach for threading C++ resources through the abstraction layers.
  7. A pattern for future cross-layer plumbing: Future changes that require the engine to access C++ resources (e.g., GPU stream handles, custom allocators) can follow the same pattern: add wrappers in bellperson's supraseal module, re-export from bellperson's public API, and consume in the engine.

The Thinking Process

The reasoning visible in the surrounding messages reveals a methodical, architectural mindset. When the build failed in message [msg 2201], the assistant did not simply add supraseal-c2 as a dependency of cuzk-core—a quick fix that would have worked but blurred the abstraction boundaries. Instead, it traced the dependency graph, identified the correct layer (bellperson) that already bridges the two sides, and added the necessary re-exports there.

The assistant's comment in message [msg 2202]—"The engine can't directly access supraseal_c2 — it needs to go through bellperson or pipeline"—shows an awareness of the system's architectural constraints. The phrase "or pipeline" acknowledges the alternative but the subsequent actions show a clear preference for bellperson. This is a design decision made implicitly: bellperson is the semantically correct layer because it is the Groth16 proving abstraction, and the mutex is a resource needed for Groth16 proving.

The three-message sequence ([msg 2203], [msg 2204], [msg 2205]) follows a clean pattern: add the functionality at the lowest appropriate layer (bellperson's supraseal module), expose it at the public API boundary (bellperson's groth16 module), and consume it at the call site (the engine). This is textbook layered architecture.

Broader Significance

Message [msg 2205] is easy to overlook. It is a single edit, a few characters changed in an import statement. But it is the keystone of the Phase 8 implementation. Without this edit, the engine cannot allocate the per-GPU mutexes, the dual workers cannot interlock, and the entire optimization collapses.

More importantly, this message reveals a fundamental truth about the cuzk proving engine: it is not a monolithic application but a carefully layered system of libraries with explicit dependency boundaries. The separation between cuzk-core (engine logic), bellperson (proving abstraction), and supraseal-c2 (CUDA implementation) is not incidental—it is a deliberate architecture that allows each layer to be tested, replaced, or optimized independently. The assistant's decision to respect these boundaries, even when a shortcut was available, demonstrates an understanding that architectural discipline in systems programming pays dividends in maintainability and correctness.

The edit also foreshadows the next challenge: even after fixing the import path, the SendableGpuMutex wrapper would still cause compilation errors due to raw pointer extraction in async contexts (as seen in message [msg 2207]). The architectural work was necessary but not sufficient—the type system had more constraints to satisfy. But that is a story for the next message.