The FFI Glue: How a Single Line of Plumbing Unlocked Dual-Worker GPU Interlock in the cuzk SNARK Proving Engine

Introduction

In the high-stakes world of Filecoin storage proof generation, every second of GPU idle time is a direct cost burned. The cuzk SNARK proving engine, a custom Groth16 proof pipeline for Filecoin's Proof-of-Replication (PoRep), had been chasing a persistent performance problem: GPU utilization gaps caused by a coarse-grained C++ static mutex that locked the entire generate_groth16_proofs_c function. The Phase 8 architecture, designed to fix this, proposed a dual-worker GPU interlock scheme where two workers per GPU device could interleave their work — one performing CPU-side preprocessing while the other held the GPU for CUDA kernel execution. But implementing this required threading a mutex pointer through a multi-layer software stack spanning C++, Rust FFI, and application-level Rust code. The message at the center of this article — message index 2156 — represents the critical FFI plumbing step that bridged the C++ and Rust worlds.

The Message

The subject message is deceptively brief:

## Step 2: FFI plumbing in supraseal-c2/src/lib.rs

Add the gpu_mtx parameter to the extern decl and both wrapper functions: [edit] /home/theuser/curio/extern/supraseal-c2/src/lib.rs Edit applied successfully.

Four lines. One file edit. Yet this single message is the linchpin connecting two vastly different execution environments — the CUDA kernel code written in C++ and the Rust proving pipeline that orchestrates proof generation. Without this FFI bridge, the mutex refactored in Step 1 (the C++ side) would remain invisible to the Rust code that needs to create, pass, and destroy it.

Why This Message Was Written: The Reasoning and Motivation

To understand why this message exists, one must understand the architecture it serves. Phase 8's central insight was that the existing static mutex in generate_groth16_proofs_c was locking far too much code. The original C++ function had a static std::mutex declared at function scope, meaning every call to the function — from any thread — would contend for the same lock. The lock covered the entire function body, including CPU-side preprocessing steps (assignment deserialization, b_g2_msm computation) that did not touch the GPU at all. This meant that while one worker held the mutex and ran CPU preprocessing, the GPU sat idle, and no other worker could even begin its own preprocessing.

The Phase 8 design, documented in the earlier proposal, called for narrowing the mutex to cover only the CUDA kernel region — the NTT (Number Theoretic Transform), MSM (Multi-Scalar Multiplication), batch additions, and tail MSMs that actually execute on the GPU. CPU preprocessing and the b_g2_msm computation would run outside the lock, allowing two workers per GPU to interleave: Worker A could be doing CPU work while Worker B held the GPU for kernel execution, and vice versa.

But this design required a fundamental change to the mutex's lifecycle. A static mutex is created once and lives for the duration of the process. The narrowed mutex, however, needed to be shared across multiple Rust-level GPU workers that the engine would spawn. The engine needed to create one mutex per GPU device, pass it down through the call stack to the C++ function, and ensure it was properly destroyed when the engine shut down. This meant the mutex could no longer be static — it had to be a parameter, passed explicitly from the Rust side through the FFI boundary into the C++ function.

This is the precise motivation for message 2156. Step 1 (messages 2151–2154) had already refactored the C++ side: the static mutex was removed, the function signature gained a std::mutex* parameter, and the lock scope was narrowed to the CUDA kernel region. But the C++ changes alone were useless without the FFI layer that would allow Rust code to pass the mutex pointer. Message 2156 is the bridge.

How Decisions Were Made

The message reveals several implicit design decisions worth examining. First, the choice to pass a raw std::mutex* pointer across the FFI boundary rather than using a more idiomatic Rust synchronization primitive. This is a pragmatic decision driven by the existing architecture: the C++ code already uses std::mutex, and the Rust code calls into C++ through an extern "C" function. Passing a raw pointer is the simplest and most efficient way to share the mutex across the language boundary, avoiding the overhead of wrapping it in a Rust Mutex or using a lock-free alternative.

Second, the decision to add the mutex parameter to "both wrapper functions." The supraseal-c2/src/lib.rs file contains two Rust wrapper functions around the C++ extern: generate_groth16_proof (the primary entry point used by the pipeline) and generate_groth16_proofs (a simpler variant). Both needed the mutex parameter for consistency, even if only one path was initially used by the dual-worker architecture. This forward-looking decision ensured that whichever code path was exercised, the mutex would be properly passed through.

Third, the message's placement as "Step 2" in a multi-step implementation reveals the assistant's systematic approach. Step 1 handled the C++ kernel. Step 2 handles the FFI bridge. Step 3 (subsequent messages) would modify the Rust bellperson prover layer. Step 4 would modify the pipeline and engine to spawn multiple workers and manage mutex lifecycle. This sequential, layered approach minimizes risk — each layer is validated before the next is touched.

Input Knowledge Required

Understanding this message requires knowledge spanning several domains. The reader must understand the concept of Foreign Function Interface (FFI) — the mechanism by which Rust code can call C and C++ functions. They must know that extern "C" functions in C++ are callable from Rust via extern blocks, and that passing complex types like std::mutex across this boundary requires raw pointers because Rust has no knowledge of C++'s type system.

The reader must also understand the broader cuzk architecture: that supraseal-c2 is a Rust crate wrapping SupraSeal's C++ CUDA code, that bellperson is a fork of the Bellman proving library modified for this pipeline, and that cuzk-core contains the engine and pipeline orchestration. The mutex must thread through all three layers: from the engine (which creates it), through the pipeline (which passes it to bellperson), through bellperson (which passes it to supraseal-c2), and finally across the FFI boundary into the C++ CUDA kernel.

Additionally, knowledge of the specific performance problem is essential. The reader must understand why a static mutex causes GPU idle gaps in a multi-worker scenario, and why narrowing the lock scope to only the CUDA kernel region enables the dual-worker interleave. This context is what makes the seemingly simple FFI change meaningful — it's not just plumbing, it's the enabler of a 13-17% throughput improvement.

Output Knowledge Created

This message created the FFI bridge that connects the C++ mutex refactoring to the Rust proving pipeline. Specifically, it modified supraseal-c2/src/lib.rs to:

Add a gpu_mtx: *mut std::ffi::c_void parameter to the extern "C" declaration of generate_groth16_proofs_c
Update the generate_groth16_proof wrapper function to accept and forward the mutex pointer
Update the generate_groth16_proofs wrapper function similarly The exact edits are visible in the subsequent messages (2157–2159), which show the assistant incrementally updating each function. The output knowledge is the validated FFI signature that the rest of the Rust stack can depend on. Once this message completed, the bellperson layer (Step 3) could add its own gpu_mutex parameter, and the engine (Step 4) could create per-GPU mutexes and pass them down.

Assumptions and Potential Pitfalls

The message makes several assumptions worth examining. It assumes that passing a raw std::mutex* pointer across the FFI boundary is safe — that the C++ mutex object, allocated on the Rust side (via a create_gpu_mutex helper added later), will remain valid for the duration of the C++ function call. This is a reasonable assumption given that the mutex is only accessed within the scope of the call, but it does require careful lifecycle management. If the Rust side were to destroy the mutex while a C++ call was in progress, undefined behavior would result.

The message also assumes that both wrapper functions should receive the mutex parameter, even though the dual-worker architecture primarily uses generate_groth16_proof. This is a conservative assumption that adds a small amount of complexity to the less-used code path in exchange for consistency and future-proofing.

A subtle assumption is that the std::mutex from the C++ standard library is binary-compatible across the FFI boundary. In practice, std::mutex is typically a simple wrapper around a pthread mutex on Linux, and passing a pointer to it is well-defined behavior. However, this is not guaranteed by the C++ standard, and different compilers or standard library implementations could theoretically break this assumption. The codebase's use of a specific GCC toolchain for CUDA compilation mitigates this risk.

The Thinking Process Visible in the Reasoning

The assistant's thinking process, visible in the surrounding messages, reveals a methodical approach to the FFI plumbing. In message 2155, the assistant explicitly states: "C++ changes look correct. Now let's move to the FFI layer." This checkpoint-and-proceed pattern demonstrates a careful, incremental workflow. The assistant doesn't attempt to modify all files at once; it completes one layer (C++), verifies it, then moves to the next (FFI).

The todo list management visible in message 2155 shows the assistant tracking progress: Step 1 marked "completed," Step 2 marked "in_progress." This structured approach is essential for a multi-file change spanning ~195 lines across 7 files. Without it, the risk of missing a critical edit or introducing inconsistencies between layers would be high.

The message itself, though brief, is preceded by extensive reading and analysis. Messages 2147–2149 show the assistant reading all affected files to understand the current state before making any changes. This preparation is the foundation that makes the concise FFI edit possible — the assistant already knows the exact function signatures, parameter types, and calling conventions it needs to modify.

Broader Significance

Message 2156, for all its brevity, represents a microcosm of the entire Phase 8 implementation. It is the point where design meets reality — where the abstract concept of "narrow the mutex scope" becomes concrete code that must work correctly across language boundaries. The FFI layer is often the most error-prone part of a mixed-language system, and getting it right is essential for correctness.

The benchmark results that followed (documented in the chunk summary) validate the entire approach: single-proof GPU efficiency reached 100.0% (zero idle gaps), and multi-proof throughput improved 13.2–17.2%. These numbers are the ultimate justification for the FFI plumbing in message 2156. Without this bridge, the C++ refactoring would be isolated and useless. With it, the dual-worker interlock becomes operational, and the GPU utilization gaps that plagued Phase 7 are eliminated.

In the broader narrative of the cuzk proving engine's evolution, this message is a quiet but crucial turning point. It marks the transition from a single-worker, lock-everything architecture to a dual-worker, interleaved architecture that better utilizes expensive GPU resources. And it all hinges on a single parameter threaded through the FFI boundary — a std::mutex* pointer that carries the weight of a 17% throughput improvement.