The FFI Reconnaissance: How a Single Grep Shaped the Phase 10 Two-Lock Architecture

In the middle of an intense optimization campaign for the SUPRASEAL_C2 Groth16 proof generation pipeline, a seemingly mundane command was issued:

[grep] create_gpu_mutex|alloc_gpu_mutex|GpuMutex"
Found 5 matches
/home/theuser/curio/extern/supraseal-c2/src/lib.rs:
  Line 187:     fn create_gpu_mutex() -> *mut core::ffi::c_void;
  Line 195: pub fn alloc_gpu_mutex() -> *mut core::ffi::c_void {
  Line 196:     unsafe { create_gpu_mutex() }
  Line 199: /// Free a C++ `std::mutex` previously created by `alloc_gpu_mutex`.
  Line 202: /// `ptr` must be a pointer returned by `alloc_gpu_mutex` and must not

This is message [msg 2578] in the conversation — a grep command searching for mutex-related FFI symbols across the supraseal-c2 codebase. On its surface, it is a trivial act of code navigation. But in the context of the broader optimization journey, this message represents a critical architectural pivot point: the moment when the assistant paused to survey the existing Foreign Function Interface (FFI) boundary before designing Phase 10's two-lock GPU synchronization protocol. This article examines why this message matters, what it reveals about the engineering process, and how it connects to the larger narrative of GPU optimization for Filecoin's proof generation pipeline.

Context: The Optimization Campaign So Far

To understand message [msg 2578], one must first understand the optimization trajectory that led to it. The session was deep into a multi-phase effort to optimize the SUPRASEAL_C2 Groth16 proof generation engine — a critical component in Filecoin's Proof-of-Replication (PoRep) system. The pipeline involves synthesizing circuit witnesses on the CPU, then proving them on the GPU through a sequence of Number Theoretic Transforms (NTT), batch additions, and Multi-Scalar Multiplications (MSM).

Phase 9 had just been completed, implementing PCIe transfer optimization that pre-staged GPU buffers while the CPU was still working. This cut GPU kernel time by 51% — from 3.75 seconds to 1.82 seconds per partition. But the victory was pyrrhic: the bottleneck simply shifted from PCIe transfers to CPU memory bandwidth. The CPU's prep_msm and b_g2_msm operations now dominated the critical path at 2.4 seconds per partition, leaving the GPU idle for 600 milliseconds waiting for data.

The user then proposed a bold idea in [msg 2569]: what if more GPU workers could be added (3-4 instead of 2), interlocked with two separate mutexes — one for compute (GPU kernel execution) and one for memory management (VRAM allocation and upload)? The idea was to overlap a worker's CPU-side memory operations with another worker's GPU kernel execution, hiding the CPU overhead that had become the new bottleneck.

The assistant spent messages [msg 2570] through [msg 2577] analyzing this proposal, tracing VRAM allocations, identifying deadlock risks, and ultimately designing a cooperative handoff protocol. By message [msg 2578], the assistant was ready to implement — but first, it needed to understand the FFI boundary through which GPU mutexes were created and passed between Rust and C++.

Why This Message Was Written: The FFI Boundary as a Design Constraint

The grep command in [msg 2578] was not random exploration. It was a targeted reconnaissance of the existing synchronization infrastructure. The Phase 8 dual-worker interlock had introduced a single std::mutex per GPU, created in C++ and passed as an opaque pointer to Rust via FFI. The Rust engine code stored this pointer and passed it to each GPU worker thread, which would then cast it back to a std::mutex* and acquire it before running CUDA kernels.

Phase 10's two-lock design would need to extend this pattern. Instead of one mutex per GPU, the design called for two: a compute_mtx for serializing GPU kernel execution and a mem_mtx for serializing VRAM allocation and upload. The assistant needed to answer several questions before implementation could proceed:

  1. How was the single mutex created? The create_gpu_mutex() function, declared with C-style FFI linkage, allocated a std::mutex on the C++ heap and returned a raw void* pointer. This was the established pattern.
  2. How was it wrapped for Rust safety? The alloc_gpu_mutex() function provided a safe Rust wrapper around the unsafe FFI call, while free_gpu_mutex() handled cleanup. This abstraction layer would need to be extended for two mutexes.
  3. How was the pointer stored and distributed? The engine code in engine.rs stored the opaque pointer and passed it to worker tasks. For Phase 10, a struct containing two pointers (or a single pointer to a C++ struct containing both mutexes) would be needed.
  4. What about the existing gpu_mutex_addr field? The FFI function generate_groth16_proofs_c accepted a gpu_mutex_addr parameter — a raw pointer to the mutex. Phase 10 would need to either add a second parameter or change the interface to accept a struct. The grep results confirmed that the FFI boundary was minimal and straightforward — exactly what one would expect for passing a single C++ mutex to Rust. But this simplicity also meant that extending it to two mutexes would require careful design. The assistant could not simply add a second opaque pointer; it needed to ensure that the C++ side could create, destroy, and manage both mutexes correctly, and that the Rust side could pass them through to the worker threads without race conditions or memory leaks.

The Thinking Process: Engineering Discipline in Action

What makes message [msg 2578] revealing is not the grep itself, but what it represents about the assistant's engineering approach. The assistant had just spent several messages analyzing deadlock scenarios, tracing VRAM allocation patterns, and designing a cooperative handoff protocol. Now, before writing a single line of implementation code, it paused to check the FFI boundary.

This is a hallmark of disciplined systems programming. The FFI boundary is where assumptions are tested and where bugs are born. A mutex created in C++ but managed through Rust lifetimes must be carefully handled: the C++ constructor and destructor must run, the alignment must be correct, and the pointer must remain valid across thread boundaries. The grep confirmed that the existing code used std::mutex allocated on the heap via new (implied by the create_gpu_mutex / free_gpu_mutex pair), which is the standard pattern for FFI-safe mutex passing.

The assistant was also implicitly checking whether the existing infrastructure could be extended or whether a new approach was needed. The grep found five matches, all in lib.rs, suggesting a clean, centralized FFI layer. This was good news: the two-lock extension could follow the same pattern, adding create_gpu_mutex_pair() or similar functions that return a struct containing both mutex pointers.

However, the grep also revealed an important constraint: the existing FFI function generate_groth16_proofs_c accepted a single gpu_mutex_addr parameter. Changing this to accept two mutexes would require modifying the C++ function signature, the FFI declaration, and all call sites. This is a non-trivial change that touches multiple layers of the codebase.

Input Knowledge Required

To fully understand message [msg 2578], several pieces of context are necessary:

The Phase 8 single-mutex architecture: The existing GPU interlock used one std::mutex per GPU device. This mutex serialized the entire GPU-critical section — VRAM allocation, upload, kernel execution, and cleanup — preventing two workers from using the same GPU simultaneously. Phase 10 aimed to split this into two mutexes to allow finer-grained overlap.

Rust FFI patterns: The code uses extern "C" functions declared in a Rust extern block, returning raw pointers (*mut c_void). The alloc_gpu_mutex wrapper calls the unsafe FFI function and returns the pointer. This is standard practice for passing C++ objects across the language boundary.

The CUDA programming model: GPU operations are asynchronous by default. cudaMalloc, cudaMemcpy, and kernel launches return immediately, and the CPU must explicitly synchronize (via cudaDeviceSynchronize or stream events) to ensure completion. This asynchrony is what makes the two-lock design potentially valuable — a worker could upload data to the GPU while another worker's kernels are still running, as long as they use different memory regions.

The VRAM capacity constraint: The GPU has 16 GiB of VRAM, and each partition's proof requires ~13.8 GiB during peak usage. This leaves only ~2 GiB of headroom, making it impossible for two workers to have their full buffer sets allocated simultaneously. This constraint ultimately shaped the lock protocol design.

Output Knowledge Created

Message [msg 2578] produced a concrete and actionable result: confirmation of the FFI boundary structure for mutex creation and management. The assistant now knew:

  1. The exact FFI declarations for creating, wrapping, and freeing GPU mutexes.
  2. The location of these declarations (extern/supraseal-c2/src/lib.rs).
  3. The pattern used: C++ heap allocation, opaque pointer returned to Rust, explicit free function for cleanup.
  4. The scope: Five matches, all in the same file, indicating a clean and centralized FFI layer. This knowledge directly informed the Phase 10 implementation that followed. The assistant would need to extend this pattern to support two mutexes per GPU, either by creating a C++ struct containing both mutexes or by adding a second opaque pointer parameter to the FFI functions.

The Broader Significance: A Lesson in Bottleneck Shifting

Message [msg 2578] sits at a fascinating inflection point in the optimization journey. Phase 9 had successfully accelerated the GPU, only to reveal that the CPU was now the bottleneck. The proposed two-lock design was an attempt to hide CPU latency by overlapping more work — but it was fundamentally constrained by hardware realities.

The grep for mutex FFI symbols was the moment when the assistant moved from analysis to implementation. It was checking the foundation before building the new structure. This is the kind of careful, methodical work that distinguishes robust systems programming from hacky optimization.

What makes this message particularly interesting is what happened next. The Phase 10 implementation, as documented in [chunk 27.1], ran into fundamental CUDA device-global synchronization conflicts. The cudaDeviceSynchronize and cudaMemPoolTrimTo operations inside the mem_mtx region turned out to be device-global operations that blocked while another worker held compute_mtx and ran kernels. This effectively serialized the two locks, destroying the intended overlap and causing performance regression.

The grep in [msg 2578] could not have predicted this outcome. The FFI boundary was correct; the mutex creation pattern was sound. The failure was at a deeper level — the CUDA driver's implicit synchronization model — that no amount of FFI analysis could have revealed. This is a powerful reminder that in GPU programming, the hardware's behavioral model often overrides the software's abstraction layer.

Conclusion

Message [msg 2578] is, on its surface, a simple grep command. But in the context of the SUPRASEAL_C2 optimization campaign, it represents a critical engineering moment: the transition from analysis to implementation, the reconnaissance of the FFI boundary before extending it, and the disciplined habit of checking existing infrastructure before building new abstractions.

The grep revealed a clean, minimal FFI layer for mutex management — exactly what the Phase 8 single-mutex design required, and exactly what Phase 10 would need to extend. The assistant learned the precise pattern for creating, wrapping, and destroying GPU mutexes across the Rust/C++ boundary, and confirmed that the infrastructure was centralized and well-structured.

This message also illustrates a deeper truth about systems optimization: the bottleneck is always moving. Phase 9 fixed PCIe transfers, revealing CPU memory bandwidth as the new bottleneck. Phase 10 attempted to hide CPU latency with more workers and finer-grained locking, only to discover that CUDA's device-global synchronization model defeated the abstraction. The grep for mutex FFI symbols was a necessary step in this journey — a moment of careful engineering before the inevitable encounter with hardware reality.