The FFI Boundary: A Pivotal Read in the Phase 10 Two-Lock Design

In the middle of a deep optimization campaign targeting Filecoin's Groth16 proof generation pipeline, a single message stands out for what it reveals about the engineering process: message [msg 2579], in which the assistant reads the Rust FFI file lib.rs to examine the existing mutex creation interface. On its surface, this is a mundane action — a developer reading a few lines of code. But in context, it represents a critical moment of design verification, where abstract architectural thinking meets the concrete constraints of a cross-language boundary.

The Optimization Context

The message sits within a multi-week effort to optimize supraseal-c2, the CUDA-accelerated Groth16 prover used in Filecoin's Proof-of-Replication (PoRep) pipeline. By Phase 9, the team had achieved remarkable GPU kernel speedups — NTT operations were 72% faster, and per-partition GPU time had dropped from 3.75 seconds to 1.82 seconds. But the optimization had shifted the bottleneck: the CPU-side prep_msm (1.91s) and b_g2_msm (0.48s) operations now dominated the critical path, leaving the GPU idle for 600ms per partition while waiting for the CPU thread to finish.

The user's insight in [msg 2569] was elegant: if the GPU is idle waiting for the CPU, add more GPU workers. With two workers, one could run GPU kernels while the other did CPU preparation. But Phase 9's pre-staging optimization had inadvertently serialized the workers, negating the dual-worker benefit. The user proposed a two-lock architecture: a compute lock for GPU kernel execution and a memory-management lock for VRAM allocation and pre-staging uploads. With three or four workers, one could hold the compute lock running kernels, another could hold the memory lock doing pre-staging, and a third could do CPU-only work without any lock at all.

The Message Itself

Message [msg 2579] is deceptively simple. The assistant issues a read tool call on /home/theuser/curio/extern/supraseal-c2/src/lib.rs, requesting lines 180–192. The returned content shows:

extern "C" {
    fn create_gpu_mutex() -> *mut core::ffi::c_void;
    fn destroy_gpu_mutex(mtx: *mut core::ffi::c_void);
}

This is the FFI (Foreign Function Interface) boundary between Rust and C++. The create_gpu_mutex function allocates a C++ std::mutex on the heap and returns an opaque pointer. The Rust side never touches the mutex directly — it just passes the pointer through to the CUDA code. The destroy_gpu_mutex function frees it.

The assistant also sees the struct definition ending at line 184 and the doc comments beginning at line 191, but the critical content is the two FFI function declarations.

Why This Read Matters

This read is not random code browsing. It is a deliberate design probe. The assistant has been reasoning through the two-lock architecture across five prior messages ([msg 2570] through [msg 2578]), and has reached a critical juncture: the design must cross the Rust/C++ boundary.

The existing architecture allocates a single std::mutex per GPU device, wraps it as an opaque pointer, and passes it through FFI to the CUDA code. The two-lock design requires two mutexes per device — a compute_mtx and a mem_mtx. This means the FFI interface must change. The assistant needs to understand:

  1. How the mutex is currently created — to replicate the pattern for a second mutex
  2. How the pointer is passed — to understand whether to pass two pointers or a struct
  3. How the mutex is destroyed — to ensure cleanup is correct for both locks The read reveals that the current interface is minimal: create_gpu_mutex() returns *mut c_void, and destroy_gpu_mutex() takes *mut c_void. There is no struct, no wrapper type on the Rust side. This simplicity is both a blessing and a constraint. It means the Rust side treats the mutex as a pure opaque handle — it never inspects or manipulates it. But it also means that adding a second mutex requires either: - Adding two new FFI functions (create_compute_mutex, create_mem_mutex) - Or creating a C++ struct that holds both mutexes and returning a single opaque pointer to that struct The assistant's subsequent actions (visible in the next chunk's summary) show that the chosen approach was to create a gpu_locks struct on the C++ side containing both mem_mtx and compute_mtx, and pass a single opaque pointer to the Rust side. The FFI read was the essential prerequisite for this decision.

The Reasoning Chain

To understand why this particular read happens at this moment, we must trace the reasoning chain across the preceding messages.

In [msg 2570], the assistant first engages with the user's two-lock proposal, thinking through the lock ordering: "Worker A: holds compute lock, running GPU kernels. Worker B: holds memory lock, doing pre-stage alloc + upload + pool trim. Worker C: doing CPU-only work."

In [msg 2572], the assistant spawns a subagent task to examine the engine code in engine.rs, discovering how mutexes are created and distributed to workers. This reveals the pattern: one mutex per GPU, created once and shared by all workers on that device.

In [msg 2573], the assistant poses a design question to the user about two locking strategies (nested locking vs. cooperative handoff), and the user responds with a detailed analysis of the two-lock design.

In [msg 2574], the assistant identifies a critical deadlock scenario: if Worker A holds compute_mtx and needs mem_mtx to free VRAM, while Worker B holds mem_mtx and is blocked on a cudaMalloc that can't proceed because Worker A hasn't freed its buffers, neither worker can progress. This is a resource deadlock, not a lock-ordering deadlock.

In [msg 2575], the assistant performs a comprehensive VRAM allocation trace, confirming that peak usage during H-MSM is ~13.8 GiB on a 16 GiB GPU, leaving only ~2 GiB headroom. This means two workers' buffers cannot coexist — pre-staging must be serialized.

The user's response in [msg 2575] reframes the goal: the win from splitting locks is not overlapping pre-staging with compute, but overlapping the CPU b_g2_msm and epilogue with the next worker's GPU compute. This is a more modest but achievable gain of ~600ms per partition.

In [msg 2576], the assistant commits to working out the detailed design: "Let me first nail down the exact lock protocol to avoid deadlocks, then map out all the code changes needed."

In [msg 2577], the assistant reads the CUDA code to understand the current lock usage. In [msg 2578], it greps for the FFI mutex functions. And then in [msg 2579], it reads the actual FFI declarations.

Input and Output Knowledge

The input knowledge required to understand this message is substantial. The reader must know:

Assumptions and Potential Mistakes

The assistant makes several assumptions in this read:

  1. That the FFI interface is the right place to change. An alternative would be to keep the single-mutex FFI and manage the second lock entirely within C++, hidden behind the same opaque pointer. The assistant assumes that exposing both locks through FFI (or a struct containing both) is necessary.
  2. That the Rust side needs to know about both locks. In fact, the Rust side never uses the mutex directly — it just passes the pointer. The assistant could have created a C++ struct containing both mutexes and kept the same single-pointer FFI, changing only the C++ side. The read of lib.rs is partly to verify whether the Rust side does anything with the mutex beyond passing it.
  3. That the existing pattern is the right template. The assistant assumes that creating two mutexes follows the same pattern as creating one. This is correct but misses a subtlety: the two locks need different lifetimes and different ownership semantics. The compute_mtx is held for the duration of kernel execution, while mem_mtx is held only during allocation/upload. The existing single-mutex pattern doesn't distinguish these phases.
  4. That the opaque pointer approach is sufficient. For the two-lock design, passing a struct with two mutexes as a single opaque pointer is cleaner than passing two separate pointers. The assistant's read confirms that the Rust side treats the pointer as fully opaque, so either approach works. A potential mistake in the reasoning chain is the assumption that splitting locks will yield the ~600ms gain the user identified. The gain depends on the CPU b_g2_msm and epilogue (which run outside any lock) overlapping with the next worker's GPU compute (which runs inside compute_mtx). But if the CPU work is memory-bandwidth-bound and the GPU compute is also competing for PCIe bandwidth or memory bandwidth, the overlap may not be clean. The assistant does not yet have benchmark data for this scenario — it's working from a theoretical model.

The Thinking Process

The assistant's thinking process in this message is not explicitly visible (there is no reasoning block), but it is strongly implied by the sequence of actions. The assistant is systematically gathering information before committing to code changes. The pattern is:

  1. Understand the problem (Phase 9 bottleneck analysis, [msg 2568])
  2. Engage with the proposal (user's two-lock idea, <msg id=2569-2570>)
  3. Analyze constraints (deadlock analysis, VRAM tracing, <msg id=2574-2575>)
  4. Survey the codebase (engine.rs for worker creation, groth16_cuda.cu for lock usage, lib.rs for FFI boundary) This is classic defensive engineering: before touching any code, understand all the interfaces that will need to change. The assistant is building a mental map of the full change surface. The read of lib.rs is particularly strategic. The FFI boundary is the most constrained part of the system — changing it requires modifying both Rust and C++ code in lockstep, and any mismatch causes linker errors or undefined behavior. By reading the exact declarations, the assistant ensures it knows the precise function signatures it needs to replicate or modify.

Significance

Message [msg 2579] is a small but essential step in a complex optimization. It exemplifies a key engineering virtue: the willingness to slow down and verify before building. The assistant could have jumped straight to implementation, assuming the FFI interface was as expected. Instead, it read the actual file, confirming the exact declarations and the surrounding context.

This read also reveals the multi-language nature of the optimization. The system spans Rust (async engine), C++ (CUDA wrapper), and CUDA C++ (GPU kernels). Each language boundary is a potential source of bugs. The FFI boundary between Rust and C++ is particularly fragile because it relies on ABI compatibility, pointer lifetimes, and manual memory management. Reading the exact declarations is not pedantry — it is a necessary risk-reduction step.

In the broader narrative of the optimization campaign, this message marks the transition from analysis to implementation. The assistant has spent several messages reasoning about deadlocks, tracing VRAM allocations, and surveying code. With the FFI interface confirmed, the next step is to write the actual code: the gpu_locks struct, the new FFI functions, the modified worker creation logic, and the restructured lock regions in the CUDA kernel launcher. The read of lib.rs is the last piece of reconnaissance before the build phase begins.