Reading the Rust–C++ Mutex Boundary: A Critical Step in Designing Phase 10's Two-Lock GPU Interlock

Introduction

In the course of optimizing a high-throughput Groth16 proving pipeline for Filecoin's Proof-of-Replication (PoRep), a single message stands out as a quiet but pivotal moment of architectural investigation. The message at index 2581 is deceptively simple: it is a [read] command issued by the AI assistant to inspect the file /home/theuser/curio/extern/bellperson/src/groth16/prover/supraseal.rs, specifically the section defining the GPU mutex wrapper types on the Rust side of the FFI boundary. This read is not a casual glance—it is a deliberate act of knowledge-gathering, performed in service of designing Phase 10 of a multi-phase optimization campaign. To understand why this read matters, one must understand the long chain of reasoning that led to it, the bottleneck that precipitated it, and the delicate cross-language synchronization architecture it seeks to modify.

The Bottleneck That Changed Everything

The story begins with Phase 9, a PCIe transfer optimization that achieved a dramatic 51% reduction in GPU kernel time—from 3.75 seconds per partition down to 1.82 seconds. This was a genuine victory, but it revealed an uncomfortable truth: the bottleneck had simply shifted. The GPU was now finishing its work 600 milliseconds before the CPU thread could feed it the next partition. The critical path had moved from GPU kernel execution to CPU-side memory bandwidth contention. Specifically, the prep_msm function (which splits circuit vectors) consumed 1.91 seconds per partition, and b_g2_msm (a G2 MSM operation) consumed another 0.48 seconds—together forming a 2.39-second CPU critical path that left the GPU idle for nearly a quarter of each partition cycle.

At high concurrency (15+ concurrent proof requests), the situation worsened. The 10 synthesis workers, all competing for the same 8-channel DDR5 memory bandwidth, inflated CPU times by 12–27%, pushing steady-state throughput to ~41 seconds per proof instead of the ~32 seconds observed in isolation. The assistant diagnosed this precisely: "The real wall is DDR5 bandwidth, which we can't fix from CUDA code."

The User's Proposal: A Two-Lock Architecture

Faced with this CPU-memory-bandwidth wall, the user proposed a structural change to the GPU worker architecture (msg 2569). The existing Phase 8 design used a single std::mutex per GPU, which serialized CUDA kernel execution across workers but allowed CPU preprocessing to overlap. The user's insight was that by splitting this single lock into two—a compute_mtx for GPU kernel execution and a mem_mtx for VRAM allocation and upload—more workers could be added (3–4 instead of 2), allowing one worker to run GPU kernels while another handled memory management and a third performed CPU-only preprocessing. The goal was to hide the CPU overhead behind additional GPU work, effectively increasing throughput by better utilizing the GPU while the CPU struggled with memory bandwidth.

The Subject Message: Why Read supraseal.rs?

This brings us to message 2581. The assistant is in the middle of a multi-step investigation. It has already read the CUDA kernel code (groth16_cuda.cu), traced the VRAM allocation lifecycle, analyzed deadlock scenarios, and received the user's design guidance. Now it needs to understand the Rust-side plumbing that connects the CUDA proving engine to the higher-level Rust orchestration layer.

The file being read, supraseal.rs, defines the FFI boundary for the GPU mutex. The relevant code shows:

pub type GpuMutexPtr = *mut std::ffi::c_void;

/// Marker: GpuMutexPtr is `Send` because it points to a heap-allocated
/// C++ std::mutex that outlives all workers.
#[derive(Clone, Copy)]
pub struct SendableGpuMutex(pub GpuMutexPtr);
unsafe impl Send for SendableGpuMutex {}
unsafe impl Sync for SendableGpuMutex {}

This is a carefully constructed wrapper. The GpuMutexPtr is an opaque pointer to a C++ std::mutex allocated on the C++ heap. The SendableGpuMutex struct wraps this pointer and explicitly implements Send and Sync—Rust's marker traits for thread safety. The Send implementation is justified by a comment explaining that the underlying C++ mutex outlives all workers, meaning it is safe to transfer ownership between threads. The Sync implementation is more subtle: it asserts that it is safe to share references to the mutex across threads, which is true because std::mutex is inherently designed for shared access.

The assistant is reading this code to answer a critical design question: how hard will it be to replace the single mutex pointer with a two-lock struct? The current architecture passes one opaque pointer per GPU. A two-lock design would need to pass either two opaque pointers or a single pointer to a struct containing two mutexes. The FFI boundary must change, and the Rust-side wrapper must be updated accordingly.

Assumptions Embedded in the Current Design

The existing code reveals several assumptions that the Phase 10 design must challenge:

  1. Single mutex is sufficient. The entire GPU interlock was designed around the premise that serializing CUDA kernel execution is the only synchronization needed. The SendableGpuMutex type encodes this assumption in its very name—it is a mutex, singular.
  2. The mutex outlives all workers. This is stated explicitly in the comment and is critical for safety. The C++ mutex is allocated once per GPU and never destroyed while workers exist. A two-lock design must maintain this guarantee for both locks.
  3. Thread safety can be asserted at the type level. The unsafe impl Send and unsafe impl Sync are manual assertions that bypass Rust's compile-time checking. Any change to the mutex structure must be accompanied by a careful re-evaluation of these safety guarantees.
  4. The opaque pointer pattern is sufficient. The current design hides the C++ mutex behind a *mut c_void, keeping the Rust side ignorant of the C++ implementation details. A two-lock struct could be hidden behind the same pattern, but the Rust side would need to know whether to pass one pointer or two.

The Thinking Process Visible in This Message

Although the message itself is just a read command, it sits within a rich context of reasoning. In the preceding messages (2570–2580), the assistant has:

Input Knowledge Required to Understand This Message

To fully grasp the significance of this read, one must understand:

  1. The Groth16 proving pipeline for Filecoin PoRep. The system generates proofs across multiple partitions, each requiring substantial GPU computation (NTT, MSM) and CPU preprocessing (vector splitting, G2 MSM).
  2. The multi-phase optimization history. Phase 8 introduced dual-worker GPU interlock with a single mutex, achieving 2.4x baseline throughput. Phase 9 added PCIe pre-staging, cutting GPU kernel time but shifting the bottleneck to CPU memory bandwidth.
  3. CUDA memory management. VRAM allocation (cudaMalloc), pool trimming (cudaMemPoolTrimTo), and asynchronous transfers are all device-global operations that cannot be fully isolated from kernel execution on a single GPU.
  4. Rust's FFI and thread safety model. The Send and Sync traits, opaque pointer patterns, and the discipline of unsafe impls are essential for understanding why the mutex wrapper is designed as it is.
  5. Resource deadlock vs. lock-ordering deadlock. The assistant's analysis distinguished between a classic lock-ordering deadlock (which can be prevented by consistent acquisition order) and a resource deadlock (where VRAM capacity, not lock ordering, is the constraint).

Output Knowledge Created by This Message

This read produces several forms of knowledge:

  1. Confirmation of the FFI boundary structure. The assistant now knows exactly how the mutex pointer flows from C++ allocation through Rust's SendableGpuMutex to the worker tasks.
  2. Identification of the extension point. The SendableGpuMutex type is the natural place to add a second lock. The pattern of wrapping an opaque pointer in a Send+Sync struct can be replicated for a two-lock struct.
  3. Awareness of the safety documentation. The comment explaining why Send is safe ("outlives all workers") provides the template for documenting the two-lock variant.
  4. Basis for the implementation plan. With this information, the assistant can proceed to design the two-lock struct in C++, update the create_gpu_mutex FFI function, modify SendableGpuMutex to hold two pointers, and adjust the engine code that distributes mutexes to workers.

What Happens Next

The very next message (msg 2582) shows the assistant reading the engine code to understand how the mutex pointer is passed to workers. This confirms that the investigation is proceeding methodically: first the CUDA kernel code, then the VRAM lifecycle, then the Rust FFI wrapper, and finally the engine orchestration layer. Each read builds on the previous ones, creating a complete mental model of the synchronization architecture before any code is written.

The Phase 10 implementation that follows (in chunk 1 of segment 27) will introduce a gpu_locks struct with mem_mtx and compute_mtx, increase gpu_workers_per_device to 3, and restructure the lock regions. But it will also encounter a fundamental conflict: CUDA device-global synchronization operations inside mem_mtx will effectively serialize the two locks, causing OOM failures and performance regressions. The careful investigation visible in message 2581—understanding the FFI boundary, the thread safety assumptions, and the pointer-passing mechanics—is what makes it possible to diagnose and ultimately resolve that conflict.

Conclusion

Message 2581 is a textbook example of how optimization work in a complex, cross-language system requires meticulous architectural reconnaissance. The read command itself is trivial—a few lines of code displayed—but the context that motivates it is rich with design reasoning, bottleneck analysis, and deadlock prevention. The assistant is not merely reading code; it is building a complete understanding of the synchronization boundary between Rust and C++, evaluating extension points, and preparing to modify a carefully designed thread-safety contract. In the broader narrative of the optimization campaign, this message represents the moment when the design of Phase 10 shifts from conceptual analysis to implementation planning—a quiet but essential transition that determines whether the next phase succeeds or fails.