The Raw Pointer That Wouldn't Cross the Async Boundary: Debugging Rust's Send Constraint in a GPU Proving Engine

Introduction

In the midst of implementing Phase 8: Dual-Worker GPU Interlock for the cuzk SNARK proving engine, a seemingly straightforward build command produced a compilation error that exposed a subtle tension between Rust's type system and the realities of cross-language FFI. Message <msg id=2211> captures this moment: the assistant runs cargo build --release -p cuzk-daemon and is greeted by a truncated compiler error pointing at a tokio::spawn call, where the async block fails the Send bound. The error message trails off into an ellipsis, but the problem is already clear to anyone following the conversation: a raw C++ mutex pointer, wrapped in a Send-marked newtype, is leaking through Rust's concurrency abstractions and violating the type system's guarantees.

This message is a snapshot of a debugging process that spans five rapid iterations ([msg 2208] through [msg 2213]), each attempting to thread a *mut c_void pointer through increasingly nested layers of asynchronous Rust without triggering the compiler's Send checker. To understand why this single build output matters, we must trace the path that led to it — a path that begins in C++ CUDA kernels, crosses three FFI boundaries, and ends in a tokio::spawn call deep inside the engine's GPU worker loop.

The Architecture Behind the Error

The Phase 8 dual-worker GPU interlock was designed to solve a specific performance problem. In earlier phases of the cuzk proving engine, a static std::mutex inside the C++ generate_groth16_proofs_c function serialized all GPU access, creating idle gaps where one GPU worker held the lock while performing CPU-side preprocessing, and other workers waited. The insight of Phase 8 was to narrow the mutex scope to cover only the CUDA kernel region (NTT, MSM, batch additions, tail MSMs), allowing CPU preprocessing and the b_g2_msm computation to run outside the lock. With two workers per GPU, one could perform CPU work while the other held the mutex for kernel execution, achieving perfect interleaving.

This architectural change required threading a C++ std::mutex* pointer through the entire call chain:

  1. C++ layer (groth16_cuda.cu): The static mutex was removed, and a std::mutex* parameter was added to the kernel entry point. Two helper functions — alloc_gpu_mutex and destroy_gpu_mutex — were added to create and destroy mutexes on the C++ side.
  2. FFI layer (supraseal-c2/src/lib.rs): The extern C declarations and Rust wrapper functions were updated to accept and pass the opaque *mut c_void pointer.
  3. Bellperson layer: The prove_from_assignments and create_proof_batch_priority_inner functions gained a gpu_mutex parameter.
  4. Pipeline layer (pipeline.rs): The gpu_prove function and its internal callers were updated.
  5. Engine layer (engine.rs): The GPU worker spawn loop was rewritten to create per-GPU mutexes and spawn multiple workers sharing the same mutex. The SendableGpuMutex wrapper type was introduced to make the raw pointer usable across threads:
#[derive(Clone)]
pub struct SendableGpuMutex(pub *mut c_void);

// Safety: the mutex is accessed under exclusive C++ locking
unsafe impl Send for SendableGpuMutex {}
unsafe impl Sync for SendableGpuMutex {}

This is a classic FFI pattern: the Rust compiler cannot know whether a raw pointer is safe to transfer between threads, so the programmer asserts safety via unsafe impl Send. The assumption is that the underlying C++ std::mutex provides its own thread safety, and the pointer itself is merely a handle.

The Two-Front Send Battle

The build error in <msg id=2211> reveals that this assumption was not sufficient. The compiler error points to line 1251 of engine.rs:

1251 |                   tokio::spawn(async move {
     |                   ------------ ^---------
     |                   |            |
     |  _________________|____________within this `{async block@...}`
     | |                 |
     | |                 required by a bound introduced by this call
1252 | |                     info!(worker_id = worker_id, gpu = gpu_ordinal, sub_id = worker_sub_id, "pipeline GPU worker started");
1253 | |                ...

The tokio::spawn function requires the future it receives to implement Send, because the spawned task may be moved to a different thread by the Tokio runtime. The async move block captures gpu_mutex — a SendableGpuMutex that contains a *mut c_void. Even though the wrapper type is explicitly marked Send, the compiler still sees the raw pointer field inside the closure's capture set, and the error propagates.

This is a subtle point of Rust's type system: marking a struct Send makes the struct itself Send, but when the struct is captured by value in a closure, the compiler generates an anonymous type for the closure's capture set. The compiler looks at the fields of the captured struct, not just the struct's trait impl, when determining whether the closure is Send. In some cases — particularly with async move blocks and nested closures — the compiler's analysis can "see through" the wrapper and flag the inner raw pointer.

The assistant's first fix attempt ([msg 2208]) tried to move the entire SendableGpuMutex into the inner spawn_blocking closure rather than extracting the raw pointer. But this failed ([msg 2209]) because the inner closure also required Send, and the raw pointer inside the wrapper still caused the same issue at a deeper nesting level.

The second fix ([msg 2210]) converted the pointer to a usize (which is trivially Send) and cast it back inside the closure:

let gpu_mtx_val = gpu_mutex.0 as usize;
// ... inside spawn_blocking:
let gpu_mutex_ptr = gpu_mtx_val as *mut c_void;

But this only addressed the inner spawn_blocking closure. The outer tokio::spawn(async move { ... }) block still captured the SendableGpuMutex directly, triggering the error shown in <msg id=2211>.

The Reasoning Process

The assistant's thinking, visible across messages [msg 2208] through [msg 2213], reveals a methodical debugging process driven by compiler feedback:

  1. Observation: The spawn_blocking closure captures gpu_mtx_ptr (a raw *mut c_void), which isn't Send.
  2. Hypothesis 1: Moving the whole SendableGpuMutex (which is Send) into the closure will fix it, because the raw pointer is never extracted outside the closure.
  3. Test: Build fails — the closure still captures the raw pointer field.
  4. Hypothesis 2: Convert the pointer to usize before capture, cast back inside.
  5. Test: Build fails again — but now at the outer tokio::spawn, not the inner closure.
  6. Refined understanding: The async move block itself captures SendableGpuMutex, and the compiler sees through the wrapper.
  7. Hypothesis 3: Convert to usize before entering the async block, so the async block captures only a usize.
  8. Test: Success — both the outer and inner closures now capture only integer values. This is textbook Rust debugging: each compiler error reveals one more layer of the type-checking onion. The assistant correctly identifies that the Send impl on the wrapper struct is not enough — the compiler's closure capture analysis is field-level, not type-level, for the purposes of auto-trait derivation on the anonymous closure type.

Assumptions and Corrections

Several assumptions were made during this debugging session, and each was corrected by the compiler:

| Assumption | Reality | Correction | |---|---|---| | SendableGpuMutex marked Send makes closures capturing it Send | The compiler sees through the wrapper to the raw pointer field | Convert to usize before capture | | Only the inner spawn_blocking closure has the issue | The outer tokio::spawn async block has the same problem | Apply the same usize conversion at the outermost capture point | | The usize conversion fix for the inner closure is sufficient | The outer async block captures SendableGpuMutex first | Move the conversion before the async block |

The key insight is that Rust's auto-trait derivation for closure types does not simply check whether captured values implement Send. For structs with manual unsafe impl Send, the compiler may still inspect the struct's fields when determining the auto-trait impls of the anonymous closure type. This is a known edge case in Rust's type system, documented in the language reference as "auto-trait leakage" — the compiler can see through newtype wrappers for the purposes of auto-trait satisfaction on generated types.

Input Knowledge Required

To understand this message, a reader needs:

  1. Rust's Send trait: Understanding that Send indicates a type can be safely transferred between threads, and that tokio::spawn requires Send futures.
  2. FFI and raw pointers: Knowledge that *mut c_void is the Rust representation of an opaque C/C++ pointer, and that raw pointers are neither Send nor Sync by default.
  3. The cuzk architecture: Awareness that the proving engine uses a multi-layer FFI chain (C++ → supraseal-c2 → bellperson → cuzk-core) and that a C++ std::mutex must be shared across threads.
  4. Async Rust patterns: Understanding tokio::spawn, async move blocks, and spawn_blocking for running blocking code in async contexts.
  5. Closure capture semantics: Knowing that move closures capture variables by value, and that the compiler generates anonymous types for capture sets.

Output Knowledge Created

This message produces several forms of knowledge:

  1. A failed build artifact: The compiler error serves as a concrete record of a type-checking failure, documenting the exact location and nature of the problem.
  2. Evidence of a multi-layer Send issue: The error at line 1251 (outer tokio::spawn) combined with the earlier error at line 1354 (inner spawn_blocking) proves that the Send constraint must be satisfied at two nesting levels.
  3. A debugging trajectory: The sequence of build attempts ([msg 2209], [msg 2211], [msg 2214]) creates a trail of compiler feedback that guides the assistant toward the correct fix.
  4. A reusable pattern: The as usize / as *mut c_void cast pattern for threading raw pointers through async boundaries becomes a documented technique for future FFI work in this codebase.

The Resolution

The fix that ultimately succeeds ([msg 2212]–[msg 2213]) converts the mutex pointer to a usize before entering the async move block, so the async block captures only a plain integer:

let gpu_mutex_val = gpu_mutex.0 as usize;
// ...
tokio::spawn(async move {
    // gpu_mutex_val is a usize — trivially Send
    // ...
    let gpu_mutex_ptr = gpu_mutex_val as *mut c_void;
    // ...
});

With this change, both the outer tokio::spawn and the inner spawn_blocking capture only usize values, which are trivially Send. The build succeeds ([msg 2214]), and the daemon and bench tool both compile cleanly ([msg 2215]). The Phase 8 implementation is ready for benchmarking.

Conclusion

Message <msg id=2211> is a small but revealing moment in a larger engineering effort. It captures the exact point where a carefully designed FFI abstraction meets the unforgiving reality of Rust's type system. The SendableGpuMutex wrapper was a reasonable attempt to make a C++ mutex pointer usable across threads, but it failed to account for the compiler's field-level analysis of closure capture sets. The debugging process that follows — five rapid iterations of hypothesis, test, and refinement — demonstrates a systematic approach to type-level debugging that is characteristic of experienced Rust developers.

More broadly, this message illustrates a recurring theme in systems programming: the boundary between safe and unsafe code is never as clean as we hope. Every unsafe impl is a promise to the compiler, but the compiler may hold us to that promise in unexpected ways. The usize cast pattern that resolves this issue is itself a form of "lying to the compiler" — we erase the type information and restore it later — but it is a controlled, local lie that makes the larger truth (the C++ mutex is genuinely thread-safe) expressible within Rust's type system.

The build error in <msg id=2211> is not a failure. It is the compiler doing its job, enforcing the safety guarantees that make Rust suitable for high-performance, concurrent systems programming. And the assistant's response — methodical, compiler-driven, and ultimately successful — is exactly the kind of engineering discipline that such systems demand.