The Compiler That Wouldn't Send: A Raw Pointer Standoff in Rust's Async Proving Engine

Introduction

In the high-stakes world of Filecoin proof-of-replication (PoRep) proving, every millisecond counts. The cuzk SNARK proving engine, a sophisticated piece of infrastructure for generating Groth16 proofs at scale, was undergoing a major architectural transformation: Phase 8: Dual-Worker GPU Interlock. This optimization aimed to eliminate GPU idle gaps by narrowing a C++ static mutex and allowing two GPU workers per device to interleave their CPU preprocessing with CUDA kernel execution. But before any benchmark could be run, before any throughput improvement could be measured, the Rust compiler had to be satisfied. And it was not.

Message [msg 2209] captures a single moment in this implementation journey: a failed build attempt. It is a short message—just a bash command and its truncated error output—but it represents a critical diagnostic juncture where the assistant's assumptions about Rust's type system collided with the language's strict concurrency safety guarantees.

The Message

The assistant executed:

cd /home/theuser/curio/extern/cuzk && cargo build --release -p cuzk-daemon 2>&1 | tail -30

And the compiler responded with an error:

     |                         --------------------------- ^-------------------------------------
     |                         |                           |
     |  _______________________|___________________________within this `{closure@cuzk-core/src/engine.rs:1354:61: 1354:99}`
     | |                       |
     | |                       required by a bound introduced by this call
1355 | | ...                       std::env::set_var("CUDA_VISIBLE_DEVICES", &gpu_str);
1356 | | ...         ...

The error, though truncated in the tail -30 output, is unmistakably about the Send trait. The compiler is complaining that a closure passed to tokio::spawn_blocking does not satisfy the Send requirement. The closure at engine.rs:1354:61—inside a spawn_blocking call within a tokio::spawn async block—captures something that cannot be sent to another thread.

The Reasoning and Context

To understand why this message was written, we must trace the chain of reasoning that led to it.

The Architecture Problem

The Phase 8 design required multiple GPU workers per device, all sharing a single C++ std::mutex that guards access to the CUDA kernel region (NTT+MSM operations). This mutex is allocated on the C++ side and returned to Rust as an opaque pointer (*mut c_void). The pointer must be passed through several layers:

1. C++ groth16_cuda.cu → FFI boundary
2. supraseal-c2/src/lib.rs (Rust FFI wrappers)
3. bellperson (Rust proving library)
4. cuzk-core/src/pipeline.rs (the gpu_prove function)
5. cuzk-core/src/engine.rs (the async engine that spawns workers) The assistant had already completed steps 1–4 and was working on step 5: updating the engine to create one C++ mutex per GPU and spawn gpu_workers_per_device workers per GPU, each receiving the same mutex pointer.

The First Build Failure

In [msg 2206], the first build attempt failed with a Send error. The assistant correctly diagnosed the problem in [msg 2207]: the SendableGpuMutex wrapper type is marked Send, but the code extracted the raw pointer via gpu_mutex.0 before passing it into the spawn_blocking closure. The extracted *mut c_void is not Send.

The assistant's fix in [msg 2208] was to move the entire SendableGpuMutex into the closure instead of extracting the pointer:

// Before:
let gpu_mtx_ptr = gpu_mutex.0;
// ... later in spawn_blocking:
let gpu_mtx_ptr = gpu_mtx_ptr;

// After (attempted fix):
// Move the whole SendableGpuMutex into the closure

The Second Build Failure (Message 2209)

Message [msg 2209] is the result of this attempted fix. The build still fails. The compiler error points to the same location—the closure at line 1354—still not Send. The truncated error output shows the compiler's diagnostic arrows pointing at the closure within spawn_blocking.

The assistant's assumption was that moving the SendableGpuMutex wrapper (which is Send) into the closure would satisfy the compiler. But the error reveals that this assumption was incorrect. The SendableGpuMutex struct contains a *mut c_void field, and even though the struct implements Send, the compiler still sees the raw pointer when the closure captures the struct and then accesses its inner field.

The Deeper Problem: Rust's Send Semantics and Raw Pointers

This error touches on a subtle aspect of Rust's concurrency safety model. The Send trait indicates that a type's ownership can be safely transferred between threads. Raw pointers (*mut T, *const T) are explicitly not Send because Rust cannot guarantee memory safety across threads without additional synchronization.

The SendableGpuMutex wrapper was designed to paper over this:

#[derive(Clone)]
pub struct SendableGpuMutex(pub *mut c_void);
// Safety: this mutex is only used from one thread at a time due to external locking
unsafe impl Send for SendableGpuMutex {}

But the problem is subtle: even though SendableGpuMutex is Send, the closure captures the struct and then the compiler examines how the captured variable is used. If the closure accesses .0 to extract the raw pointer, the compiler may still reject it because the closure's generated type contains a raw pointer field.

The exact mechanics depend on how the closure is constructed. In [msg 2208], the assistant moved the entire SendableGpuMutex into the outer async block's capture, but the spawn_blocking inner closure still needed access to the raw pointer. The error shows that the inner closure's captures still transitively included the raw pointer.

Input Knowledge Required

To fully understand this message, one needs:

1. Rust's Send trait and its interaction with closures: Knowing that closures implement Send only if all captured variables are Send, and that the compiler examines the closure's capture set structurally.
2. tokio::spawn and spawn_blocking semantics: tokio::spawn requires the future to be Send because it may be moved between threads. spawn_blocking similarly requires the closure to be Send.
3. Raw pointer safety in Rust: *mut c_void is not Send or Sync, requiring explicit unsafe marker trait implementations on wrapper types.
4. The FFI pattern for sharing C++ objects: Passing an opaque pointer across language boundaries and the need to share it across threads.
5. The cuzk proving engine architecture: Understanding that GPU workers are spawned as async tasks, each running a spawn_blocking closure that calls into C++ CUDA code via FFI.

Output Knowledge Created

This message produced a concrete piece of knowledge: the attempted fix of moving SendableGpuMutex into the closure was insufficient. The compiler still rejected the code. This negative result is valuable because it narrows the solution space.

The error output, though truncated, provides specific diagnostic information:

• The problematic closure is at engine.rs:1354:61 to 1354:99
• The Send bound is required by tokio::spawn at line 1251
• The closure is inside a spawn_blocking call This information guides the next iteration of fixes. Indeed, in the subsequent messages ([msg 2210] through [msg 2213]), the assistant tries progressively more aggressive approaches:
• Converting the pointer to usize (which is Send) and casting back
• Capturing the mutex address as usize before entering the async block
• Passing the usize through the entire chain

Assumptions and Their Corrections

Several assumptions were tested and corrected through this process:

Assumption 1: SendableGpuMutex being Send would be sufficient to pass it through async boundaries. Correction: The compiler examines the closure's capture set structurally, and accessing the inner raw pointer still triggers the Send check on the raw pointer type.

Assumption 2: Moving the wrapper into the closure rather than extracting the pointer would solve the problem. Correction: The closure still needs to access the pointer value to pass it to the C++ function, which requires extracting .0 at some point.

Assumption 3: The error was about the spawn_blocking closure specifically. Correction: The error actually originates from the outer tokio::spawn async block (line 1251), which requires the entire async block's captured environment to be Send. The spawn_blocking closure is nested inside, and its captures contribute to the outer block's capture set.

The Thinking Process

The assistant's reasoning in the messages surrounding [msg 2209] reveals a systematic debugging approach:

1. Observation: Build fails with Send error (message 2206)
2. Diagnosis: Raw pointer *mut c_void extracted from SendableGpuMutex isn't Send (message 2207)
3. Hypothesis: Moving the wrapper struct instead of extracting the pointer will satisfy the compiler (message 2208)
4. Test: Rebuild (message 2209) → Hypothesis fails
5. Refined diagnosis: The closure captures the struct but still accesses .0, exposing the raw pointer (message 2210)
6. New hypothesis: Convert to usize (which is Send) and cast back (message 2210)
7. Further refinement: Capture as usize before the async block to avoid any raw pointer in captures (message 2212)
8. Final fix: Success (message 2214) This is classic compiler-driven development: the compiler's error messages guide the developer toward a correct solution, with each iteration refining the understanding of the type system's requirements.

Broader Significance

While message [msg 2209] is just one build failure in a long optimization session, it illustrates a fundamental tension in systems programming: the need to share opaque resources (like C++ mutexes) across async boundaries while satisfying Rust's strict safety guarantees. The Send trait, designed to prevent data races, becomes a significant obstacle when interfacing with foreign code that has its own synchronization mechanisms.

The solution ultimately adopted—converting the pointer to a usize integer before entering the async context and casting it back inside—is a pragmatic workaround. It exploits the fact that usize is Send (it's just a number) while acknowledging that the safety responsibility has been transferred from the compiler to the programmer. This pattern is common in high-performance Rust systems that interface with C/C++ libraries, where the Rust type system's guarantees must occasionally be circumvented for practical interoperation.

Conclusion

Message [msg 2209] captures a moment of negative feedback in an optimization journey. A build failure, a truncated error message, and a hypothesis proven wrong. But in that failure lies valuable information: the compiler's rejection of the SendableGpuMutex approach forced the assistant to dig deeper into Rust's closure capture semantics and find a more robust solution. The Phase 8 dual-worker GPU interlock would eventually be implemented successfully, benchmarked at 13–17% throughput improvement, and committed to the repository. But before any of that, there was this moment—a raw pointer standing defiantly against the Send bound, and a developer reading the compiler's output, learning, and iterating.