The Narrowed Mutex: How Phase 8's C++ Refactor Unlocked GPU Interleaving in the cuzk SNARK Prover

Introduction

In high-performance GPU-accelerated proving systems, every microsecond of GPU idle time is wasted compute capacity. When the cuzk SNARK proving engine for Filecoin's PoRep (Proof-of-Replication) was benchmarked in Phase 7, a critical bottleneck emerged: the GPU was sitting idle for significant periods while CPU-side preprocessing ran under a coarse-grained static mutex. The fix—Phase 8's Dual-Worker GPU Interlock—hinged on a single, precisely targeted message that refactored the C++ CUDA kernel in groth16_cuda.cu. This message, <msg id=2151>, represents the foundational code change that enabled two GPU workers per device to interleave their work, eliminating GPU idle gaps and achieving a 13–17% throughput improvement.

The Message

The assistant wrote:

## Step 1: C++ Mutex Refactor in groth16_cuda.cu

>

The key changes: 1. Add std::mutex* parameter to the function signature 2. Remove the static mutex (lines 132-134) 3. Acquire lock BEFORE per-GPU thread launch (line 571), AFTER prep_msm_thread launch (line 194) 4. Release lock AFTER per-GPU threads join, BEFORE prep_msm_thread.join() 5. Reorder joins: per-GPU first, then prep_msm [edit] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu Edit applied successfully.

This is a dense, surgical description of a structural change to the core Groth16 proof generation function. At first glance, it appears to be a simple mutex refactor. In reality, it embodies the entire architectural insight of Phase 8: that the static mutex in generate_groth16_proofs_c was protecting too much code, forcing sequential execution of CPU and GPU work that could safely overlap.

Context: The GPU Idle Gap Problem

To understand why this message was written, we must trace back to the Phase 7 benchmarks ([msg 2146]). Phase 7 introduced per-partition dispatch, which broke a single proof's partitions into individually schedulable units. This allowed the engine to begin GPU work on one partition while another partition's CPU synthesis was still running. However, benchmark results revealed a persistent GPU utilization gap: even with two proofs in flight, the GPU was idle ~15% of the time.

The root cause, diagnosed in the Phase 8 design document, was a static std::mutex inside the C++ generate_groth16_proofs_c function. This mutex was acquired at the very beginning of the function and held until the very end, covering both CPU preprocessing (the b_g2_msm computation and other host-side work) and the CUDA kernel region (NTT, MSM, batch additions, tail MSMs). Because the mutex was static (shared across all threads), only one thread could execute generate_groth16_proofs_c at a time—even if the GPU was capable of running kernels from multiple threads concurrently.

The consequence was stark: when Worker A held the mutex and was running CUDA kernels, Worker B could not even begin its CPU preprocessing. Worker B had to wait until Worker A released the mutex, which only happened after all GPU work completed. This created a structural serialization that left the GPU idle between Worker A's last kernel and Worker B's first kernel, because Worker B's CPU preprocessing had to run after Worker A released the mutex but before Worker B could launch its own CUDA kernels.

The Reasoning Behind the Five Changes

The assistant's five-point plan is a masterclass in minimal, targeted refactoring. Each point addresses a specific aspect of the problem:

1. Add std::mutex* parameter to the function signature

The static mutex was a global variable declared inside the function body. By replacing it with a passed-in pointer, the mutex becomes an external resource that the caller can manage. This is critical for two reasons: (a) it allows the Rust side to allocate a mutex per GPU (via create_gpu_mutex/destroy_gpu_mutex helpers), enabling per-device locking rather than a single global lock; and (b) it makes the mutex's lifetime explicit, avoiding the pitfalls of static initialization order.

2. Remove the static mutex (lines 132-134)

The original code declared a static mutex at lines 132–134, just inside the function. Removing it is the logical consequence of change #1—the mutex is now provided by the caller, so the static declaration is dead code.

3. Acquire lock BEFORE per-GPU thread launch, AFTER prep_msm_thread launch

This is the heart of the narrowed scope. The original code acquired the lock at function entry, before any work began. The new code launches the prep_msm_thread (which handles CPU preprocessing, including the b_g2_msm computation) without holding the lock. The lock is acquired only when the per-GPU threads are about to be launched—that is, when the CUDA kernel region begins.

The ordering is precise: prep_msm_thread is launched first (line 194 in the original), then the lock is acquired, then per-GPU threads are launched (line 571). This ensures that CPU preprocessing runs in parallel with another worker's GPU work, because the lock is not held during preprocessing.

4. Release lock AFTER per-GPU threads join, BEFORE prep_msm_thread.join()

Symmetrically, the lock is released as soon as the per-GPU threads (which run the CUDA kernels) have completed, but before waiting for prep_msm_thread to finish. This means that while one worker is joining its prep_msm_thread (a fast operation since the thread is likely already done), another worker can acquire the lock and begin its own CUDA kernel region.

5. Reorder joins: per-GPU first, then prep_msm

This is a subtle but important detail. The original code likely joined prep_msm_thread first (since it was launched first), then the per-GPU threads. By reordering to join per-GPU threads first, the lock can be released earlier, and the prep_msm_thread.join() runs outside the lock, allowing maximum overlap.

Assumptions Made

The assistant made several assumptions in this message:

  1. The static mutex was the sole cause of GPU idle gaps. This assumption was validated by the Phase 7 benchmarks, which showed that GPU utilization correlated with mutex contention. However, it assumes that no other synchronization primitive (e.g., implicit CUDA stream synchronization, driver-level serialization) contributed to the gap.
  2. CPU preprocessing (b_g2_msm and related work) is safe to run outside the lock. This assumes that the CPU preprocessing does not access any shared GPU state that would require synchronization. The assistant implicitly trusts that the prep_msm_thread only touches data that is either thread-local or properly synchronized by other means.
  3. The mutex pointer will be valid for the duration of the call. Since the mutex is now passed from Rust, the assistant assumes the Rust side will manage its lifetime correctly—allocating it before any worker threads use it and destroying it only after all workers have stopped.
  4. Two workers per GPU is sufficient. The design targets two GPU workers per device. This assumes that the CPU preprocessing time is roughly equal to the GPU kernel time, so two workers can perfectly interleave. If the ratio were different, more or fewer workers might be needed.

Potential Mistakes and Incorrect Assumptions

While the refactor is sound, there are subtle risks:

  1. The prep_msm_thread may access shared mutable state. If the CPU preprocessing writes to buffers that the CUDA kernels later read, and those buffers are not properly synchronized, running preprocessing outside the lock could introduce data races. The assistant assumes this is safe, but the assumption depends on the specific implementation of prep_msm.
  2. Lock acquisition order and fairness. The mutex is a plain std::mutex, which does not guarantee fairness. If one worker holds the lock for a long time (e.g., due to a large batch), another worker could starve. The assistant implicitly assumes that GPU kernel times are roughly equal across partitions, so starvation is unlikely.
  3. The reordered joins could mask errors. If a per-GPU thread throws an exception, the prep_msm_thread join (now happening later) might not be reached if the exception propagates. However, the existing error handling (via RustError return codes) likely mitigates this.
  4. The assistant did not verify the exact line numbers after the edit. The message states "Edit applied successfully" but does not show the resulting diff. This is a minor risk—the edit tool may have applied changes incorrectly, and the assistant only verified the C++ changes in the next message ([msg 2154]).

Input Knowledge Required

To understand this message, the reader needs:

Output Knowledge Created

This message produces:

The Thinking Process

The assistant's reasoning, visible in the five numbered points, reveals a systematic, surgical approach. The assistant did not rewrite the entire function; instead, it identified the minimal set of changes that would achieve the desired interleaving. The thinking proceeds as follows:

  1. Identify the root cause: The static mutex covers too much code, serializing workers.
  2. Design the solution: Pass the mutex from outside, narrow its scope to only the CUDA kernel region.
  3. Determine the precise lock/unlock points: Lock after prep_msm_thread launch (so CPU preprocessing runs unlocked), unlock after per-GPU threads join (so the next worker can start its CUDA work while this worker finishes its CPU join).
  4. Reorder joins: Per-GPU first (to release lock early), then prep_msm (to run outside lock). This thinking is informed by the Phase 8 design document (referenced in [msg 2146]), which the assistant had read before starting implementation. The assistant also read all relevant source files in parallel ([msg 2147], [msg 2148], [msg 2149]) to understand the full call chain before making any changes—a disciplined approach that avoids cascading surprises.

Conclusion

Message [msg 2151] is a textbook example of a high-leverage refactor: a small change to a single function signature and its lock scope that unlocks a 13–17% throughput improvement across the entire proving pipeline. By narrowing the C++ static mutex to cover only the CUDA kernel region, the assistant enabled two GPU workers per device to interleave their CPU preprocessing and GPU kernel execution, eliminating the GPU idle gap that had plagued Phase 7. The five-point plan is precise, minimal, and well-reasoned, demonstrating a deep understanding of both the CUDA execution model and the Groth16 proof generation pipeline. This message is the foundation upon which the rest of Phase 8—the FFI plumbing, Rust bellperson changes, engine worker spawning, and benchmark validation—was built.