The Illusion of Lock Isolation: How CUDA Device-Global Synchronization Defeated a Two-Lock Optimization

In the high-stakes world of Filecoin proof generation, every millisecond counts. The SUPRASEAL_C2 pipeline—responsible for producing Groth16 proofs for Proof-of-Replication (PoRep)—had already undergone nine phases of optimization, each squeezing more throughput from a system of CPU synthesis workers, GPU kernels, and PCIe transfers. Phase 10 promised a breakthrough: a two-lock design that would allow memory management and GPU compute to overlap, hiding the CPU-side overhead that had become the dominant bottleneck. But when the implementation crashed with out-of-memory errors and 102-second proof times, the assistant sat down to debug—and discovered a fundamental truth about CUDA programming: some operations cannot be isolated.

This message ([msg 2619]) captures the moment of diagnosis. It is a window into the assistant's reasoning process as it works through timing logs, reconstructs the sequence of events across three GPU workers, and ultimately identifies the root cause: device-global synchronization operations that silently defeat lock-based isolation on a single CUDA device.

The Optimization Journey So Far

To understand the significance of this message, one must appreciate the context. The SUPRASEAL_C2 pipeline generates Groth16 proofs for Filecoin's Proof-of-Replication, a process that demands ~200 GiB of peak memory and involves a complex call chain from Go orchestration through Rust FFI into C++/CUDA kernels. By Phase 9, the team had achieved a 14.2% throughput improvement through PCIe transfer optimization, but the bottleneck had shifted: CPU-side operations—specifically prep_msm (1.9s) and b_g2_msm (0.48s)—now dominated the per-partition wall time at ~2.4s, leaving the GPU idle for ~600ms per partition waiting for the CPU thread.

The Phase 10 design, documented in c2-optimization-proposal-10.md, proposed a two-lock architecture to address this. The key insight was that VRAM allocation and pre-staging upload took only ~18ms—a tiny fraction of the per-partition time. If this short memory-management phase could overlap with another worker's GPU kernel execution, the CPU overhead could be effectively hidden. The design introduced two mutexes per GPU device: mem_mtx (for VRAM allocation and pre-staging) and compute_mtx (for GPU kernel execution). With gpu_workers_per_device set to 3, the hope was that one worker could be pre-staging memory while another was running kernels, keeping the GPU continuously busy.

The Implementation and Initial Failure

The implementation in groth16_cuda.cu restructured the generate_groth16_proofs_c function, replacing the single gpu_lock with the two-lock protocol. The mem_mtx region handled VRAM allocation, cudaMemPoolTrimTo, and cudaDeviceSynchronize, followed by the pre-staging upload. The compute_mtx region wrapped GPU kernel execution, thread join, and cleanup. After fixing a type-casting error (the FFI parameter needed to change from std::mutex* to void*), the build succeeded.

But the correctness test told a different story. With c=1 j=1 (one proof, one job), the first attempt failed after 41.1 seconds. The assistant's initial fix—adding cudaDeviceSynchronize back into the mem_mtx region—didn't help; the second attempt also failed at 41.4 seconds. Something deeper was wrong.

The Debugging Process: Step by Step

Message [msg 2619] begins with the assistant staring at the timing logs and saying, "Now I see what's happening." What follows is a masterclass in diagnostic reasoning, as the assistant reconstructs the exact sequence of events across the three GPU workers.

The reconstruction reveals a cascade of failures:

  1. Worker 0 acquires mem_mtx, successfully allocates 12 GiB of VRAM (4 GiB for d_a + 8 GiB for d_bc), performs pre-staging upload in ~18ms, and releases mem_mtx. The timing log shows prestage_setup=ok and alloc_ms=1291—a suspiciously long 1.3 seconds for the cudaMalloc call, suggesting the CUDA driver is performing internal defragmentation.
  2. Worker 1 acquires mem_mtx next. It sees only 1.5 GiB of free VRAM—Worker 0's 12 GiB allocation is still live on the device. The pre-staging check fails (skip_vram), and Worker 1 releases mem_mtx without allocating anything.
  3. Worker 2 acquires mem_mtx and sees even less: 1.4 GiB free. It also skips VRAM pre-staging and releases the lock.
  4. Worker 0 acquires compute_mtx and begins running GPU kernels. Meanwhile, Workers 1 and 2 are waiting to enter compute_mtx.
  5. Worker 0 finishes its kernels, frees d_a and d_bc via cudaFree (synchronous), and releases compute_mtx.
  6. Worker 1 enters compute_mtx and attempts the fallback path—allocating VRAM on-the-fly rather than from pre-staged buffers. This is where the OOM occurs. The assistant pauses at this point and re-examines the timing data. It notices sync_ms=616 in the fallback path—616 milliseconds spent in cudaDeviceSynchronize. This is the critical clue.

The Moment of Insight

"Wait—the real issue is," the assistant writes, "cudaDeviceSynchronize inside mem_mtx blocks while Worker 0 has compute_mtx running kernels! Worker 1 holds mem_mtx and is blocked on cudaDeviceSynchronize, while Worker 2 is waiting for mem_mtx. This serializes everything."

This is the key insight. The cudaDeviceSynchronize call inside mem_mtx is a device-global operation—it waits for all pending work on the entire CUDA device to complete. When Worker 1 holds mem_mtx and calls cudaDeviceSynchronize, it blocks until Worker 0's kernels (running under compute_mtx) finish. The two locks, which were supposed to allow overlap, are effectively serialized by the implicit synchronization.

But then the assistant corrects itself: "Wait, actually if Worker 0's kernels are done (sync finished), then d_a and d_bc should be freed. Let me re-read: alloc_ms=1291—that's 1.3 seconds for cudaMalloc. That's the actual cudaMalloc blocking while trying to allocate 12 GiB when there isn't enough VRAM."

The assistant realizes that the problem is twofold. First, the cudaDeviceSynchronize in mem_mtx destroys the intended lock overlap by blocking on another worker's compute. Second, the fallback path inside compute_mtx lacks a cudaDeviceSynchronize and cudaMemPoolTrimTo call, so it tries to allocate VRAM from a pool still cluttered with cached allocations from the previous worker's async operations.

Root Cause: Device-Global Operations Defeat Lock Isolation

The fundamental problem is that certain CUDA operations are device-global—they affect the entire GPU, not just a specific stream or context. cudaDeviceSynchronize waits for all streams on the device. cudaMemPoolTrimTo operates on the device-wide memory pool. These operations cannot be isolated within a single lock region because they implicitly synchronize with work happening under a different lock.

The two-lock design assumed that memory management (mem_mtx) and compute (compute_mtx) could operate independently on the same device. But cudaDeviceSynchronize inside mem_mtx creates a hidden dependency: it blocks until all GPU work completes, including work running under compute_mtx. This turns the intended overlap into strict serialization, as each worker's mem_mtx phase waits for the previous worker's compute_mtx phase to finish.

Worse, the serialization creates a resource deadlock pattern. Worker 0 holds 12 GiB of VRAM while running kernels under compute_mtx. Worker 1's mem_mtx can't allocate because the VRAM is occupied, and it can't proceed to compute_mtx because Worker 0 holds that lock. Even after Worker 0 releases compute_mtx, the freed VRAM may not be immediately available because the async memory pool hasn't been trimmed—and the fallback path inside compute_mtx doesn't perform this trim.

The Fix: Moving Synchronization into the Compute Lock

The assistant's conclusion is pragmatic: "The fallback path runs inside compute_mtx without any pool trim or DeviceSync. It needs one." The fix is to add a cudaDeviceSynchronize + cudaMemPoolTrimTo at the start of the compute_mtx region when the pre-staging path was skipped. This ensures that VRAM is properly reclaimed before the fallback allocation attempts.

But the deeper lesson is that the two-lock design itself is flawed when device-global operations are involved. The cudaDeviceSynchronize inside mem_mtx must be removed entirely, relying instead on the fallback path inside compute_mtx when pre-staging fails. This means the intended overlap between memory management and compute is largely unachievable on a single CUDA device—at least with this approach.

Broader Implications: Hardware Constraints Override Software Abstraction

This debugging session reveals a critical theme in GPU programming: hardware constraints override software abstraction. The CUDA device model is fundamentally a shared resource with global synchronization points. No amount of lock splitting can isolate operations that implicitly synchronize the entire device. The cudaDeviceSynchronize call is not just a convenience for waiting—it is a reflection of the hardware's design, where memory management and compute share the same execution resources.

For the SUPRASEAL_C2 optimization effort, this means that overlapping memory management with compute on a single GPU requires a fundamentally different approach. Perhaps the pre-staging must be done on a separate CUDA stream that can be synchronized independently. Perhaps the memory pool must be managed at a finer granularity. Or perhaps the overlap can only be achieved across multiple GPUs, where each device has its own independent synchronization domain.

Conclusion

Message [msg 2619] is a testament to the value of careful diagnostic reasoning. The assistant did not simply conclude "the two-lock design doesn't work"—it traced the exact sequence of events, identified the specific CUDA operations causing the failure, and understood why the abstraction failed. The cudaDeviceSynchronize inside mem_mtx was not just a performance issue; it was a fundamental violation of the lock isolation assumption.

The article that could be written from this single message would teach any CUDA programmer a crucial lesson: when designing lock-based concurrency for GPU code, every operation must be examined for device-global side effects. A lock can protect against data races, but it cannot protect against the implicit synchronization imposed by the hardware. In the world of CUDA, some operations are simply unisolatable—and the sooner a developer learns to recognize them, the sooner they can design systems that work with the hardware rather than against it.