When Hardware Overrides Abstraction: The Phase 10 Two-Lock Design Failure in CUDA Memory Management
In the high-stakes world of Filecoin proof generation, every millisecond counts. The SUPRASEAL_C2 pipeline, responsible for Groth16 proofs in the PoRep consensus mechanism, had been through nine optimization phases—each one peeling back a layer of bottleneck to reveal the next. By Phase 9, the team had achieved remarkable throughput by overlapping PCIe transfers with GPU computation, but a new bottleneck had emerged: CPU memory bandwidth contention. The solution seemed elegant: split the GPU work into two lock-protected regions, allowing one worker to pre-stage VRAM allocations while another ran kernels. This was Phase 10. It failed spectacularly.
Message [msg 2626] captures the moment of reckoning. After three failed builds, two OOM crashes, and a correctness test that produced a proof in 102 seconds—more than double the Phase 8 baseline—the assistant finally diagnosed the root cause. The diagnosis was not a bug in the usual sense. There was no off-by-one error, no race condition, no null pointer dereference. The problem was deeper: the CUDA device model itself does not permit the kind of lock-level isolation that the design assumed. Memory management operations on a single GPU are inherently device-global, and no amount of mutex cleverness can change that.
The Phase 10 Ambition: Overlapping Memory Management with Computation
To understand why message [msg 2626] represents such a critical turning point, we must first understand what Phase 10 was trying to achieve. The preceding optimization phases had transformed the proof generation pipeline from a naive serial process into a sophisticated multi-worker architecture. Phase 7 introduced per-partition dispatch, Phase 8 added dual-GPU-worker interlock to eliminate CPU-side mutex contention, and Phase 9 optimized PCIe transfers. Each phase improved throughput, but each also revealed the next bottleneck.
By the end of Phase 9, the bottleneck had shifted to the CPU. The critical path through prep_msm (1.9 seconds) and b_g2_msm (0.48 seconds) dominated the per-partition wall time at roughly 2.4 seconds, leaving the GPU idle for approximately 600 milliseconds per partition waiting for the CPU thread. At high concurrency, the ten synthesis workers competed with the CPU MSM operations for 8-channel DDR5 memory bandwidth, inflating CPU times by 2–12×.
The Phase 10 design, documented in c2-optimization-proposal-10.md, proposed a two-lock architecture. Instead of a single mutex protecting the entire GPU work region, there would be two:
mem_mtx: A short-hold lock protecting only VRAM allocation and pre-staging upload, expected to take approximately 18 milliseconds.compute_mtx: A longer-hold lock serializing GPU kernel execution, which took roughly 1.8 seconds per partition. The insight was that these two phases could overlap. While Worker A heldcompute_mtxand ran GPU kernels, Worker B could acquiremem_mtx, allocate VRAM for the next partition, and upload data via PCIe. When Worker A releasedcompute_mtx, Worker B would already have its memory ready and could immediately begin kernel execution. The expected per-partition wall time would drop from ~3.7 seconds to ~1.8–2.0 seconds, a potential 30–38% throughput improvement. The design was clean, the reasoning sound, and the implementation straightforward. Thegpu_locksstruct was defined, the FFI interface was updated to pass the opaque lock pointer, andgpu_workers_per_devicewas set to 3. The code compiled without errors.
The Failure: Correctness at 2× the Baseline
The first correctness test failed with an OOM error. The second, after adding cudaDeviceSynchronize to the mem_mtx region, produced a proof—but at 72.5 seconds wall time with a prove time of 102 seconds. This was dramatically worse than the Phase 8 baseline of approximately 41 seconds per proof.
The timing logs told a stark story. As the assistant noted in message [msg 2626]:
Only the first partition pre-staged successfully. All subsequent ones fall back toskip_vram— and fall back to the old non-prestaged path withgpu_total_msof 3.3-3.9s (Phase 8 performance).
The first worker would acquire mem_mtx, successfully allocate 12 GiB of VRAM, and pre-stage the data. But when the second worker acquired mem_mtx, it found only 1.5 GiB free—the first worker's 12 GiB allocation was still live on the GPU. The pre-staging logic correctly detected insufficient memory and fell back to skip_vram, which meant the worker would skip pre-staging and instead allocate memory inside compute_mtx using the older, slower path.
The result was that only one partition ever benefited from pre-staging. All subsequent partitions fell back to Phase 8 performance, and the serialization overhead from the cudaDeviceSynchronize call made everything worse.
The Diagnosis: Device-Global Operations Cannot Be Isolated
The assistant's diagnosis in message [msg 2626] is precise and revealing:
The issue is clear:cudaDeviceSynchronizeinmem_mtxwaits for the current compute_mtx holder's kernels, meaningmem_mtxis held for the entire duration of the other worker's kernel execution. This serializes everything.
The root cause was that cudaDeviceSynchronize and cudaMemPoolTrimTo—the very functions needed to make VRAM available for pre-staging—are device-global operations. They interact with ALL CUDA streams on the device, including the streams running kernels under compute_mtx. When Worker B called cudaDeviceSynchronize inside mem_mtx, it blocked until Worker A's kernels (running under compute_mtx) completed. This meant mem_mtx was held for the entire kernel execution duration, not the intended 18 milliseconds.
The two locks had become effectively serialized. Worker B could not release mem_mtx until Worker A released compute_mtx, and Worker C could not enter mem_mtx until Worker B had finished. The entire pipeline collapsed into a sequential chain, worse than the original single-mutex design because of the added overhead of the cudaDeviceSynchronize call.
The Fix: Abandoning the Two-Lock Illusion
The assistant's fix was as elegant as it was pragmatic:
Don't callcudaDeviceSynchronizeinmem_mtxat all. Instead, rely on the fallback path insidecompute_mtxwhen pre-staging fails. The fallback path already works correctly (Phase 8 proved this).
The insight was that the pre-staging optimization was not worth the complexity it introduced. The fallback path—allocate VRAM inside compute_mtx, run kernels, free VRAM—had been proven correct and performant in Phase 8. The two-lock design was trying to squeeze out additional overlap, but the CUDA device model made that overlap impossible without device-global synchronization.
The revised approach was brutally simple: in mem_mtx, just try cudaMalloc directly. If it succeeds, great—pre-stage the data and release the lock quickly. If it fails with OOM, skip pre-staging and let the fallback path handle allocation inside compute_mtx. No cudaMemGetInfo, no cudaMemPoolTrimTo, no cudaDeviceSynchronize. Those operations belong inside compute_mtx where they naturally synchronize with the kernel execution they support.
The Broader Lesson: Hardware Constraints Trump Software Abstractions
Message [msg 2626] is more than a debugging note—it is a case study in the limits of software abstraction when confronted with hardware reality. The two-lock design was conceptually sound: separate two independent operations (memory management and computation) into independent lock regions, and let them overlap. But CUDA GPUs do not treat memory management and computation as independent operations. Memory pools are device-global resources. Synchronization primitives like cudaDeviceSynchronize operate on the entire device, not on individual streams or contexts.
This is a recurring theme in high-performance GPU programming. The hardware exposes a flat, globally-synchronized memory model, and any attempt to introduce fine-grained concurrency within a single device must contend with this reality. Multi-GPU systems can achieve true parallelism because each GPU has its own memory space and synchronization domain, but within a single GPU, the device-global nature of memory management is inescapable.
The assistant's thinking process, visible in the progression from message [msg 2613] through [msg 2626], shows a methodical debugging approach. Each hypothesis—"the pool trim isn't working," "we need sync before trim," "the sync is blocking on other streams"—was tested against the timing logs. The final diagnosis required understanding not just the code's logic but the CUDA runtime's behavior at the hardware level.
Conclusion
Message [msg 2626] marks the moment when the Phase 10 optimization hit the wall of hardware reality. The two-lock design was not wrong in principle, but it was wrong for the CUDA device model. The fix—removing device-global synchronization from the memory management lock and relying on the proven fallback path—was a retreat from ambition to pragmatism. But it was a productive retreat: it saved the team from pursuing a dead-end optimization and clarified the fundamental constraints under which any single-GPU optimization must operate.
The lesson for GPU programmers is clear: when designing concurrent access patterns for a single CUDA device, assume that memory management operations are device-global unless proven otherwise. The hardware does not respect your lock boundaries. Only by understanding the device model at this level can you avoid the kind of serialization collapse that nearly derailed Phase 10.