The Final Edit: Implementing Phase 10's Two-Lock Architecture for GPU Work Overlap

Message 2597: "Now replace the old cleanup + unlock section: [edit] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu Edit applied successfully."

At first glance, message 2597 appears to be a mundane status update — a simple confirmation that an edit was applied. But in the context of the broader Phase 10 optimization effort, this message represents the culmination of a carefully reasoned restructuring of one of the most performance-critical code paths in the Filecoin PoRep proving pipeline. It is the final edit in a sequence that transforms a single-mutex GPU synchronization strategy into a two-lock architecture designed to overlap CPU-side memory management with GPU kernel execution, potentially yielding a 30–38% throughput improvement.

The Bottleneck That Drove the Design

To understand why this message was written, one must trace the optimization journey that preceded it. Throughout Phases 6 through 9 of the SUPRASEAL_C2 optimization campaign, the assistant and user had progressively identified and addressed bottlenecks in the Groth16 proof generation pipeline. Phase 9's PCIe transfer optimization had successfully reduced GPU kernel time to approximately 1.8 seconds per partition, but detailed TIMELINE analysis revealed a critical bottleneck shift: the CPU critical path — specifically prep_msm (1.9s) and b_g2_msm (0.48s) — now dominated the per-partition wall time at approximately 2.4 seconds. The GPU was idle for roughly 600 milliseconds per partition, waiting for the CPU thread to finish its work.

The root cause was CPU memory bandwidth contention. At high concurrency, the ten synthesis workers competed with the CPU MSM operations for the system's 8-channel DDR5 memory bandwidth, inflating CPU times by 2–12×. The bottleneck had moved from PCIe transfers and GPU kernel execution to CPU memory bandwidth contention — a fundamentally different class of problem that required a structural architectural change rather than a simple parameter tuning.

The Two-Lock Design: A Carefully Reasoned Protocol

The user proposed a two-lock design to better overlap CPU and GPU work. The assistant analyzed the deadlock risks — particularly resource deadlock on VRAM — and designed a simplified protocol using two mutexes: a mem_mtx for VRAM allocation and upload, and a compute_mtx for GPU kernel execution. The key insight was that since d_a and d_bc (the two 6 GiB device memory buffers) are freed synchronously via cudaFree inside compute_mtx, the next worker could safely allocate in mem_mtx without needing a cooperative free slot. This eliminated the need for complex memory handoff protocols.

The design was documented in c2-optimization-proposal-10.md and the implementation proceeded in a series of careful edits:

1. Message 2589: The gpu_locks struct was defined with two std::mutex members, and the FFI functions create_gpu_mutex / destroy_gpu_mutex were updated to allocate and destroy the new struct.
2. Message 2590: The fallback mutex path was updated to use the new struct.
3. Message 2592: The single gpu_lock acquisition was replaced with a mem_mtx scope around the pre-staging section (VRAM allocation and async upload).
4. Message 2596: The per-GPU thread creation and join section was wrapped with a compute_mtx scope.
5. Message 2597: The old cleanup and unlock section — the final remnant of the single-mutex architecture — was replaced.

What Message 2597 Actually Changed

Message 2597 is the final piece of the restructuring puzzle. The old cleanup section, which had been under the single gpu_lock, contained code to destroy CUDA events and streams, call cudaHostUnregister, join the prep_msm_thread, and execute the epilogue (point arithmetic). In the original single-mutex design, all of this ran while holding the GPU mutex, preventing any other worker from starting its GPU work.

The new design moved the event/stream destruction and cudaHostUnregister to after the compute_mtx scope, since these operations are tiny and have no VRAM impact. The prep_msm_thread.join() and epilogue were also moved outside any lock, since they are pure CPU operations that don't touch the GPU. This restructuring was the logical conclusion of the design principle: hold locks only for the minimum duration required to protect shared resources, and never hold a lock across operations that don't need it.

Assumptions Embedded in the Design

The two-lock design rested on several critical assumptions about CUDA behavior:

1. Synchronous cudaFree is truly synchronous: The design assumed that when d_a and d_bc are freed via cudaFree inside compute_mtx, the VRAM is immediately available for the next worker's mem_mtx allocation. This is correct for cudaFree — it is a synchronous call that blocks until the memory is freed.
2. Pool trim is sufficient without cudaDeviceSynchronize: The design assumed that cudaMemPoolTrimTo inside mem_mtx would be sufficient to reclaim pool-cached memory from stream-ordered frees (cudaFreeAsync) that occurred inside compute_mtx, without needing an explicit cudaDeviceSynchronize.
3. Memory management and compute can be isolated: The fundamental assumption was that VRAM allocation and upload (under mem_mtx) could proceed concurrently with GPU kernel execution (under compute_mtx) on the same CUDA device, because they use different hardware resources.
4. Three workers per GPU would improve throughput: The design increased gpu_workers_per_device from 2 to 3, assuming that the overlap would hide the CPU overhead and reduce per-partition wall time from approximately 3.7 seconds to 1.8–2.0 seconds.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible in the preceding messages, shows a careful, methodical approach to the restructuring. In message 2591, the assistant explicitly enumerated the old code structure and the desired new structure, mapping each section to its new lock region. This systematic approach — read the old code, plan the new structure, then edit in small, verifiable steps — reflects a disciplined engineering methodology.

The assistant also demonstrated awareness of subtle CUDA semantics. In the design discussion preceding the implementation, the assistant considered whether cudaDeviceSynchronize would be needed inside mem_mtx, analyzed the implications of stream-ordered pool frees, and concluded that pool trim alone should suffice because "by the time compute_mtx was released, all stream operations were synchronized." This reasoning shows a deep understanding of CUDA's asynchronous execution model and memory management.

The Irony: Assumptions That Would Soon Be Challenged

What makes message 2597 particularly interesting is what happened next. The very assumptions that made the two-lock design elegant would prove to be its undoing. When the assistant ran correctness tests after building, the system suffered severe OOM failures and performance regressions, with prove time ballooning to 102 seconds. Debugging the timing logs revealed that only the first worker successfully pre-staged VRAM; subsequent workers found insufficient free memory because the first worker's 12 GiB allocation was still live.

The root cause was that cudaDeviceSynchronize and cudaMemPoolTrimTo inside mem_mtx are device-global operations that implicitly synchronize the entire CUDA device. When one worker held compute_mtx running kernels, and another worker tried to call these device-global operations inside mem_mtx, the operations blocked — effectively serializing the two locks and destroying the intended overlap. All subsequent partitions were forced to use the slow fallback path.

The assistant ultimately diagnosed the fundamental conflict: memory management operations on a single CUDA device cannot be fully isolated from compute operations. The solution was to remove the device-wide synchronization from the mem_mtx region entirely, relying instead on the already-proven fallback path inside compute_mtx when pre-staging fails.

Input Knowledge Required

To fully understand message 2597, one needs knowledge of:

• CUDA memory management: The distinction between synchronous cudaFree and stream-ordered cudaFreeAsync, the behavior of cudaMemPoolTrimTo, and the device-global nature of cudaDeviceSynchronize.
• The SUPRASEAL_C2 pipeline: The role of d_a and d_bc as 6 GiB device memory buffers for proof inputs, the pre-staging path that uploads them, and the per-GPU thread that runs NTT, batch addition, and MSM kernels.
• The optimization history: The progression from Phase 6 through Phase 9, each addressing a different bottleneck (partition dispatch, GPU interlock, PCIe transfers).
• C++ concurrency: The use of std::mutex and std::unique_lock for scoped locking.

Output Knowledge Created

Message 2597 produced the final edit that completed the two-lock restructuring of generate_groth16_proofs_c. The output is a codebase where:

• The mem_mtx region handles VRAM allocation and pre-staging upload (~18ms of lock hold time).
• The compute_mtx region serializes GPU kernel execution (NTT, batch_add, tail_msm).
• Event/stream destruction, cudaHostUnregister, and CPU-only operations (prep_msm_thread.join(), epilogue) run outside any lock.
• Three workers per GPU can potentially overlap their memory and compute phases.

Conclusion

Message 2597 is a deceptively simple edit that represents the culmination of a deep optimization analysis. It embodies the engineering tension between elegant software abstractions (two independent locks for independent operations) and hardware realities (a CUDA device is a single shared resource with global synchronization semantics). The edit itself was correct in its logic — the lock regions were properly scoped, the data dependencies were correctly analyzed, and the design followed sound concurrency principles. But it would take the discovery of device-global synchronization conflicts to reveal that on a single GPU, memory management and compute cannot be fully decoupled. This is a powerful lesson in systems engineering: the most elegant abstraction is only as good as the hardware it abstracts.