The Two-Lock Proposal: A Pivotal Design Decision in GPU Proving Pipeline Optimization

The Message

In the middle of an intense optimization session for the SUPRASEAL_C2 Groth16 proof generation pipeline, the user sent a concise but architecturally significant message:

To check/try, now we have 2 gpu workers, maybe we can try to add 1/2 more? Interlocked on the same compute lock + some new memory-management lock (needs to be ordered correctly to not deadleck)

This single sentence, at message index 2569 in the conversation, crystallizes a design insight that would drive an entire optimization phase — and ultimately reveal a fundamental constraint about CUDA device-level synchronization that no amount of software abstraction could circumvent.

Context: Why This Message Was Written

To understand why this message was necessary, one must trace the optimization journey that preceded it. The team had been systematically working through a series of optimization phases for the cuzk SNARK proving engine, a persistent GPU-resident system for generating Filecoin Proof-of-Replication (PoRep) proofs.

Phase 8 had introduced a dual-worker GPU interlock: instead of a single GPU worker serializing all work, two workers per GPU could interleave their execution. While one worker held the GPU mutex and ran CUDA kernels, the other could be doing CPU-side preprocessing (prep_msm, b_g2_msm). This achieved a 2.4x improvement over baseline, reaching 37.4 seconds per proof with 100% GPU utilization.

Phase 9 then implemented PCIe transfer optimization — pre-staging NTT input data to GPU VRAM before the compute lock was acquired. This cut GPU kernel time by 51% (from 3.75s to 1.82s per partition). However, the benchmark results told a frustrating story: despite dramatically faster GPU kernels, steady-state throughput only improved 14% (from 37.4s to 32.1s in isolation, and actually regressed to 41.0s at high concurrency).

The bottleneck had shifted. Detailed timing instrumentation revealed the culprit: the CPU critical path — prep_msm (1.91s) plus b_g2_msm (0.48s) — now totaled 2.39s per partition, outstripping the 1.82s GPU kernel time. The GPU sat idle for 600ms every partition, waiting for the CPU thread to finish. At high concurrency (c=15+), ten synthesis workers competed with the CPU MSM operations for 8-channel DDR5 memory bandwidth, inflating CPU times by 2–12x.

This was the wall the team had hit. The GPU was no longer the bottleneck — the CPU's memory bandwidth was. And the dual-worker design from Phase 8, which had previously hidden CPU overhead through interleaving, was being undermined by Phase 9's pre-staging serialization.

The User's Proposal: A Two-Lock Architecture

The user's message proposes a specific architectural evolution. The core idea is elegant: instead of a single gpu_mtx that serializes both memory management and GPU compute, split the lock into two independent mutexes:

1. A compute lock (compute_mtx) — held only during GPU kernel execution (NTT, batch addition, tail MSM)
2. A memory-management lock (mem_mtx) — held during VRAM allocation, pre-staging uploads, and pool management With this split, three or more workers per GPU could operate in a pipelined fashion: - Worker A holds compute_mtx, running GPU kernels on 12 GiB of pre-staged data - Worker B holds mem_mtx, allocating VRAM and uploading the next partition's data - Worker C does CPU-only work (prep_msm, b_g2_msm) with no lock at all The critical phrase "needs to be ordered correctly to not deadleck" shows the user's awareness of the fundamental challenge: if locks are acquired in inconsistent orders across workers, classic deadlock ensues. If Worker A holds compute_mtx and tries to acquire mem_mtx (to free buffers), while Worker B holds mem_mtx and tries to acquire compute_mtx (to run kernels), neither can proceed.

Assumptions Embedded in the Proposal

The proposal makes several implicit assumptions that are worth examining:

That memory management and compute can be cleanly separated on a single CUDA device. This assumption turned out to be incorrect, as the subsequent implementation would discover. CUDA operations like cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global — they affect all streams and all outstanding operations. A worker holding mem_mtx and calling cudaDeviceSynchronize to ensure VRAM is reclaimable would block until the worker holding compute_mtx finished its kernels, effectively serializing the two locks.

That VRAM capacity is sufficient for three workers' buffers. On a 16 GiB GPU, peak usage during H-MSM is ~13.8 GiB, leaving only ~2 GiB headroom. A third worker attempting to pre-stage 12 GiB while two others' buffers are live would exhaust VRAM. The design would need a cooperative handoff mechanism — the next worker frees the previous worker's buffers before allocating its own.

That the CPU-side bottleneck (prep_msm + b_g2_msm) can be hidden by adding more workers. The assumption is that with three workers, the 2.39s CPU critical path can be overlapped with the 1.82s GPU kernel time of another worker, reducing per-partition wall time to ~1.8s. However, this assumes the CPU work doesn't itself become a bottleneck at higher worker counts — which it does, because all workers share the same DDR5 memory bandwidth.

The Thinking Process Visible in the Message

Despite its brevity, the message reveals sophisticated reasoning. The user doesn't just say "add more workers" — they specify the mechanism ("interlocked on the same compute lock") and immediately identify the risk ("needs to be ordered correctly to not deadleck"). This shows an understanding of:

• The current single-mutex bottleneck (one worker blocks both memory and compute)
• The opportunity to overlap memory management with compute
• The lock-ordering problem inherent in multi-lock designs
• The specific deadlock scenario where two workers each hold one lock and need the other The phrase "some new memory-management lock" is deliberately vague — the user is proposing the concept, leaving the detailed design (what exactly the memory lock protects, how long it's held, what resources it covers) to the implementation phase. This is characteristic of a high-level architectural direction rather than a fully specified design.

Input Knowledge Required

To fully understand this message, one needs:

1. Phase 8's dual-worker interlock: The existing architecture where two workers per GPU share a single std::mutex, with one worker holding it during GPU kernels while the other does CPU preprocessing.
2. Phase 9's pre-staging optimization: The PCIe transfer optimization that moved VRAM allocation and data upload into the locked region, reducing GPU kernel time but increasing lock hold time.
3. The CPU bottleneck discovery: The detailed timing analysis showing prep_msm (1.91s) + b_g2_msm (0.48s) = 2.39s CPU critical path, exceeding the 1.82s GPU kernel time.
4. CUDA memory management: Understanding of cudaMalloc, cudaFree, cudaMallocAsync, cudaFreeAsync, stream-ordered memory pools, and cudaMemPoolTrimTo.
5. Deadlock fundamentals: The classic condition where two threads each hold one resource and wait for the other, creating a circular wait.

Output Knowledge Created

This message triggered a cascade of output:

1. The Phase 10 design document (c2-optimization-proposal-10.md): A detailed specification of the two-lock architecture, including lock ordering, VRAM lifecycle management, and expected performance improvements (30-38% throughput gain in isolation).
2. The gpu_locks struct implementation: A C++ struct containing two std::mutex instances (compute_mtx and mem_mtx), replacing the single std::mutex* pointer in the FFI boundary.
3. Restructured generate_groth16_proofs_c: The main CUDA function reorganized into three phases — memory management under mem_mtx, GPU kernels under compute_mtx, and CPU epilogue with no lock.
4. The critical debugging session: When the implementation failed with OOM errors and performance regression (prove time ballooning to 102s), the assistant traced the root cause to cudaDeviceSynchronize and cudaMemPoolTrimTo being device-global operations that implicitly serialized the two locks.
5. The fundamental lesson: Memory management operations on a single CUDA device cannot be fully isolated from compute operations. Device-global synchronization primitives defeat the separation that software-level lock splitting attempts to achieve.

The Deeper Significance

This message represents a turning point in the optimization journey. It's the moment when the team recognized that the single-mutex architecture had reached its limits and a more sophisticated concurrency model was needed. The user's proposal was technically sound in principle — splitting locks to increase parallelism is a classic technique. But it collided with the physical reality of GPU hardware: a single CUDA device has global state (memory pools, synchronization domains) that cannot be partitioned.

The subsequent debugging revealed that the two-lock design, as implemented, actually made things worse. The cudaDeviceSynchronize call inside mem_mtx blocked until the worker holding compute_mtx finished its kernels, effectively serializing the two locks and destroying the intended overlap. Meanwhile, the third worker's CPU work competed for DDR5 bandwidth, inflating prep_msm times further.

This is a beautiful example of how hardware constraints override software abstractions. No amount of careful lock ordering or deadlock prevention can work around the fact that a GPU is a single device with global synchronization domains. The lesson is that on a single GPU, the critical path is fundamentally serial — you can overlap CPU work with GPU work, but you cannot parallelize GPU memory management with GPU compute.

Conclusion

The user's message at index 2569 is deceptively simple — a six-line proposal that appears to be a straightforward extension of existing ideas. But it encapsulates the entire arc of Phase 10: the insight that finer-grained locking could unlock more parallelism, the careful design to avoid deadlocks, the implementation that seemed correct on paper, and the humbling discovery that the GPU hardware itself enforces serialization where the software tried to create parallelism. The message stands as a testament to the gap between theoretical concurrency models and the practical realities of heterogeneous computing systems.