The Moment of Truth: Building Phase 10's Flawed Two-Lock Design

In the high-stakes world of GPU-accelerated SNARK proving, where every second of proof generation time translates directly to operational cost, optimization is a relentless pursuit. The opencode conversation captured in message [msg 2634] represents a critical inflection point in that pursuit — the moment when a promising but fundamentally flawed design meets the cold reality of empirical testing. This message, issued by the AI assistant during a multi-session investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep, is deceptively simple on its surface: a single bash command to force-rebuild CUDA code. But beneath that simplicity lies a wealth of engineering judgment, risk assessment, and the disciplined application of the scientific method to systems optimization.

The Context: Phase 10's Troubled Design

To understand why this message matters, one must first understand what Phase 10 was attempting to accomplish. The cuzk SNARK proving engine had already undergone nine phases of optimization, progressively reducing proof generation time from well over a minute to approximately 32 seconds per proof in single-worker mode ([msg 2628]). The bottleneck had shifted multiple times: first GPU kernel efficiency, then PCIe transfer bandwidth, then CPU-side synchronization overhead. By Phase 9, the dominant constraint had become DDR5 memory bandwidth contention, with CPU-side operations like prep_msm and b_g2_msm competing with synthesis workers for access to the system's 8-channel memory.

Phase 10 was designed around an elegant idea: split the single GPU mutex into two locks — a compute_mtx for GPU kernel execution and a mem_mtx for VRAM allocation. The theory was that with three GPU workers (gw=3), one worker could be running GPU kernels under compute_mtx while another worker simultaneously pre-staged its buffers under mem_mtx, overlapping CPU work like b_g2_msm and prep_msm with GPU computation. This promised to hide the ~2.4 seconds of CPU critical path work behind the ~1.8 seconds of GPU kernel time, potentially achieving near-perfect overlap.

However, the implementation had already revealed deep problems by the time of this message. The assistant had discovered that CUDA memory management APIs — cudaDeviceSynchronize, cudaMemPoolTrimTo, cudaMemGetInfo — are device-global operations. Any call to these functions in mem_mtx would block until ALL GPU streams completed, including those running under another worker's compute_mtx. This defeated the entire purpose of the lock split. Furthermore, the RTX 5070 Ti's 16 GB VRAM was simply too small to accommodate pre-staged buffers from multiple workers simultaneously — during H-MSM, peak VRAM usage reached approximately 13.8 GiB, leaving no room for a second worker's buffers.

The assistant had made a last-ditch edit to salvage the design: strip all CUDA management APIs from mem_mtx and simply attempt cudaMalloc directly, failing fast on OOM. This edit, documented in [msg 2628], was the code state that [msg 2634] was about to build and test.

The Decision to Test

The immediate trigger for this message was the user's prompt at [msg 2629]: "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed." The assistant had just completed a comprehensive analysis of Phase 10's problems in [msg 2628], documenting thirteen findings about why the two-lock design was failing. The natural inclination might have been to abandon the approach immediately — the theoretical analysis already suggested it was doomed. But the assistant chose a different path: build, test, and benchmark.

This decision reveals a key aspect of the assistant's engineering philosophy. Rather than concluding from theory alone, it insisted on empirical validation. The preamble to the bash command states this explicitly: "The code has Phase 10 two-lock changes that haven't been compiled yet. Let me build, test, and benchmark." This is the scientific method applied to systems optimization — form a hypothesis, implement it, measure it, and let the data guide the next decision.

The specific command — rm -rf target/release/build/supraseal-c2-* — is itself noteworthy. It reflects a deep understanding of the Rust/CUDA build pipeline. When .cu files change, the Rust build system's incremental compilation cache can become stale, leading to subtle bugs where old object files are linked instead of freshly compiled ones. The force-rebuild ensures that every CUDA kernel is compiled from the latest source, eliminating an entire class of "did I actually test my changes?" uncertainty.

Assumptions and Risks

The assistant was operating under several assumptions when issuing this command. First, that the CUDA code would compile successfully despite the significant restructuring of the mutex architecture. Second, that even if pre-staging mostly failed (as the VRAM analysis suggested), the two-lock design might still provide some benefit by hiding the ~0.5 seconds of b_g2_msm execution behind GPU kernel time. Third, that the empirical data from benchmarking would be more informative than theoretical analysis alone for deciding whether to proceed with Phase 10.

The risks were real. A failed build would waste time. A successful build that produced incorrect proofs could corrupt benchmark data. And perhaps most insidiously, the sunk-cost fallacy — having invested significant effort in the Phase 10 implementation, the assistant might be tempted to rationalize away negative results. The message shows no evidence of this bias; the tone is clinical and forward-looking.

What Followed

The subsequent messages tell a sobering story. The build succeeded ([msg 2635]). The daemon started with three GPU workers ([msg 2639]). The correctness test passed — the proof completed rather than failing ([msg 2644]). But the performance was catastrophic: 73.8 seconds per proof, more than double Phase 9's 32.1-second baseline. The daemon logs revealed the reason: most partitions fell back to the non-prestaged path, with gpu_total_ms of 3.3–3.9 seconds instead of the hoped-for 1.8 seconds. The two-lock design had not only failed to improve throughput — it had regressed performance by forcing every partition through the slow fallback path.

This empirical data ultimately led the assistant to abandon Phase 10 entirely, revert to Phase 9's proven single-lock approach, and design Phase 11 with three targeted interventions to address the real bottleneck: DDR5 memory bandwidth contention ([chunk 28.0]). The Phase 10 failure was documented in a post-mortem, and the lessons learned — particularly about CUDA device-global API constraints and VRAM capacity limits — informed every subsequent optimization.

Broader Significance

Message [msg 2634] exemplifies a pattern that recurs throughout high-performance systems engineering: the tension between theoretical elegance and practical constraints. The two-lock design was intellectually appealing — splitting a contended resource into two independent locks is a classic concurrency optimization. But it failed because CUDA's API semantics violated the assumptions underlying that pattern. The device-global nature of CUDA memory management meant that no amount of lock splitting could isolate memory operations from compute operations on the same GPU.

The message also demonstrates the importance of building a test harness before committing to a design direction. The assistant's disciplined approach — implement, build, test, measure, evaluate — prevented weeks of wasted effort on a fundamentally flawed architecture. The entire Phase 10 experiment, from design to abandonment, spanned only a handful of messages. This rapid feedback loop is the hallmark of effective optimization work.

For anyone studying this conversation, [msg 2634] serves as a reminder that in systems optimization, data trumps theory. The assistant could have concluded from the VRAM analysis alone that Phase 10 was doomed. Instead, it built the code, ran the benchmark, and let the 73.8-second result speak for itself. That discipline — the willingness to be proven wrong by one's own experiments — is what separates effective optimization from wishful thinking.