The Moment of Reckoning: Assessing a Failed GPU Interlock Design Through Git Diff

The Message

git diff --stat extern/supraseal-c2/cuda/groth16_cuda.cu
 extern/supraseal-c2/cuda/groth16_cuda.cu | 174 +++++++++++++++----------------
 1 file changed, 84 insertions(+), 90 deletions(-)

git diff extern/supraseal-c2/cuda/groth16_cuda.cu | head -200
diff --git a/extern/supraseal-c2/cuda/groth16_cuda.cu b/extern/supraseal-c2/cuda/groth16_cuda.cu
index 14d0045e..062d706d 100644
--- a/extern/supraseal-c2/cuda/groth16_cuda.cu
+++ b/extern/supraseal-c2/cuda/groth16_cuda.cu
@@ -123,31 +123,42 @@ struct groth16_proof {
 #define __builtin_popcountll(x) __popcnt64(x)
 #endif
 
-// Phase 8: Allocate/free a std::mutex on the C++ heap, returned as an
-// opaque pointer for Rust to hold. This ensures correct C++ ABI alignment
-// and constructor/destructor...

At first glance, message [msg 2631] appears to be nothing more than a routine git diff command—a developer checking what changes have accumulated in a working tree. But in the context of the broader optimization campaign documented across segments 23–28 of this conversation, this message represents something far more significant: a moment of diagnostic assessment after a failed architectural experiment. The assistant is not merely inspecting code; it is taking stock of the wreckage of Phase 10, a two-lock GPU interlock design that promised throughput gains but instead delivered regression, OOM errors, and a fundamental confrontation with the realities of CUDA's device-global synchronization model.

The Context of Failure

To understand why this message was written, one must understand the trajectory that led to it. The cuzk SNARK proving engine had been through eight prior optimization phases, each targeting a specific bottleneck in the Groth16 proof generation pipeline. Phase 9 had achieved a 14.2% throughput improvement through PCIe transfer optimization, but analysis revealed that DDR5 memory bandwidth contention—not GPU compute—was the primary bottleneck. With 10 synthesis workers and prep_msm all doing heavy memory reads simultaneously, the 8-channel DDR5 memory subsystem was saturating, causing synthesis and prep_msm times to inflate by 12–27% under load.

Phase 10 was designed as a surgical intervention: split the single GPU mutex into two locks—compute_mtx for GPU kernel execution and mem_mtx for VRAM allocation—so that three GPU workers could overlap their CPU-side work (b_g2_msm, prep_msm, epilogue) with another worker's GPU kernel execution. The theory was sound: if one worker is running GPU kernels under compute_mtx, a second worker could acquire mem_mtx, pre-stage its VRAM buffers, and be ready to start kernels the moment the first worker finished. This would hide the ~0.5s of b_g2_msm and the Rust epilogue overhead, potentially improving throughput by hiding CPU-side latency behind GPU compute.

The implementation, however, collided with an unforgiving reality: CUDA memory management APIs are device-global. cudaDeviceSynchronize, cudaMemPoolTrimTo, and even cudaMemGetInfo interact with all streams on the device, including those running under another worker's compute_mtx. This meant that when Worker B entered mem_mtx to pre-stage its buffers, its cudaDeviceSynchronize call would block until Worker A's kernels (running under compute_mtx) completed. The two locks, rather than enabling overlap, serialized everything. Furthermore, on a 16 GB GPU, peak VRAM during H-MSM reaches ~13.8 GiB, leaving no room for a second worker's pre-staged buffers. Pre-staging under mem_mtx almost always failed.

The result was a regression: the first successful Phase 10 run produced a 72.5s proof time, nearly double Phase 9's 41.3s baseline. Most partitions fell back to the non-prestaged path with 3.3–3.9s gpu_total_ms instead of the 1.8s achieved in Phase 9.

What This Message Reveals

Message [msg 2631] is the assistant's response to this failure. It is not a triumphant commit or a breakthrough benchmark—it is a diagnostic pause. The assistant issues two git diff commands to examine the working tree and understand exactly what changes have accumulated across multiple editing rounds.

The first command, git diff --stat, provides a high-level summary: 174 lines changed, 84 insertions, 90 deletions. The net change is a slight reduction in code size, which is telling—the assistant has been simplifying, not adding complexity. The deletions outnumber insertions because the assistant has been stripping away the failed cudaDeviceSynchronize calls from mem_mtx, removing cudaMemGetInfo queries, and paring down the pre-staging logic to a bare cudaMalloc attempt that fails fast on OOM.

The second command pipes the first 200 lines of the full diff through head. The output shows the transition from a single std::mutex (Phase 8's proven design) to the gpu_locks struct containing compute_mtx and mem_mtx. The diff reveals the structural changes: the struct definition, the constructor, and the function signature changes that ripple through the CUDA code. The assistant is reading this diff to understand the full scope of what has been touched—every function signature that changed, every lock acquisition that was restructured, every fallback path that was added.

Assumptions and Their Consequences

The Phase 10 design rested on several assumptions that proved incorrect. The first and most consequential was that CUDA memory management could be isolated from CUDA compute on the same device. The assumption was that mem_mtx could safely call cudaDeviceSynchronize and cudaMemPoolTrimTo without interfering with kernels running under compute_mtx. This assumption was rooted in a mental model that treated CUDA operations as stream-local, when in fact key memory management APIs are device-global. The cudaDeviceSynchronize call in mem_mtx blocked on all streams, including those belonging to the worker holding compute_mtx, defeating the entire purpose of the lock split.

The second assumption was that VRAM capacity would be sufficient for two workers' buffers. The 16 GB VRAM of the RTX 5070 Ti is consumed by a single worker's H-MSM phase: 4 GiB for d_a, 8 GiB for d_bc, and approximately 1.8 GiB for MSM temporary storage, totaling ~13.8 GiB. With only ~2.2 GiB of headroom, a second worker's pre-staging allocation of 12 GiB (d_a + d_bc) would always fail. The assistant discovered this empirically: the daemon logs showed prestage_setup=skip_vram for every partition after the first, confirming that pre-staging was a dead end for multi-worker configurations on this hardware.

The third assumption was that the two-lock complexity would be justified by measurable throughput gains. Even if pre-staging failed, the assistant hypothesized that hiding b_g2_msm's ~0.5s behind another worker's kernel execution might still yield improvement. But the serialization introduced by device-global CUDA operations negated even this modest benefit.

Input and Output Knowledge

To understand this message, the reader needs substantial context about the cuzk optimization pipeline: the Groth16 proof generation flow, the role of pre-staging in PCIe transfer optimization, the CUDA device model and its synchronization semantics, and the specific hardware constraints (16 GB VRAM, RTX 5070 Ti, AMD Threadripper 7995WX with 96 cores). The reader must also understand the Phase 9 baseline (41.3s/proof at c=15 j=15) and the Phase 10 design spec documented in c2-optimization-proposal-10.md.

The output knowledge created by this message is the diff itself—a concrete record of what changes were made during the failed Phase 10 experiment. This diff serves multiple purposes: it documents the attempted approach for future reference, it enables the assistant to cleanly revert to Phase 9 if needed, and it provides the raw material for a post-mortem analysis. The diff is also the foundation for the Phase 10 post-mortem that will later be written into cuzk-project.md and c2-optimization-proposal-11.md.

The Thinking Process

The assistant's thinking is visible in the choice of commands. The --stat flag first asks "how much has changed?"—a quantitative assessment of the damage. The answer (174 lines) is modest enough that a clean revert is feasible. Then head -200 asks "what exactly changed?"—a qualitative assessment of the structural modifications. The assistant is reading the diff to understand whether the changes can be salvaged, whether the two-lock approach has any remaining value, or whether a full revert to Phase 9's single-lock design is the only viable path.

The diff output shown in the message is truncated to the first 200 lines, but it reveals the key structural change: the replacement of a single std::mutex with a gpu_locks struct. The old comment read "Phase 8: Allocate/free a std::mutex on the C++ heap"; the new code introduces the two-lock struct. The assistant is examining this diff to decide the next step—and the decision, as the subsequent messages reveal, is to abandon Phase 10, revert to Phase 9's single-lock design, and redirect effort toward Phase 11's three-intervention approach targeting DDR5 memory bandwidth contention directly.

This message, then, is the pivot point. It is the moment of honest assessment before a strategic retreat. The assistant does not double down on the failed design or attempt another workaround. Instead, it reads the diff, absorbs the lesson, and prepares to move forward with a fundamentally different approach—one that addresses the actual bottleneck (memory bandwidth contention) rather than attempting an architectural restructuring that CUDA's device model cannot support.