The Autopsy of a Failed Optimization: How CUDA's Device-Global Semantics Defeated a Two-Lock GPU Interlock

Introduction

In the high-stakes world of Filecoin proof-of-replication (PoRep) proving, every millisecond counts. The cuzk SNARK proving engine, a pipelined Groth16 proof generator designed for the Curio storage mining platform, had been on a relentless optimization march. Phase 9 had achieved a respectable 32.1 seconds per proof, but the architecture still suffered from a critical inefficiency: the GPU was the bottleneck, but not because of its compute capacity. The GPU was waiting on the CPU.

Phase 10 was supposed to fix this. The idea was elegant: split the single GPU mutex into two locks — one for memory management and one for compute — allowing multiple workers to overlap their CPU-side preparation work with another worker's GPU kernel execution. It was the kind of optimization that looks perfect on paper: a textbook lock-splitting refactor that should yield immediate throughput gains.

But CUDA doesn't read textbooks.

Message 2628 in this conversation is a remarkable document. It is not a triumphant announcement of success, nor is it a despairing admission of defeat. It is something rarer in the world of AI-assisted coding sessions: a structured autopsy of a failed optimization attempt, produced mid-stream by the assistant as a knowledge checkpoint. The message catalogs thirteen discoveries — many of them hard-won through failed builds, OOM crashes, and performance regressions — and lays bare a fundamental design challenge that no amount of clever C++ refactoring could solve: CUDA memory management APIs are device-global, and you cannot isolate memory operations from compute operations on the same GPU.

This article examines message 2628 in depth: why it was written, what decisions it captures, the assumptions that proved incorrect, and the knowledge it both consumes and produces. It is a case study in the messy reality of GPU pipeline optimization, where the hardware's fundamental semantics can defeat even the most elegant software design.

The Message in Context

Message 2628 arrives at a critical juncture in the conversation. The assistant had just spent several rounds implementing, debugging, and benchmarking the Phase 10 two-lock design. The preceding messages (2600–2626) show a rapid-fire sequence of edits, builds, failures, and analysis:

What the Message Actually Says

Message 2628 is structured as a formal status document with five major sections: Goal, Instructions, Discoveries, Accomplished, and Relevant Files. It is wrapped in <conversation_data> tags, indicating it is source material from the coding session rather than a direct instruction to any subsequent process.

The Goal section states the ambition: "split the single GPU mutex into two locks (compute_mtx for GPU kernels, mem_mtx for VRAM allocation) so that 3 workers can overlap CPU work (b_g2_msm, prep_msm, epilogue) with GPU kernel execution, improving throughput beyond Phase 9's 32.1s/proof isolation result."

The Instructions section provides environmental context: the Rust/CUDA codebase, test data paths, build commands, hardware specs (AMD Ryzen Threadripper PRO 7995WX with 96 Zen4 cores, 754 GiB RAM; RTX 5070 Ti with 16 GB VRAM, PCIe gen5), and process notes like committing to git often and logging to specific paths.

The Discoveries section is the heart of the document. It lists 13 findings organized into three groups:

Why This Message Was Written

The most striking thing about message 2628 is that it exists at all. In typical AI-assisted coding sessions, the assistant responds to user prompts with tool calls, code edits, and analysis — it does not typically produce standalone structured documents summarizing the state of play. The presence of this message signals a specific need: the assistant needed to consolidate its understanding before proceeding.

Several factors drove this need:

First, the complexity of the failure mode. The Phase 10 two-lock design failed not because of a simple bug but because of a fundamental property of the CUDA API. The insight that cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global — that they interact with all streams on a device, not just the caller's stream — is non-trivial. It requires synthesizing observations from multiple failed runs, log analyses, and timing measurements into a coherent mental model. Writing it down forces that synthesis.

Second, the need to reset the approach. The assistant had just made an edit (removing DeviceSync from mem_mtx) that hadn't been built or tested yet. Before continuing down that path, it needed to evaluate whether the entire two-lock approach was viable. The document's "Fundamental Design Challenge" section explicitly questions whether pre-staging can ever work with gw≥2 on a 16 GiB GPU, and suggests that the real benefit of the split might be only ~0.5s per partition (hiding b_g2_msm), not the hoped-for 1.8s parity.

Third, the handoff quality of the document. The message reads like a handoff note — the kind of document an engineer writes when they're about to step away from a complex debugging session and want their future self (or a colleague) to be able to pick up where they left off. The "Current Code State (NEEDS FIX)" subsection explicitly flags that the last edit hasn't been built, and the "Not Yet Done" list enumerates four remaining tasks. This is particularly valuable in an AI context where conversation context is ephemeral.

Fourth, the emotional and cognitive load of repeated failure. The assistant had attempted the two-lock design multiple times: first with DeviceSync in mem_mtx (serialization), then with a fallback path in compute_mtx (72.5s regression), then with DeviceSync removed from mem_mtx (untested). Each attempt consumed build cycles, debugging time, and cognitive bandwidth. The status document serves as a circuit breaker — a moment to step back, assess what's known, and decide whether to continue or pivot.

The Decisions Captured in This Message

Message 2628 is not itself a decision point — it contains no tool calls, no code edits, no explicit choices. But it captures the decisions that were made in the preceding messages and frames the decisions that still need to be made.

The decision to split the mutex (made in earlier messages, documented here as finding #7): The gpu_locks struct with compute_mtx and mem_mtx was implemented, the function signature changed from std::mutex* to void* for ABI compatibility. This was the core architectural change of Phase 10.

The decision to add DeviceSync in mem_mtx (made in message 2613, documented here as finding #8): This was the first attempted fix for the OOM issue, and it proved counterproductive — it caused serialization because cudaDeviceSynchronize is device-global.

The decision to add DeviceSync+pool-trim in compute_mtx (made in message 2620, documented here as finding #12): When pre-staging fails and the fallback path runs inside compute_mtx, it needs to reclaim async pool memory from the previous worker. This works but doesn't solve the fundamental problem.

The decision to remove DeviceSync from mem_mtx (made in message 2626, documented here as "Current Code State"): The latest edit strips mem_mtx down to just cudaMalloc attempts with fail-fast on OOM. This edit hasn't been built or tested yet.

The implicit decision to question the entire approach (captured in the "Fundamental Design Challenge" section): The document explicitly evaluates whether the two-lock design can deliver meaningful improvement given that pre-staging will almost always fail with gw≥2. This frames the next decision: continue refining the two-lock approach, or abandon it and try something else.

Assumptions That Proved Incorrect

The Phase 10 journey is a graveyard of assumptions. Message 2628 catalogs several that turned out to be wrong:

Assumption 1: Lock splitting works on GPUs the same way it works on CPUs. In CPU programming, splitting a coarse-grained lock into multiple fine-grained locks is a standard technique for reducing contention. The assumption was that the same pattern would apply to GPU synchronization — that mem_mtx and compute_mtx could operate independently. The discovery that CUDA memory management APIs are device-global shattered this assumption. You cannot have one thread managing VRAM while another runs kernels on the same device, because cudaDeviceSynchronize, cudaMemPoolTrimTo, and even cudaMemGetInfo implicitly synchronize across all streams.

Assumption 2: Pre-staging would work with multiple workers on a 16 GiB GPU. The Phase 10 design assumed that worker B could pre-stage its 12 GiB of VRAM buffers while worker A was running kernels. But peak VRAM during H-MSM is ~13.8 GiB (d_a 4G + d_bc 8G + MSM temp 1.8G), leaving only ~2.2 GiB free. There's no room for a second worker's buffers. Finding #11 documents this: "Pre-staging will almost always fail with gw≥2."

Assumption 3: The two-lock split would yield throughput proportional to the GPU utilization gap. Phase 9 showed 48.5% GPU utilization (1.82s kernel time out of 3.76s TIMELINE). The assumption was that overlapping CPU work with GPU kernels would close this gap. But the analysis in message 2628 reveals that the CPU critical path (prep_msm 1.91s + b_g2_msm 0.48s = 2.39s) exceeds GPU time (1.82s), meaning the GPU is not actually the bottleneck — DDR5 memory bandwidth is. The real constraint is not GPU utilization but memory bandwidth contention between synthesis workers and GPU preparation threads.

Assumption 4: The fallback path would be fast enough. When pre-staging fails, the code falls back to allocating VRAM inside compute_mtx. The assumption was that this fallback would be only slightly slower than pre-staging. In reality, the fallback path produced gpu_total_ms of 3.3–3.9s (Phase 8 performance) compared to the pre-staged path's 1.8s — more than doubling GPU time.

Assumption 5: The cudaDeviceSynchronize cost in mem_mtx would be negligible. The assumption was that cudaDeviceSynchronize would be a quick ~1ms operation because the previous worker had already finished. In practice, it blocked for the entire duration of the other worker's kernel execution (hundreds of milliseconds to seconds), because cudaDeviceSynchronize waits for ALL streams on the device.

Input Knowledge Required to Understand This Message

Message 2628 is dense with domain-specific knowledge. To fully understand it, a reader needs:

Knowledge of the Filecoin PoRep proving pipeline. The message references concepts like Groth16 proofs, C1 outputs, partition synthesis, and the distinction between porep-c2 and other proof types. Understanding that a single proof involves multiple partitions (each requiring ~1.8s of GPU time) and that the pipeline processes them sequentially is essential.

Knowledge of CUDA programming and GPU architecture. The message discusses cudaDeviceSynchronize, cudaMemPoolTrimTo, cudaMemGetInfo, cudaMalloc, cudaMallocAsync, cudaFreeAsync, stream-ordered allocations, and device-global semantics. It references VRAM accounting (d_a 4G, d_bc 8G, MSM temp 1.8G), PCIe gen5 transfer characteristics, and the distinction between synchronous and asynchronous memory operations.

Knowledge of the cuzk architecture. The message references the synthesis dispatcher, GPU workers, partition workers, the gpu_prove() function, prep_msm_thread, b_g2_msm, epilogue, and the FFI boundary between Rust and C++. Understanding the relationship between cuzk-core/src/engine.rs (which spawns workers), supraseal-c2/src/lib.rs (the FFI layer), and groth16_cuda.cu (the CUDA implementation) is necessary.

Knowledge of the optimization history. The message references Phase 8 (single-worker baseline), Phase 9 (pre-staged NTT uploads + Pippenger deferred sync), and Phase 10 (two-lock interlock). Understanding what each phase achieved and where the bottlenecks shifted is critical.

Knowledge of the hardware platform. The message specifies an AMD Ryzen Threadripper PRO 7995WX (96 Zen4 cores, 8-channel DDR5 at ~300 GB/s theoretical) and an RTX 5070 Ti (Blackwell sm_120, 16 GB VRAM, PCIe gen5). The DDR5 memory bandwidth figure is particularly important — it's the constraint that ultimately limits throughput.

Output Knowledge Created by This Message

Message 2628 creates several forms of knowledge that persist beyond the immediate conversation:

A documented failure mode for two-lock GPU interlock designs. The message provides a clear, reproducible account of why splitting a GPU mutex into memory and compute locks fails on CUDA: because CUDA memory management APIs are device-global. This is a non-obvious insight that could save future engineers from repeating the same mistake. The message even generalizes it: "Any cudaMalloc/cudaFree/cudaDeviceSynchronize/cudaMemPoolTrimTo/cudaMemGetInfo in mem_mtx will interact with kernels running under compute_mtx."

A precise VRAM accounting for the 32 GiB PoRep circuit. The message documents that peak VRAM during H-MSM is ~13.8 GiB (d_a 4G + d_bc 8G + MSM temp 1.8G), and that pre-staging requires 12 GiB (d_a 4G + d_bc 8G). This accounting is essential for any future optimization that involves VRAM budgeting.

A quantitative assessment of the two-lock approach's remaining value. The message estimates that even if pre-staging always fails, the two-lock split might still help by ~0.5s per partition (hiding b_g2_msm). This is a much smaller gain than the hoped-for 1.8s parity, but it's a concrete, testable hypothesis.

A prioritized action plan. The "Not Yet Done" section lists four items in order: build and test the latest edit, evaluate whether two-lock helps, benchmark gw=3 sweep, and consider alternative approaches. This creates a clear path forward for the next phase of work.

A complete file manifest. The "Relevant Files" section catalogs every source file, config file, log file, and git commit involved in the Phase 10 work. This is invaluable for anyone who needs to reproduce the results or continue the work.

A baseline for the next optimization phase. The Phase 9 results (finding #6: DDR5 memory bandwidth is the wall) and the Phase 10 failure analysis together create a foundation for Phase 11. The message implicitly argues that the next optimization should target memory bandwidth contention rather than GPU utilization.

The Thinking Process Visible in the Message

While message 2628 is a structured document rather than a stream of consciousness, the thinking process is visible in its organization and emphasis.

The progression from specific to general. The message starts with specific Phase 9 results (GPU kernel time 1.82s, prep_msm 1.91s, b_g2_msm 0.48s), then moves to specific Phase 10 implementation findings (the gpu_locks struct, the DeviceSync serialization, the VRAM accounting), and finally arrives at the general principle: CUDA memory management APIs are device-global. This is classic inductive reasoning — building from observations to a general law.

The willingness to question the core premise. The "Fundamental Design Challenge" section is remarkable for its honesty. It explicitly states that the two-lock design may not be worth the complexity: "if pre-staging always fails with gw≥2, the two-lock design may only help by ~0.5s/partition (hiding b_g2_msm), not the hoped-for 1.8s→1.8s parity." This is not the kind of conclusion an engineer reaches easily after investing significant effort in a design. It reflects a disciplined willingness to kill a failing approach.

The separation of signal from noise. The message carefully distinguishes between problems that have been solved (finding #12: the fallback path inside compute_mtx works) and problems that remain open (finding #13: current state is slow because most partitions fall back). It also distinguishes between the proximate cause (DeviceSync in mem_mtx causes serialization) and the root cause (CUDA APIs are device-global).

The forward-looking orientation. Despite being a document about failures, the message is relentlessly forward-looking. Every finding is framed in terms of its implications for next steps. The "Current Code State" section explicitly says "The build needs to be done and tested." The "Not Yet Done" section enumerates concrete tasks. The message is not a post-mortem — it's a mid-action status check.

The meta-cognitive awareness. The message demonstrates awareness of its own limitations. It flags the untested edit explicitly: "This edit has NOT been built or tested yet." It acknowledges uncertainty: "Evaluate whether two-lock actually helps." It even suggests alternative approaches: "Maybe the two-lock split should abandon pre-staging entirely." This meta-cognitive framing — the assistant knowing what it doesn't know — is a hallmark of effective debugging.

Mistakes and Incorrect Assumptions in the Message Itself

While message 2628 is largely accurate, there are a few points worth examining critically:

The assumption that pre-staging will "almost always" fail with gw≥2. This is a reasonable inference from the VRAM accounting (13.8 GiB peak usage, 12 GiB needed for pre-staging, 16 GiB total), but it depends on the exact timing of allocations and deallocations. If worker A's d_a and d_bc are freed before worker B's pre-staging runs, there might be windows where pre-staging succeeds. The message doesn't explore this timing dependency — it assumes the worst case. This is a conservative assumption that may underestimate the two-lock design's potential.

The characterization of cudaMemGetInfo as potentially serializing. Finding #10 says "cudaMemGetInfo may also serialize: Any CUDA memory management API can implicitly synchronize." This is technically true — cudaMemGetInfo can trigger a driver-level synchronization — but in practice it's much lighter-weight than cudaDeviceSynchronize. The message lumps all CUDA memory APIs together, which may overstate the problem.

The estimate that the two-lock split helps by only ~0.5s. The message estimates that hiding b_g2_msm (0.48s) is the remaining benefit. But this ignores the Rust epilogue time that also runs outside compute_mtx. If the epilogue adds significant time, the benefit could be larger. The message doesn't quantify the epilogue cost.

The implicit assumption that Phase 11 should target memory bandwidth. The message's analysis strongly points toward DDR5 bandwidth contention as the root cause, but it doesn't fully prove this. The evidence is that prep_msm inflates 12-27% under load and b_g2_msm spikes from 380ms to 2.7-4.9s — but these could also be caused by cache contention or TLB effects. The message treats memory bandwidth as the explanation without ruling out other hypotheses.

These are minor points. The message is remarkably accurate and honest given the complexity of the situation.

The Broader Significance

Message 2628 is significant beyond its immediate context because it illustrates a pattern that recurs throughout engineering: the moment when a promising optimization meets the hard constraints of the underlying hardware.

The two-lock GPU interlock was a textbook software design pattern applied to a hardware-software boundary where the textbook doesn't apply. CUDA's device-global semantics are not a bug — they're a consequence of the GPU's architecture as a single-device execution model. The assistant's journey from "this should work" to "this doesn't work because of a fundamental property of CUDA" to "let me document exactly why" is the essence of systems engineering.

The message also demonstrates the value of structured thinking in debugging. By organizing its findings into a clear hierarchy — from specific observations to general principles to actionable next steps — the assistant creates a document that is useful both as a record of what was learned and as a guide for what to do next. This is especially valuable in AI-assisted coding, where context is ephemeral and the assistant cannot rely on its own memory across sessions.

Finally, the message is a testament to the importance of documenting failure. In a field where success stories dominate, the detailed autopsy of a failed optimization — complete with VRAM accounting, timing measurements, and honest assessments of remaining value — is a rare and valuable artifact. It teaches not just what doesn't work, but why, and what might work instead.

Conclusion

Message 2628 is a structured status report produced at a critical inflection point in the Phase 10 optimization effort. It documents the failure of the two-lock GPU interlock design, identifies the root cause (CUDA's device-global memory management semantics), and frames the decisions that need to be made next. It is notable for its honesty about failure, its willingness to question the core premise of the optimization, and its forward-looking orientation despite the setback.

The message creates lasting knowledge: a documented failure mode for two-lock GPU designs, a precise VRAM accounting for the 32 GiB PoRep circuit, a quantitative assessment of the remaining optimization headroom, and a prioritized action plan for the next phase. It consumes deep domain knowledge about CUDA programming, GPU architecture, and the Filecoin proving pipeline, and it synthesizes that knowledge into a coherent analysis that would be valuable to any engineer working on GPU-accelerated proving systems.

In the end, the Phase 10 two-lock design was abandoned. The subsequent Phase 11 would take a different approach — targeting DDR5 memory bandwidth contention directly rather than trying to overlap CPU and GPU work through lock splitting. But the knowledge captured in message 2628 — the precise understanding of why the two-lock approach failed — was essential to making that pivot. In engineering, knowing what doesn't work is often as valuable as knowing what does. Message 2628 is a masterclass in that kind of knowledge.