From Failed Locks to Memory Bandwidth: The Phase 10 Post-Mortem and the Birth of Phase 11

Introduction

The cuzk SNARK proving engine optimization campaign is a story of systematic, data-driven engineering applied to one of the most computationally demanding problems in the Filecoin ecosystem: generating Groth16 proofs for Proof-of-Replication (PoRep). Over nine prior phases, the assistant and user had progressively optimized the pipeline, reducing proof generation time from well over a minute to approximately 32 seconds per proof in isolation. Each phase targeted a specific bottleneck: GPU kernel efficiency, PCIe transfer bandwidth, CPU-side synchronization overhead. By Phase 9, the dominant constraint had become DDR5 memory bandwidth contention, with CPU-side operations like prep_msm and b_g2_msm competing with synthesis workers for access to the system's 8-channel memory.

Phase 10 was supposed to be the next leap forward. The design was elegant on paper: split the single GPU mutex into two locks — compute_mtx for GPU kernel execution and mem_mtx for VRAM allocation — allowing multiple workers to overlap their CPU-side preparation work with another worker's GPU kernel execution. But as this chunk of the conversation reveals in excruciating detail, the two-lock design collided with fundamental constraints of CUDA's architecture and the hardware's physical limits. The result was a regression from 32.1 seconds to 73.8 seconds per proof, a comprehensive post-mortem that identified DDR5 memory bandwidth contention as the true bottleneck, and the birth of Phase 11 — a three-intervention plan grounded in empirical waterfall timing analysis.

This article traces that arc: from the abandonment of the flawed two-lock design, through the comprehensive benchmarking and waterfall analysis that revealed the real bottleneck, to the design of Phase 11's targeted interventions. It is a case study in how disciplined measurement transforms failure into insight.

Part I: The Collapse of Phase 10

The Two-Lock Dream

The Phase 10 design was rooted in a clear observation from Phase 9's benchmarking. The GPU kernel time per partition was 1.82 seconds, but the total TIMELINE span per partition was 3.76 seconds. The gap was filled by CPU-side work: prep_msm at 1.91 seconds and b_g2_msm at 0.48 seconds. GPU silicon utilization was only 48.5% — the GPU spent nearly half its time waiting for the CPU to prepare data.

The solution seemed obvious: split the single GPU mutex so that while Worker A held compute_mtx and ran GPU kernels, Worker B could hold mem_mtx to pre-stage VRAM buffers for the next partition, and Worker C could be doing CPU-side preparation (b_g2_msm, epilogue) outside any lock. With three GPU workers per device (gw=3), the theory predicted near-perfect overlap, pushing throughput toward the GPU's true silicon limit.

The implementation involved significant changes to groth16_cuda.cu: replacing the single std::mutex* parameter with a void* pointing to a gpu_locks struct containing both mutexes, restructuring the pre-staging and kernel execution regions to use different locks, and adding fallback paths for when pre-staging under mem_mtx failed. The design was documented in c2-optimization-proposal-10.md, and the code was carefully crafted across multiple editing rounds.

The First Signs of Trouble

Even as the code was being written, warning signs emerged. The assistant's own design document ([msg 2628]) catalogued thirteen discoveries, including several that directly challenged the viability of the two-lock approach:

Discovery #8: cudaDeviceSynchronize in mem_mtx causes serialization. This function is a device-global operation — it blocks until ALL GPU streams complete, including those running under another worker's compute_mtx. When Worker B called cudaDeviceSynchronize inside mem_mtx to ensure the GPU was ready for new allocations, it blocked until Worker A's kernels finished, completely defeating the purpose of the lock split.

Discovery #9: cudaMemPoolTrimTo is also device-global. The async memory pool is shared across all streams on the device. Trimming it from one worker affects all workers.

Discovery #11: Pre-staging will almost always fail with gw≥2. On a 16 GiB GPU, peak VRAM during H-MSM reaches approximately 13.8 GiB (d_a at 4 GiB, d_bc at 8 GiB, MSM temp at 1.8 GiB). With only ~2.2 GiB of headroom, there is no room for a second worker's pre-staged buffers (12 GiB for d_a + d_bc). The pre-staging path would always fall back to the non-prestaged path.

Despite these warnings, the assistant pressed forward. The last edit removed cudaDeviceSynchronize, cudaMemGetInfo, and pool trim from mem_mtx, replacing them with a simpler "try cudaMalloc directly, fail fast on OOM" approach. This edit had not been built or tested.

The Build That Succeeded Into a Dead End

The build sequence (<msg id=2634-2636>) was tense but successful. The assistant force-rebuilt the CUDA code by deleting cached build artifacts (rm -rf target/release/build/supraseal-c2-*), then compiled the daemon and benchmark tools. The only warning was a pre-existing cfg condition issue in an unrelated dependency (bellpepper-core). The code compiled cleanly.

The daemon was started with gpu_workers_per_device = 3 ([msg 2639]), and the assistant waited 15 seconds for SRS parameter loading and GPU initialization ([msg 2640]). Everything looked normal. The stage was set for the moment of truth.

The 73.8-Second Canary

The first benchmark ([msg 2644]) was devastating:

=== Batch Summary ===
total time:    73.8s
completed:     1
failed:        0
throughput:    0.813 proofs/min (73.8s/proof)

A single proof took 73.8 seconds — more than double Phase 9's 32.1-second baseline. The proof passed correctness (the mathematics were sound), but the performance was catastrophic. The two-lock design was 2.3× slower than the single-lock approach it was supposed to replace.

The Diagnostic Grep

The assistant immediately investigated ([msg 2645]), running a targeted grep against the daemon log:

grep -E "(CUZK_TIMING|prestage|gpu_total|PROOF|partition_ms|GPU_START|GPU_END)" /home/theuser/cuzk-p10-daemon.log | tail -60

The output contained the smoking gun:

CUZK_TIMING: prestage_setup=skip_vram
CUZK_TIMING: pre_destructor_ms=11444
TIMELINE,177264,GPU_END,807490ec-...,worker=1,partition=4,gpu_ms=11444

Every partition's pre-staging attempt was skipped because VRAM was insufficient (prestage_setup=skip_vram). The pre_destructor_ms of 11.4 seconds per partition — compared to Phase 9's 1.82-second GPU kernel time — showed that the fallback path inside compute_mtx was catastrophically slow. The cudaDeviceSynchronize call in the fallback serialized all GPU work across all streams, turning the two-lock design into a single-lock design with extra overhead.

The individual operations were healthy: prep_msm_ms=1744 (1.74s), b_g2_msm_ms=371 (0.37s), async_dealloc_ms=824 (0.82s). The problem was structural: they all executed sequentially under compute_mtx because the mem_mtx path always failed.

The Fundamental Design Challenge

The Phase 10 failure revealed a fundamental truth about CUDA programming on consumer GPUs: memory management APIs are device-global and cannot be isolated by user-space mutexes. The cudaDeviceSynchronize, cudaMemPoolTrimTo, and even cudaMemGetInfo functions operate at the device level, not the stream level. They interact with all streams on the device, including those running under another worker's compute_mtx. No amount of lock splitting can escape this architectural constraint.

Combined with the VRAM limitation — 16 GiB is insufficient to accommodate pre-staged buffers from multiple workers when a single worker peaks at ~13.8 GiB — the two-lock approach was fundamentally unsalvageable on this hardware. The design was not wrong in principle; on a multi-GPU system with abundant VRAM, splitting memory management from compute could yield real benefits. But on a single RTX 5070 Ti with 16 GB VRAM, it was doomed from the start.

Part II: The Reversion and the Benchmarking Campaign

Reverting to Phase 9

The decision to abandon Phase 10 was reached through a combination of theoretical analysis and empirical evidence. The assistant's structured status report ([msg 2628]) had already laid out the fundamental design challenge. The 73.8-second benchmark provided the empirical confirmation. The code was reverted to Phase 9's proven single-lock approach (<msg id=2664-2667>), and the working tree was cleaned.

But the reversion was not a retreat — it was a pivot. The assistant now had a much deeper understanding of the system's constraints. The question was: if the GPU lock wasn't the bottleneck, what was?

The Comprehensive Benchmark Sweep

To answer this question, the assistant ran a comprehensive benchmark sweep across concurrency levels (<msg id=2670-2675>), from c=5 j=5 through c=20 j=15. The results revealed a clear pattern: throughput plateaued at approximately 38 seconds per proof, regardless of concurrency. Adding more workers did not improve throughput — it only increased contention.

The key metric was GPU utilization. At low concurrency (c=5 j=5), GPU utilization was modest. At high concurrency (c=20 j=15), it reached 90.8% — the GPU was nearly fully utilized. But throughput was still stuck at ~38s/proof. The bottleneck was not the GPU; the GPU was waiting for data from the CPU, and the CPU was waiting for memory.

The Waterfall Timing Analysis

The breakthrough came when the assistant performed a waterfall timing analysis (<msg id=2676-2686>), extracting TIMELINE events from the daemon logs and plotting them as a Gantt chart. This visualization revealed the true bottleneck with stunning clarity.

The waterfall showed that under load, both synthesis and prep_msm inflated dramatically. Synthesis time increased from ~2 seconds in isolation to 5-8 seconds under load. Prep_msm inflated from ~1.9 seconds to 2.4-3.0 seconds. These two operations — both heavy memory readers — were competing for the same resource: DDR5 memory bandwidth.

The system has an AMD Ryzen Threadripper PRO 7995WX with 96 Zen4 cores and 8-channel DDR5 memory, providing approximately 300 GB/s theoretical bandwidth. But when 192 rayon synthesis threads and 192 groth16_pool threads (from b_g2_msm) all perform heavy memory reads simultaneously, the memory subsystem saturates. The result is that both operations inflate, and the GPU — which needs the output of both — sits idle waiting for data.

The waterfall analysis revealed that b_g2_msm was particularly problematic. Although it only takes ~0.4 seconds in isolation, under load it can spike to 2.7-4.9 seconds because all 192 groth16_pool threads run Pippenger's algorithm simultaneously, competing with synthesis threads for L3 cache and memory bandwidth. This 0.4-second window is the critical bottleneck: during that window, the memory bandwidth contention is at its peak.

The DDR5 Bandwidth Wall

The conclusion was clear: the bottleneck was not GPU lock contention, not PCIe transfer bandwidth, not CPU core count — it was DDR5 memory bandwidth. The system had reached the memory bandwidth wall. No amount of GPU optimization could improve throughput when the GPU was spending 90.8% of its time waiting for data that the CPU couldn't deliver fast enough because the memory subsystem was saturated.

This insight transformed the optimization strategy. Instead of trying to overlap CPU and GPU work through clever locking (Phase 10's approach), the next phase would need to reduce the memory bandwidth contention itself. The question became: what specific operations are causing the contention, and how can they be mitigated?

Part III: Designing Phase 11

Researching the Contention Points

With the bottleneck identified, the assistant began researching the specific code paths responsible for memory bandwidth contention (<msg id=2690-2708>). This involved reading source files, tracing thread pool configurations, and understanding the parallelism model of the Groth16 proving pipeline.

Three major contention points emerged:

1. Async deallocation storms. The async_dealloc mechanism frees GPU memory asynchronously, but the Rust side uses munmap() to release CPU-side mappings. When 192 rayon threads all call munmap() concurrently, the result is a TLB (Translation Lookaside Buffer) shootdown storm — the kernel must flush TLB entries across all cores, causing massive overhead. The solution: bound async deallocation to a single thread.

2. Thread pool oversubscription in b_g2_msm. The groth16_pool thread pool defaults to 192 threads (matching the rayon synthesis pool). When b_g2_msm runs, all 192 threads execute Pippenger's algorithm simultaneously, competing with the 192 synthesis threads for L3 cache capacity and memory bandwidth. The solution: reduce the groth16_pool thread count (via gpu_threads = 32) to shrink b_g2_msm's memory footprint and L3 cache pressure.

3. Memory-phase overlap between b_g2_msm and synthesis. The b_g2_msm operation's ~0.4-second window is the period of peak memory bandwidth contention, when all 192 groth16_pool threads run Pippenger simultaneously with 192 rayon synthesis threads. The solution: add a lightweight atomic throttle flag to briefly pause synthesis workers during the b_g2_msm window, reducing competition for memory bandwidth.

The Critical Refinement: `par_map` with 1 Item

The original Phase 11 design included a more aggressive intervention: a full semaphore interlock between prep_msm and synthesis, pausing synthesis workers during the entire prep_msm window. This was based on the assumption that prep_msm was heavily parallelized and contributed significantly to memory bandwidth contention.

But then the assistant made a critical discovery by reading the source code (<msg id=2709-2717>). The prep_msm function, when called with num_circuits=1, uses par_map with a single item. In the bellpepper library, par_map with one item runs on a single thread — there is no parallelism at all. The entire prep_msm operation, which takes ~1.9 seconds, runs on one thread.

This discovery reshaped the optimization strategy. If prep_msm is single-threaded, it cannot be a significant contributor to memory bandwidth contention. The full semaphore interlock between prep_msm and synthesis was overkill. The three-intervention plan was revised to target only the actual contention points: TLB shootdowns from deallocation storms, thread pool oversubscription in b_g2_msm, and the brief memory-phase overlap between b_g2_msm and synthesis.

The Three Interventions

The final Phase 11 design, documented in c2-optimization-proposal-11.md, consists of three targeted interventions:

Intervention 1: Bound async deallocation to a single thread. The async_dealloc mechanism currently allows all 192 rayon threads to call munmap() concurrently, causing TLB shootdown storms. By serializing deallocation through a single thread, the TLB flush overhead is eliminated. This is expected to improve throughput by approximately 3-5%.

Intervention 2: Reduce the groth16_pool thread count. The gpu_threads configuration parameter controls the size of the groth16_pool thread pool. The default is 192 threads, matching the rayon pool. By reducing it to 32 threads (via gpu_threads = 32), b_g2_msm's memory footprint and L3 cache pressure are reduced, decreasing competition with synthesis threads. This is expected to improve throughput by approximately 2-4%.

Intervention 3: Add a lightweight atomic throttle flag. During b_g2_msm's ~0.4-second window, all 192 groth16_pool threads run Pippenger simultaneously with 192 rayon synthesis threads. A lightweight atomic flag briefly pauses synthesis workers during this window, reducing competition for memory bandwidth. This is expected to improve throughput by approximately 2-3%.

The combined expected improvement is 7-12%, potentially bringing proof time from ~38 seconds to ~34-35 seconds at high concurrency.

Documentation and Roadmap

The Phase 11 design was documented in c2-optimization-proposal-11.md, a detailed design spec covering all three interventions with implementation steps, risk assessments, and expected throughput improvements. The cuzk-project.md file was updated with a Phase 10 post-mortem (documenting why the two-lock design failed) and a Phase 11 roadmap entry. All documentation changes were committed to git.

The user approved the revised three-intervention plan, and the assistant is positioned to implement Phase 11 with careful benchmarking of each intervention independently — ensuring that the impact of each change can be measured in isolation before combining them.

Lessons and Methodology

The Value of Negative Results

The Phase 10 failure is not a waste — it is a valuable data point. The knowledge that CUDA memory management APIs are device-global and cannot be isolated by user-space mutexes is a hard-won insight that will inform all future optimization on this platform. The precise VRAM accounting (13.8 GiB peak, 16 GiB total) is now documented. The understanding that the fallback path adds catastrophic overhead when pre-staging fails is quantified.

In engineering, knowing what doesn't work is often as valuable as knowing what does. The Phase 10 post-mortem ensures that this knowledge is preserved and that the same mistakes are not repeated.

The Power of Instrumentation

The entire waterfall analysis that identified DDR5 bandwidth contention was only possible because of the instrumentation infrastructure built in Phase 9. The CUZK_TIMING log points, the TIMELINE events, the prestage_setup flags — these were not added for Phase 10; they were already in place, providing the diagnostic data needed to understand Phase 10's failure and to discover the true bottleneck.

This is a crucial lesson for any optimization effort: invest in measurement infrastructure early. The time spent adding timing instrumentation is repaid many times over when things go wrong.

The Discipline of Data-Driven Optimization

The arc from Phase 10 to Phase 11 exemplifies a disciplined, scientific approach to optimization:

Form a hypothesis: The two-lock GPU interlock will improve throughput by overlapping CPU and GPU work.
Implement the hypothesis: Write the code, build, and deploy.
Test the hypothesis: Run benchmarks and measure the outcome.
Analyze the results: The hypothesis is falsified (73.8s vs 32.1s baseline).
Diagnose the root cause: The grep reveals prestage_setup=skip_vram and 11.4s pre_destructor_ms.
Form a new hypothesis: DDR5 memory bandwidth contention is the true bottleneck.
Test the new hypothesis: The waterfall timing analysis confirms that synthesis and b_g2_msm inflate under load due to memory bandwidth competition.
Design targeted interventions: Three interventions to reduce contention, each independently benchmarkable.
Document everything: The Phase 10 post-mortem and Phase 11 design spec are committed to the project repository. This cycle — hypothesis, experiment, analysis, new hypothesis — is the scientific method applied to systems engineering. It is slow, methodical, and sometimes painful (especially when a promising design fails). But it produces reliable knowledge, and it prevents the wasted effort of pursuing dead ends.

The Importance of Reading the Code

One of the most instructive moments in this chunk is the discovery that prep_msm with num_circuits=1 is single-threaded. The assistant initially assumed it was heavily parallelized and designed a full semaphore interlock to pause synthesis during prep_msm. But reading the actual source code revealed that par_map with one item runs on one thread — the entire ~1.9-second operation is serial.

This discovery was only possible because the assistant read the code rather than relying on assumptions. It is a reminder that in complex systems, assumptions about performance characteristics are often wrong. The only reliable source of truth is the code itself, combined with empirical measurement.

Conclusion

This chunk of the cuzk optimization conversation is a masterclass in systematic engineering. It begins with the collapse of a promising but fundamentally flawed design, traces through the disciplined reversion and comprehensive benchmarking that identified the true bottleneck, and culminates in the design of a targeted, evidence-based optimization plan.

The Phase 10 two-lock GPU interlock failed because it violated a fundamental constraint of CUDA's architecture: memory management APIs are device-global and cannot be isolated by user-space mutexes. The Phase 11 three-intervention plan succeeds because it targets the actual bottleneck revealed by empirical measurement: DDR5 memory bandwidth contention between synthesis threads and b_g2_msm threads competing for the same memory channels.

The contrast between Phase 10 and Phase 11 is instructive. Phase 10 was an architectural redesign — clever, ambitious, and wrong. Phase 11 is a set of targeted interventions — modest, grounded in data, and likely to work. The difference is not in the sophistication of the design but in the quality of the diagnosis. Phase 10 was designed based on a plausible theory about where the bottleneck might be. Phase 11 was designed based on measured evidence of where the bottleneck actually is.

This is the essence of data-driven optimization: measure first, then optimize. The Phase 10 failure was not a setback — it was the measurement that made Phase 11 possible.