From Two Locks to Bandwidth Wars: The Phase 10 Post-Mortem and Phase 11 Memory-Bandwidth Optimization Plan

Introduction

The cuzk SNARK proving engine optimization campaign represents one of the most intensive, data-driven performance engineering efforts in the Filecoin ecosystem. Over ten prior phases, the assistant and user had progressively optimized the Groth16 proof generation pipeline for Proof-of-Replication (PoRep), reducing proof time from well over a minute to approximately 32 seconds per proof in isolation. Each phase targeted a specific bottleneck: GPU kernel efficiency, PCIe transfer bandwidth, CPU-side synchronization overhead, and synthesis micro-optimizations. By Phase 9, the system was humming—but a new bottleneck had emerged.

Phase 10 was supposed to be the next leap forward. The design was elegant on paper: split the single GPU mutex into two locks—compute_mtx for GPU kernel execution and mem_mtx for VRAM allocation—allowing multiple workers to overlap their CPU-side preparation work with another worker's GPU kernel execution. But as this segment reveals in excruciating detail, the two-lock design collided with fundamental constraints of CUDA's architecture and the hardware's physical limits.

This article traces the arc of Segment 28: from the abandonment of the flawed two-lock design, through the disciplined reversion to Phase 9's proven approach, the comprehensive benchmarking campaign that revealed the true bottleneck, the waterfall timing analysis that pinpointed DDR5 memory bandwidth contention, and finally the design and documentation of Phase 11's three targeted interventions. It is a case study in how disciplined measurement transforms failure into insight, and how the scientific method applied to systems engineering produces reliable knowledge.

Part I: The Collapse of Phase 10

The Two-Lock Dream

The Phase 10 design was rooted in a clear observation from Phase 9's benchmarking. The GPU kernel time per partition was 1.82 seconds, but the total TIMELINE span per partition was 3.76 seconds. The gap was filled by CPU-side work: prep_msm at 1.91 seconds and b_g2_msm at 0.48 seconds. GPU silicon utilization was only 48.5%—the GPU spent nearly half its time waiting for the CPU to prepare data.

The solution seemed obvious: split the single GPU mutex so that while Worker A held compute_mtx and ran GPU kernels, Worker B could hold mem_mtx to pre-stage VRAM buffers for the next partition, and Worker C could be doing CPU-side preparation (b_g2_msm, epilogue) outside any lock. With three GPU workers per device (gw=3), the theory predicted near-perfect overlap, pushing throughput toward the GPU's true silicon limit.

The implementation involved significant changes to groth16_cuda.cu: replacing the single std::mutex* parameter with a void* pointing to a gpu_locks struct containing both mutexes, restructuring the pre-staging and kernel execution regions to use different locks, and adding fallback paths for when pre-staging under mem_mtx failed. The design was documented in c2-optimization-proposal-10.md, and the code was carefully crafted across multiple editing rounds.

The First Signs of Trouble

Even as the code was being written, warning signs emerged. The assistant's own design document catalogued thirteen discoveries, including several that directly challenged the viability of the two-lock approach:

Discovery #8: cudaDeviceSynchronize in mem_mtx causes serialization. This function is a device-global operation—it blocks until ALL GPU streams complete, including those running under another worker's compute_mtx. When Worker B called cudaDeviceSynchronize inside mem_mtx to ensure the GPU was ready for new allocations, it blocked until Worker A's kernels finished, completely defeating the purpose of the lock split.

Discovery #9: cudaMemPoolTrimTo is also device-global. The async memory pool is shared across all streams on the device. Trimming it from one worker affects all workers.

Discovery #11: Pre-staging will almost always fail with gw≥2. On a 16 GiB GPU, peak VRAM during H-MSM reaches approximately 13.8 GiB (d_a at 4 GiB, d_bc at 8 GiB, MSM temp at 1.8 GiB). With only ~2.2 GiB of headroom, there is no room for a second worker's pre-staged buffers (12 GiB for d_a + d_bc). The pre-staging path would always fall back to the non-prestaged path.

Despite these warnings, the assistant pressed forward. The last edit removed cudaDeviceSynchronize, cudaMemGetInfo, and pool trim from mem_mtx, replacing them with a simpler "try cudaMalloc directly, fail fast on OOM" approach. This edit had not been built or tested.

The Build That Succeeded Into a Dead End

The build sequence was tense but successful. The assistant force-rebuilt the CUDA code by deleting cached build artifacts (rm -rf target/release/build/supraseal-c2-*), then compiled the daemon and benchmark tools. The only warning was a pre-existing cfg condition issue in an unrelated dependency (bellpepper-core). The code compiled cleanly.

The daemon was started with gpu_workers_per_device = 3, and the assistant waited 15 seconds for SRS parameter loading and GPU initialization. Everything looked normal. The stage was set for the moment of truth.

The 73.8-Second Canary

The first benchmark was devastating:

=== Batch Summary ===
total time:    73.8s
completed:     1
failed:        0
throughput:    0.813 proofs/min (73.8s/proof)

A single proof took 73.8 seconds—more than double Phase 9's 32.1-second baseline. The proof passed correctness (the mathematics were sound), but the performance was catastrophic. The two-lock design was 2.3× slower than the single-lock approach it was supposed to replace.

The Diagnostic Grep

The assistant immediately investigated, running a targeted grep against the daemon log:

grep -E "(CUZK_TIMING|prestage|gpu_total|PROOF|partition_ms|GPU_START|GPU_END)" /home/theuser/cuzk-p10-daemon.log | tail -60

The output contained the smoking gun:

CUZK_TIMING: prestage_setup=skip_vram
CUZK_TIMING: pre_destructor_ms=11444
TIMELINE,177264,GPU_END,807490ec-...,worker=1,partition=4,gpu_ms=11444

Every partition's pre-staging attempt was skipped because VRAM was insufficient (prestage_setup=skip_vram). The pre_destructor_ms of 11.4 seconds per partition—compared to Phase 9's 1.82-second GPU kernel time—showed that the fallback path inside compute_mtx was catastrophically slow. The cudaDeviceSynchronize call in the fallback serialized all GPU work across all streams, turning the two-lock design into a single-lock design with extra overhead.

The individual operations were healthy: prep_msm_ms=1744 (1.74s), b_g2_msm_ms=371 (0.37s), async_dealloc_ms=824 (0.82s). The problem was structural: they all executed sequentially under compute_mtx because the mem_mtx path always failed.

The Fundamental Design Challenge

The Phase 10 failure revealed a fundamental truth about CUDA programming on consumer GPUs: memory management APIs are device-global and cannot be isolated by user-space mutexes. The cudaDeviceSynchronize, cudaMemPoolTrimTo, and even cudaMemGetInfo functions operate at the device level, not the stream level. They interact with all streams on the device, including those running under another worker's compute_mtx. No amount of lock splitting can escape this architectural constraint.

Combined with the VRAM limitation—16 GiB is insufficient to accommodate pre-staged buffers from multiple workers when a single worker peaks at ~13.8 GiB—the two-lock approach was fundamentally unsalvageable on this hardware. The design was not wrong in principle; on a multi-GPU system with abundant VRAM, splitting memory management from compute could yield real benefits. But on a single RTX 5070 Ti with 16 GB VRAM, it was doomed from the start.

Part II: The Reversion and the Benchmarking Campaign

Reverting to Phase 9

The decision to abandon Phase 10 was reached through a combination of theoretical analysis and empirical evidence. The assistant's structured status report had already laid out the fundamental design challenge. The 73.8-second benchmark provided the empirical confirmation. The code was reverted to Phase 9's proven single-lock approach, and the working tree was cleaned.

But the reversion was not a retreat—it was a pivot. The assistant now had a much deeper understanding of the system's constraints. The question was: if the GPU lock wasn't the bottleneck, what was?

The Comprehensive Benchmark Sweep

To answer this question, the assistant ran a comprehensive benchmark sweep across concurrency levels, from c=5 j=5 through c=20 j=15. The results revealed a clear pattern: throughput plateaued at approximately 38 seconds per proof, regardless of concurrency. Adding more workers did not improve throughput—it only increased contention.

The key metric was GPU utilization. At low concurrency (c=5 j=5), GPU utilization was modest. At high concurrency (c=20 j=15), it reached 90.8%—the GPU was nearly fully utilized. But throughput was still stuck at ~38s/proof. The bottleneck was not the GPU; the GPU was waiting for data from the CPU, and the CPU was waiting for memory.

The Waterfall Timing Analysis

The breakthrough came when the assistant performed a waterfall timing analysis, extracting TIMELINE events from the daemon logs and plotting them as a Gantt chart. This visualization revealed the true bottleneck with stunning clarity.

The waterfall showed that under load, both synthesis and prep_msm inflated dramatically. Synthesis time increased from ~2 seconds in isolation to 5-8 seconds under load. Prep_msm inflated from ~1.9 seconds to 2.4-3.0 seconds. These two operations—both heavy memory readers—were competing for the same resource: DDR5 memory bandwidth.

The system has an AMD Ryzen Threadripper PRO 7995WX with 96 Zen4 cores and 8-channel DDR5 memory, providing approximately 300 GB/s theoretical bandwidth. But when 192 rayon synthesis threads and 192 groth16_pool threads (from b_g2_msm) all perform heavy memory reads simultaneously, the memory subsystem saturates. The result is that both operations inflate, and the GPU—which needs the output of both—sits idle waiting for data.

The waterfall analysis revealed that b_g2_msm was particularly problematic. Although it only takes ~0.4 seconds in isolation, under load it can spike to 2.7-4.9 seconds because all 192 groth16_pool threads run Pippenger's algorithm simultaneously, competing with synthesis threads for L3 cache and memory bandwidth. This 0.4-second window is the critical bottleneck: during that window, the memory bandwidth contention is at its peak.

The DDR5 Bandwidth Wall

The conclusion was clear: the bottleneck was not GPU lock contention, not PCIe transfer bandwidth, not CPU core count—it was DDR5 memory bandwidth. The system had reached the memory bandwidth wall. No amount of GPU optimization could improve throughput when the GPU was spending 90.8% of its time waiting for data that the CPU couldn't deliver fast enough because the memory subsystem was saturated.

This insight transformed the optimization strategy. Instead of trying to overlap CPU and GPU work through clever locking (Phase 10's approach), the next phase would need to reduce the memory bandwidth contention itself. The question became: what specific operations are causing the contention, and how can they be mitigated?

Part III: Designing Phase 11

Researching the Contention Points

With the bottleneck identified, the assistant began researching the specific code paths responsible for memory bandwidth contention. This involved reading source files, tracing thread pool configurations, and understanding the parallelism model of the Groth16 proving pipeline.

Three major contention points emerged:

1. Async deallocation storms. The async_dealloc mechanism frees GPU memory asynchronously, but the Rust side uses munmap() to release CPU-side mappings. When 192 rayon threads all call munmap() concurrently, the result is a TLB (Translation Lookaside Buffer) shootdown storm—the kernel must flush TLB entries across all cores, causing massive overhead. The solution: bound async deallocation to a single thread.

2. Thread pool oversubscription in b_g2_msm. The groth16_pool thread pool defaults to 192 threads (matching the rayon synthesis pool). When b_g2_msm runs, all 192 threads execute Pippenger's algorithm simultaneously, competing with the 192 synthesis threads for L3 cache capacity and memory bandwidth. The solution: reduce the groth16_pool thread count (via gpu_threads = 32) to shrink b_g2_msm's memory footprint and L3 cache pressure.

3. Memory-phase overlap between b_g2_msm and synthesis. The b_g2_msm operation's ~0.4-second window is the period of peak memory bandwidth contention, when all 192 groth16_pool threads run Pippenger simultaneously with 192 rayon synthesis threads. The solution: add a lightweight atomic throttle flag to briefly pause synthesis workers during the b_g2_msm window, reducing competition for memory bandwidth.

The Critical Refinement: `par_map` with 1 Item

The original Phase 11 design included a more aggressive intervention: a full semaphore interlock between prep_msm and synthesis, pausing synthesis workers during the entire prep_msm window. This was based on the assumption that prep_msm was heavily parallelized and contributed significantly to memory bandwidth contention.

But then the assistant made a critical discovery by reading the source code. The prep_msm function, when called with num_circuits=1, uses par_map with a single item. In the bellpepper library, par_map with one item runs on a single thread—there is no parallelism at all. The entire prep_msm operation, which takes ~1.9 seconds, runs on one thread.

This discovery reshaped the optimization strategy. If prep_msm is single-threaded, it cannot be a significant contributor to memory bandwidth contention. The full semaphore interlock between prep_msm and synthesis was overkill. The three-intervention plan was revised to target only the actual contention points: TLB shootdowns from deallocation storms, thread pool oversubscription in b_g2_msm, and the brief memory-phase overlap between b_g2_msm and synthesis.

The Three Interventions

The final Phase 11 design, documented in c2-optimization-proposal-11.md, consists of three targeted interventions:

Intervention 1: Bound async deallocation to a single thread. The async_dealloc mechanism currently allows all 192 rayon threads to call munmap() concurrently, causing TLB shootdown storms. By serializing deallocation through a single thread, the TLB flush overhead is eliminated. The assistant's research into the Rust-side deallocation code (lines 388–396 of supraseal.rs) revealed that both the C++ side (~37 GiB freed via std::thread(...).detach()) and the Rust side (~130 GiB of ProvingAssignment data) perform background-thread deallocations. Both create the same problem: unbounded concurrent deallocation threads that trigger TLB shootdown storms. The fix is to add a static std::mutex dealloc_mtx to the C++ code and a separate static Mutex<()> on the Rust side, serializing deallocation to a single thread at a time. Expected improvement: 3-5%.

Intervention 2: Reduce the groth16_pool thread count. The gpu_threads configuration parameter controls the size of the groth16_pool thread pool. The default is 192 threads, matching the rayon pool. By reducing it to 32 threads (via gpu_threads = 32), b_g2_msm's memory footprint and L3 cache pressure are reduced, decreasing competition with synthesis threads. With ~22 million G2 points and a window size of ~15 bits, each thread allocates roughly 6 MB of bucket RAM. Across 192 threads, that's ~1.1 GiB of bucket RAM—a significant allocation that evicts useful data from the L3 cache. With 32 threads, total bucket RAM drops to ~192 MB—a 6× reduction. The impact on b_g2_msm is a slowdown from ~0.4 seconds to 0.5-0.7 seconds, but since b_g2_msm runs after the GPU lock is released and concurrent with the next worker's GPU kernels, this slowdown does not affect GPU throughput. Expected improvement: 2-4%.

Intervention 3: Add a lightweight atomic throttle flag. During b_g2_msm's ~0.4-second window, all 192 groth16_pool threads run Pippenger simultaneously with 192 rayon synthesis threads. A lightweight atomic flag briefly pauses synthesis workers during this window, reducing competition for memory bandwidth. The assistant explored four implementation options—shared atomic flag via FFI, Rust-side tokio::sync::Notify, rayon yield_now() heuristic, and reduced rayon parallelism for MatVec—and recommended Option A (shared atomic flag via FFI) as the most precise and controllable approach. The overhead analysis showed that with one check per 64 chunks (each chunk is 8192 rows), that's ~250 checks per SpMV call, each a single atomic load with acquire semantics—negligible overhead. Expected improvement: 2-3%.

The combined expected improvement is 7-12%, potentially bringing proof time from ~38 seconds to ~34-35 seconds at high concurrency.

Part IV: Documenting the Plan

The Research Phase: Reading Before Writing

Before writing the design document, the assistant engaged in systematic information gathering. This involved reading the Rust-side deallocation code to understand the dual-deallocation architecture, examining the FFI boundary structure to understand b_g2_msm/prep_msm timing, verifying that the gpu_threads configuration parameter is properly plumbed through the system, examining the SpMV evaluation code to understand where the throttle check should be inserted, and reading the synthesis pipeline to trace how the throttle pointer would be threaded through.

Each of these read operations served a specific purpose in grounding the implementation plan in the actual codebase structure. The assistant was not designing from abstract principles; it was designing from concrete code analysis. This is the hallmark of disciplined performance engineering: verify assumptions against code before writing the plan.

The Design Specification

The assistant wrote c2-optimization-proposal-11.md, a detailed design specification covering all three interventions with implementation details, risk assessments, and expected outcomes. The document includes a comprehensive summary table mapping each intervention to the specific files that need modification, spanning four programming languages (C++, CUDA, Rust, and TOML configuration) and crossing multiple library boundaries.

The risk analysis is thorough. For Intervention 1, the assistant considered the scenario where dealloc takes 2.9 seconds and partition completion happens every 2 seconds—the dealloc mutex becomes a bottleneck with one queued dealloc. But this is acceptable because the detached threads simply pile up slightly, with only one running at a time. For Intervention 2, the assistant considered the impact on b_g2_msm latency and correctly identified that the slowdown is hidden behind GPU kernel execution. For Intervention 3, the assistant considered the overhead of atomic checks and confirmed it is negligible.

The Project Roadmap Update

The assistant also updated cuzk-project.md with the Phase 10 post-mortem and Phase 11 roadmap entry. The Phase 10 post-mortem documents why the two-lock design failed—the fundamental CUDA device-global synchronization conflicts and the VRAM limitation—ensuring that future developers understand why this approach was tried and why it failed, preventing repeated exploration of the same dead end.

The Phase 11 roadmap entry describes the three interventions, their expected impact, and the implementation order. The assistant verified the cross-referencing between documents, confirming that the Phase 11 entry correctly references the new proposal document.

Git Hygiene

The chunk concludes with disciplined git hygiene. The assistant ran git status, git diff --stat, and git log --oneline -5 to understand the repository state before committing. This revealed a nuanced situation: extern/supraseal-c2/cuda/groth16_cuda.cu was already staged (from the Phase 10 reversion), while cuzk-project.md was modified but unstaged (the new documentation). The assistant investigated the staged change more closely, running git diff --cached --stat to confirm that it consisted of 2 insertions and 17 deletions—exactly the signature of the Phase 10→Phase 9 revert. This verification step ensured that the working tree was in the correct state before proceeding with Phase 11 implementation.

Lessons and Methodology

The Value of Negative Results

The Phase 10 failure is not a waste—it is a valuable data point. The knowledge that CUDA memory management APIs are device-global and cannot be isolated by user-space mutexes is a hard-won insight that will inform all future optimization on this platform. The precise VRAM accounting (13.8 GiB peak, 16 GiB total) is now documented. The understanding that the fallback path adds catastrophic overhead when pre-staging fails is quantified.

In engineering, knowing what doesn't work is often as valuable as knowing what does. The Phase 10 post-mortem ensures that this knowledge is preserved and that the same mistakes are not repeated.

The Power of Instrumentation

The entire waterfall analysis that identified DDR5 bandwidth contention was only possible because of the instrumentation infrastructure built in Phase 9. The CUZK_TIMING log points, the TIMELINE events, the prestage_setup flags—these were not added for Phase 10; they were already in place, providing the diagnostic data needed to understand Phase 10's failure and to discover the true bottleneck.

This is a crucial lesson for any optimization effort: invest in measurement infrastructure early. The time spent adding timing instrumentation is repaid many times over when things go wrong.

The Discipline of Data-Driven Optimization

The arc from Phase 10 to Phase 11 exemplifies a disciplined, scientific approach to optimization:

Form a hypothesis: The two-lock GPU interlock will improve throughput by overlapping CPU and GPU work.
Implement the hypothesis: Write the code, build, and deploy.
Test the hypothesis: Run benchmarks and measure the outcome.
Analyze the results: The hypothesis is falsified (73.8s vs 32.1s baseline).
Diagnose the root cause: The grep reveals prestage_setup=skip_vram and 11.4s pre_destructor_ms.
Form a new hypothesis: DDR5 memory bandwidth contention is the true bottleneck.
Test the new hypothesis: The waterfall timing analysis confirms that synthesis and b_g2_msm inflate under load due to memory bandwidth competition.
Design targeted interventions: Three interventions to reduce contention, each independently benchmarkable.
Document everything: The Phase 10 post-mortem and Phase 11 design spec are committed to the project repository. This cycle—hypothesis, experiment, analysis, new hypothesis—is the scientific method applied to systems engineering. It is slow, methodical, and sometimes painful (especially when a promising design fails). But it produces reliable knowledge, and it prevents the wasted effort of pursuing dead ends.

The Importance of Reading the Code

One of the most instructive moments in this segment is the discovery that prep_msm with num_circuits=1 is single-threaded. The assistant initially assumed it was heavily parallelized and designed a full semaphore interlock to pause synthesis during prep_msm. But reading the actual source code revealed that par_map with one item runs on one thread—the entire ~1.9-second operation is serial.

This discovery was only possible because the assistant read the code rather than relying on assumptions. It is a reminder that in complex systems, assumptions about performance characteristics are often wrong. The only reliable source of truth is the code itself, combined with empirical measurement.

Conclusion

Segment 28 of the cuzk optimization conversation is a masterclass in systematic engineering. It begins with the collapse of a promising but fundamentally flawed design, traces through the disciplined reversion and comprehensive benchmarking that identified the true bottleneck, and culminates in the design and documentation of a targeted, evidence-based optimization plan.

The Phase 10 two-lock GPU interlock failed because it violated a fundamental constraint of CUDA's architecture: memory management APIs are device-global and cannot be isolated by user-space mutexes. The Phase 11 three-intervention plan succeeds because it targets the actual bottleneck revealed by empirical measurement: DDR5 memory bandwidth contention between synthesis threads and b_g2_msm threads competing for the same memory channels.

The contrast between Phase 10 and Phase 11 is instructive. Phase 10 was an architectural redesign—clever, ambitious, and wrong. Phase 11 is a set of targeted interventions—modest, grounded in data, and likely to work. The difference is not in the sophistication of the design but in the quality of the diagnosis. Phase 10 was designed based on a plausible theory about where the bottleneck might be. Phase 11 was designed based on measured evidence of where the bottleneck actually is.

This is the essence of data-driven optimization: measure first, then optimize. The Phase 10 failure was not a setback—it was the measurement that made Phase 11 possible.

References

[1] From Failed Locks to Memory Bandwidth: The Phase 10 Post-Mortem and the Birth of Phase 11 — Chunk 0 article covering the abandonment of Phase 10, reversion to Phase 9, benchmarking campaign, waterfall timing analysis, and Phase 11 design.

[2] From Failure to Blueprint: Documenting the Phase 11 Memory-Bandwidth Optimization Plan — Chunk 1 article covering the research phase, design specification writing, project roadmap update, and git hygiene.