The Bottleneck Shift: How Phase 9's PCIe Optimization Revealed the CPU Memory Wall and Sparked Phase 10's Two-Lock Architecture

Introduction

In the relentless pursuit of faster Groth16 proof generation for Filecoin's Proof-of-Replication (PoRep) protocol, the cuzk optimization campaign had reached a critical inflection point. Phase 9's PCIe transfer optimization had just achieved a stunning 51% reduction in GPU kernel time—from 3.75 seconds to 1.82 seconds per partition. Yet the system's steady-state throughput stubbornly plateaued at approximately 41 seconds per proof, barely 14% better than the Phase 8 baseline. Something fundamental had changed: the bottleneck had shifted.

This chunk of the conversation captures the moment when the team realized that the GPU was no longer the limiting factor. Through a series of increasingly sophisticated benchmarks, timing instrumentations, and awk-powered log analyses, they discovered that the true bottleneck had migrated to a far more stubborn constraint: CPU memory bandwidth contention on an 8-channel DDR5 system. This discovery led directly to the design of Phase 10's two-lock architecture—an ambitious attempt to overlap CPU and GPU work through finer-grained locking—and ultimately to the humbling realization that CUDA's device-global synchronization model imposes limits that no software abstraction can circumvent.

The Commit That Checkpointed Progress

The chunk opens with the assistant committing the Phase 9 PCIe optimization code to git—a process that proved surprisingly involved. The assistant first checked git status and discovered that while groth16_cuda.cu was modified, the other Phase 9 files (groth16_ntt_h.cu and the Pippenger MSM changes in the sppark submodule) were untracked. This triggered a multi-step process: committing the submodule changes separately, then staging everything together, only to discover that the sppark submodule path was gitignored, requiring a force-add with -f [12].

The commit message itself was a carefully crafted summary of Phase 9's achievements, including detailed performance numbers: the 71.6% reduction in ntt_msm_h_ms (from 2430ms to 690ms), the 61.3% reduction in gpu_total_ms, and the explicit acknowledgment that "gw=2 shows regression (41.0s) due to cudaDeviceSynchronize + pool trim serialization—needs further investigation" [11]. This commit was not merely a log entry; it was a knowledge artifact that crystallized the current state of understanding, capturing what was accomplished, by how much, and what remained unsolved.

The user's observation at this point was remarkably prescient. Noting "jumpy and inconsistent gpu use," the user hypothesized that the GPU had become so fast that synthesis herding was creating visible gaps in utilization [2]. The suggestion was to run benchmarks with higher concurrency (15–30 synthesis workers) and more proofs to let the pipeline stabilize. This diagnostic insight would prove to be the key that unlocked the true bottleneck.

The Diagnostic Benchmark Campaign

Following the user's guidance, the assistant launched a systematic benchmark sweep at concurrency levels of 15, 20, and 30 synthesis workers. The results were initially confusing: at c=15, prove times ranged from 37–45 seconds—worse than the small-run Phase 9 result of 32.1 seconds [23]. Queue times were growing, indicating that proofs were piling up faster than they could be processed. At c=30, the system crashed with an Out-of-Memory error, revealing the limits of the 754 GiB system memory under extreme load [70].

The assistant turned to the TIMELINE instrumentation built into the cuzk daemon—structured event logs that record every GPU_START, GPU_END, SYNTH_START, and SYNTH_END event with precise timestamps. Using a series of grep, sed, sort, and awk pipelines, the assistant extracted per-partition GPU timing data and computed statistical distributions [34]. The results were eye-opening:

• Average GPU time per partition: 3,672ms (3.7s)
• Median (p50): 3,218ms
• P90: 5,445ms
• P99: 7,474ms
• Maximum: 8,232ms The 3× ratio between median and maximum revealed a severe long tail in GPU compute times. But more importantly, the theoretical throughput calculation—multiplying the average partition time by 10 partitions per proof—yielded 36.7 seconds per proof, remarkably close to the observed ~41 seconds. This meant the GPU was already operating near its compute limit. The bottleneck was no longer in PCIe transfers or kernel launches; it was in the CPU-side operations that fed the GPU.

Tracing the Phantom 1.9 Seconds

The assistant then added fine-grained timing instrumentation to the pre-staging path, measuring each phase of the per-partition workflow with microsecond precision [52]. The results told a stark story:

• GPU kernels: 1,824 ms average
• prep_msm (CPU): 1,909 ms average
• b_g2_msm (CPU): 484 ms average
• Pre-stage setup: 18 ms average (negligible)
• CPU critical path: 2,393 ms (prep_msm + b_g2_msm) The GPU was finishing its work in 1.8 seconds, then sitting idle for 600 milliseconds per partition waiting for the CPU thread to complete prep_msm_thread.join(). The CPU had become the bottleneck—specifically, the 8-channel DDR5 memory bandwidth, saturated by 10 concurrent synthesis workers all competing for the same memory bus [61]. At high concurrency, the situation worsened dramatically. The synthesis workers and CPU MSM operations were competing for memory bandwidth, inflating CPU times by 2–12×. The assistant's summary laid out the theoretical floors with brutal clarity:
• 100% GPU utilization: 18.2s/proof (unreachable—CPU can't feed it)
• CPU-limited: 24s/proof (prep_msm + b_g2_msm serial)
• Observed with contention: 36–41s/proof The pre-staging optimization had been a success, but it had only revealed the next wall: the DDR5 bandwidth ceiling [102].

The User's Provocative Proposal

It was at this moment that the user sent a message that would redirect the entire optimization effort. The proposal was deceptively simple: split the single GPU mutex into two independent locks—one for compute (GPU kernel execution) and one for memory management (VRAM allocation and pre-staging uploads)—and increase the number of GPU workers from 2 to 3 or 4 [103].

The reasoning was elegant. With the existing single-mutex design, Phase 9's pre-staging operations (allocating 12 GiB of VRAM and uploading data) had been added inside the same lock that protected GPU kernel execution. This meant that while one worker held the mutex to pre-stage memory, no other worker could run kernels. The two-lock design would allow Worker A to run GPU kernels under compute_mtx while Worker B allocated and uploaded the next partition's data under mem_mtx, and Worker C did CPU-only preprocessing with no lock at all.

The user's message ended with a crucial warning: "needs to be ordered correctly to not deadleck." This single sentence would drive an entire phase of analysis.

The Resource Deadlock Analysis

The assistant's response to the user's proposal was methodical and thorough. In message 2570, the assistant articulated the three-role model that would become the conceptual foundation of Phase 10 [104]:

• Worker A: holds compute lock, running GPU kernels
• Worker B: holds memory lock, doing pre-stage alloc + upload + pool trim
• Worker C: doing CPU-only work (prep_msm, b_g2_msm, host register)—no lock needed But the assistant immediately identified the critical risk: lock ordering and deadlock. In message 2574, the assistant spelled out the exact deadlock scenario [108]:

Worker A: holds compute_mtx, wants mem_mtx to free d_bc (8 GiB)
Worker B: holds mem_mtx, wants to alloc 12 GiB but VRAM is full
         (waiting for Worker A to free d_bc first)
         Then wants compute_mtx (which A holds)

This was, as the assistant correctly identified, "a classic resource deadlock—not a lock-ordering deadlock, but a resource capacity deadlock." The distinction was crucial. A lock-ordering deadlock occurs when threads acquire locks in inconsistent orders; that can be fixed by enforcing a global lock hierarchy. A resource deadlock arises from a finite pool of a resource (here, VRAM) combined with the inability to release it without holding a lock that someone else needs. No amount of lock reordering can fix it—the fundamental problem is that there isn't enough VRAM for two workers' buffers to coexist.

The assistant explored a cooperative handoff design using a shared to_free slot, but ultimately realized that because d_a and d_bc are freed synchronously via cudaFree inside compute_mtx, the VRAM is genuinely available the moment compute_mtx is released. No cooperative free slot is needed [117].

The Simplified Two-Lock Design

The final design that emerged from this analysis was remarkably clean. The assistant traced through the entire VRAM lifecycle, mapping each allocation and deallocation to specific lines of code, and arrived at a simplified flow:

[no lock]     cudaHostRegister + prep_msm_thread starts
lock(mem_mtx)
  pool_trim + cudaMemGetInfo
  cudaMalloc 12 GiB + async upload + events
unlock(mem_mtx)
lock(compute_mtx)
  stream.wait(events), NTT kernels
  cudaFree(d_bc)
  H-MSM (uses d_a)
  gpu_ptr_t(d_a) destructor → cudaFree
  batch_add + tail_msm
unlock(compute_mtx)
[no lock]     destroy events + stream
              cudaHostUnregister
              prep_msm_thread.join()
              Epilogue

No nested locks, no cooperative free slot, no deadlock risk. The two locks were acquired and released in strict sequence: mem_mtx first, then compute_mtx. Because d_a and d_bc were freed synchronously inside compute_mtx, the VRAM was genuinely available when the next worker acquired mem_mtx [117].

The assistant documented this design in c2-optimization-proposal-10.md, complete with a risk assessment table, testing plan, and expected outcomes. The predicted improvement was a 30–38% throughput gain in isolation, reducing per-partition wall time from 3.7 seconds to 1.8–2.0 seconds.

The Commit Point and the Gap Between Design and Reality

When the user instructed the assistant to "Write down c2-optimization-plan-10.md, then proceed to implement," the assistant responded with a todowrite command—a structured task list marking the transition from design to execution [120]:

Write c2-optimization-proposal-10.md design spec [status: in_progress]
Implement two-lock struct (gpu_locks) in groth16_cuda.cu [status: pending]
Restructure generate_groth16_proofs_c with mem_mtx + compute_mtx [status: pending]
Update engine.rs default gpu_workers_per_device to 3 [status: pending]

This was the commit point. The assistant was signaling: "I understand the plan, I accept it, and I am now organizing the work into concrete, actionable steps." The todowrite mechanism was not merely a note-taking device—it was a commitment protocol.

But this commit point also marked the moment when several critical assumptions were implicitly endorsed. The design assumed that memory management and compute could be cleanly separated on a single CUDA device—that cudaMemPoolTrimTo and cudaMemGetInfo inside mem_mtx would not conflict with kernel execution happening under compute_mtx. It assumed that pool trim alone was sufficient without cudaDeviceSynchronize. It assumed that three workers would not exacerbate DDR5 contention beyond acceptable levels.

These assumptions would prove incorrect. As the subsequent implementation revealed, cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global operations that interact with the entire CUDA context, not just a single stream or worker. When the Phase 10 code was built and tested, only the first worker successfully pre-staged VRAM. Subsequent workers found insufficient free memory because the first worker's 12 GiB allocation was still live—the device-wide synchronization inside mem_mtx blocked while another worker held compute_mtx and ran kernels, effectively serializing the two locks and destroying the intended overlap.

The performance regression was dramatic: prove time ballooned to 102 seconds, and OOM failures occurred. The assistant ultimately diagnosed the fundamental conflict: memory management operations on a single CUDA device cannot be fully isolated from compute operations. The solution was to remove the device-wide synchronization from the mem_mtx region entirely, relying instead on the already-proven fallback path inside compute_mtx when pre-staging fails.

The Deeper Lessons

This chunk of the optimization journey offers several profound lessons about systems engineering at the frontier of hardware capacity.

First, optimization is a waterfall process. Each improvement reveals the next bottleneck deeper in the system. Phase 9 made the GPU dramatically faster, which exposed CPU memory bandwidth as the new constraint. The two-lock design attempted to address this, but it in turn revealed that CUDA's device-global synchronization model imposes fundamental limits on how much parallelism can be achieved on a single GPU.

Second, hardware constraints override software abstractions. The two-lock design was logically sound by every analytical measure. It identified the deadlock scenario, traced through the VRAM lifecycle, verified that cudaFree is synchronous, and designed a lock protocol with no nested locks and no deadlock risk. Yet it failed because the CUDA device itself is a single, shared resource with global synchronization semantics. No amount of careful lock ordering can work around the fact that cudaDeviceSynchronize affects all streams and all outstanding operations.

Third, the gap between design and reality is where the real learning happens. The commit point captured in this chunk is valuable precisely because it preserves the moment before failure—the point at which a team commits to a design that seems correct by every available analytical tool. The subsequent debugging revealed the true nature of the constraint, leading to a deeper understanding of the hardware and a more robust solution.

Fourth, simple tools can reveal complex truths. The assistant's use of grep, sed, sort, and awk pipelines to extract timing distributions from log files is a masterclass in pragmatic diagnostics. The 3.7-second barrier, the 3× spread between median and maximum GPU times, the 600ms idle gap—these insights came not from a sophisticated profiling tool but from decades-old Unix utilities applied with surgical precision.

Conclusion

This chunk of the cuzk optimization journey captures a pivotal transition: from the triumph of Phase 9's PCIe optimization, through the sobering discovery of the CPU memory bandwidth wall, to the ambitious design of Phase 10's two-lock architecture, and finally to the humbling realization that CUDA's device-global synchronization model imposes limits that no software abstraction can circumvent.

The story is not one of failure but of deepening understanding. Each iteration revealed new constraints, each measurement exposed a new layer of the system's behavior, and each design iteration incorporated the lessons of the previous one. The two-lock architecture, though it did not work as intended on the first attempt, set the stage for the refined design that would eventually emerge—one that acknowledges the device-global nature of CUDA memory management and works within it, rather than fighting against it.

In the end, this chunk is a testament to the iterative nature of systems optimization. The bottleneck shifts, the assumptions are tested, the designs evolve, and through it all, the understanding deepens. That is the essence of engineering at the frontier of performance.