Chunk 27.0

In this chunk, the assistant committed the Phase 9 PCIe optimization code and, following the user's guidance, ran extensive benchmarks with higher concurrency (c=15–30) to diagnose the jumpy GPU utilization. Fine-grained timing instrumentation was added to the pre-staging path, revealing that the pre-staging setup itself is negligible (~18ms) and that GPU kernel time dropped dramatically to ~1.8s per partition. However, the steady-state throughput plateaued at ~41s/proof, and the c=30 run crashed due to memory pressure. The detailed timing analysis uncovered the critical bottleneck shift: the CPU critical path—`prep_msm` (1.9s) and `b_g2_msm` (0.48s)—now dominates the per-partition wall time at ~2.4s, leaving the GPU idle for ~600ms per partition waiting for the CPU thread. At high concurrency, the 10 synthesis workers compete with the CPU MSM operations for 8-channel DDR5 memory bandwidth, inflating CPU times by 2–12×. The bottleneck has moved from PCIe transfers and GPU kernel execution to CPU memory bandwidth contention. To address this, the user proposed a two-lock design to better overlap CPU and GPU work. The assistant analyzed the deadlock risks (resource deadlock on VRAM) and designed a simplified protocol using a `mem_mtx` for VRAM allocation/upload and a `compute_mtx` for kernel execution. Since `d_a` and `d_bc` are freed synchronously inside `compute_mtx`, the next worker can safely allocate in `mem_mtx` without a cooperative free slot. The assistant wrote `c2-optimization-proposal-10.md` detailing this Phase 10 plan, which increases `gpu_workers_per_device` to 3 and restructures the lock regions to hide the CPU overhead, with an expected per-partition wall time reduction from ~3.7s to ~1.8–2.0s and a potential 30–38% throughput improvement in isolation.

The Bottleneck Shift: How Phase 9's PCIe Optimization Revealed the CPU Memory Wall and Sparked Phase 10's Two-Lock Architecture 2293 words

Message Articles

Subagent Sessions