The Validation That Revealed a Deeper Bottleneck: Phase 9's Dual-Worker Benchmark

On February 19, 2026, at 04:53:22 UTC, an AI assistant issued a single bash command that would upend the assumptions of an optimization cycle spanning days of work. The command was straightforward:

/home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch -t porep --c1 /data/32gbench/c1.json -c 5 -j 3 2>&1 | tee /tmp/cuzk-phase9-gw2-bench.log

This was the production benchmark for Phase 9: PCIe Transfer Optimization, the latest in a series of increasingly sophisticated optimizations to the cuzk SNARK proving engine — a CUDA-accelerated Groth16 proof generator for Filecoin's Proof-of-Replication (PoRep). The command launched five proofs at concurrency 3 with two GPU workers per device (gpu_workers_per_device=2), the intended production configuration. The assistant had just spent hours fixing out-of-memory (OOM) crashes, debugging CUDA memory pool interactions, and implementing memory-aware allocation. Now it was time to see if the optimizations held up under real load.

What the output revealed was deeply counterintuitive — and far more valuable than a simple success.

The Context: What Phase 9 Was Trying to Solve

To understand this message, one must understand the problem it was designed to solve. The Phase 8 baseline had established a throughput of 37.4 seconds per proof using dual GPU workers. But detailed timeline analysis (documented in the preceding segment) had identified two root causes of GPU idle gaps:

  1. Non-pinned host memory transfers: The a/b/c polynomial data (6 GiB total) was being uploaded from host to device using synchronous cudaMemcpy calls while holding the GPU mutex. These transfers consumed critical GPU time that could have been used for computation.
  2. Pippenger MSM sync stalls: Each batch in the multi-scalar multiplication (MSM) ended with a hard sync() call that blocked the GPU pipeline, preventing overlap between compute and host-to-device (HtoD) transfers. Phase 9 attacked both problems. Change 1 (Tier 1) moved the a/b/c uploads out of the GPU mutex by pinning host memory with cudaHostRegister, allocating device buffers, and issuing asynchronous cudaMemcpyAsync transfers on a dedicated stream with CUDA event-based synchronization. Change 2 (Tier 3) eliminated per-batch sync stalls in the Pippenger MSM by introducing double-buffered host result buffers (res_buf[2], ones_buf[2]) and deferring the sync() call to the next iteration, allowing GPU compute to overlap with device-to-host (DtoH) transfers.

The Road to This Benchmark

The path to this message was anything but smooth. Initial testing of Phase 9 with a single GPU worker (gpu_workers_per_device=1) had produced spectacular results: the NTT+MSM time dropped from ~2430 ms to ~690 ms (a 71.6% reduction), tail MSM from ~125 ms to ~82 ms (a 34.4% reduction), and overall GPU time per partition from ~3746 ms to ~1450–1900 ms (a 50–61% reduction). Throughput improved from 37.4 s/proof to 32.1 s/proof — a 14.2% improvement.

But when the assistant tried to run with dual workers, it hit immediate OOM failures. The root cause was subtle: the pre-staging logic allocated 12 GiB of VRAM (4 GiB for d_a + 8 GiB for d_bc) upfront, and with two workers both trying to pre-stage simultaneously, the 16 GiB GPU ran out of memory. Worse, CUDA's cudaMallocAsync/cudaFreeAsync memory pools did not release freed memory back to the synchronous cudaMalloc pool, causing subsequent allocations to fail even after cleanup.

The assistant fixed both issues by:

The Results: A Surprising Regression

The benchmark completed with all five proofs passing — no failures, no OOM crashes. The output showed:

  [1/5] COMPLETED — 60.0s (prove=47245 ms, queue=258 ms)
  [2/5] COMPLETED — 103.7s (prove=60315 ms, queue=708 ms)
  [3/5] COMPLETED — 133.5s (prove=50167 ms, queue=1155 ms)
  [4/5] COMPLETED — 113.0s (prove=60350 ms, queue=250 ms)
  [5/5] COMPLETED — 101.2s (prove=50468 ms, ...

The system throughput worked out to 41.0 seconds per proof. This was worse than the Phase 8 baseline of 37.4 s/proof, and significantly worse than the single-worker Phase 9 result of 32.1 s/proof.

This was a stunning reversal. The kernel-level optimizations were working beautifully — the GPU timing logs (examined in the next message, [msg 2465]) showed gpu_total_ms around 1,616 ms per partition, ntt_msm_h_ms between 588–855 ms, and tail_msm_ms around 90 ms. The individual GPU operations were dramatically faster. Yet the end-to-end throughput regressed.

What This Revealed: The Next Bottleneck Layer

The gap between kernel-level and system-level performance told a clear story. The GPU kernel times had improved by 50–61%, but the overall throughput got worse under dual-worker concurrency. This meant that PCIe bandwidth contention — the very thing Phase 9 was designed to alleviate — had simply moved from being a GPU-level bottleneck to being a system-level bottleneck.

With two workers both trying to upload 6 GiB of polynomial data via the PCIe bus, the transfers were competing for bandwidth. The cudaDeviceSynchronize + pool trim operations at mutex acquisition were adding latency that serialized both workers' uploads. The pre-staging allocation and upload, which happened while holding the mutex, created a new serialization point that the original non-pinned path (where each worker only allocated when it actually held the mutex) did not have.

More broadly, the benchmark revealed that the optimization strategy had reached a point of diminishing returns. The GPU was no longer the bottleneck — CPU-side processing, PCIe bandwidth, and daemon scheduling were now the limiting factors. The assistant's immediate pivot to examining the detailed timing logs in the very next message ([msg 2465]) demonstrates a methodical, data-driven approach: when a hypothesis fails, you don't guess — you measure.

Assumptions and Their Violations

The assistant made several assumptions in this message, some explicit and some implicit:

  1. That kernel-level improvements would translate to system-level gains: This was the core assumption of Phase 9, and it held for single-worker mode but broke under dual-worker concurrency. The assumption was reasonable — GPU time dominated the critical path in Phase 8 — but it failed to account for the new serialization points introduced by the optimization itself.
  2. That the OOM fix was complete: The memory-aware allocator and pool trimming logic successfully prevented OOM crashes (all 5 proofs completed), validating the fix. But the fix introduced latency that partially negated the optimization's benefits.
  3. That gpu_workers_per_device=2 would outperform gw=1: This was the production configuration that had been optimal in Phase 8. The assistant assumed the same would hold for Phase 9, but the new pre-staging logic changed the contention dynamics.
  4. That PCIe bandwidth was sufficient for dual concurrent uploads: The 6 GiB per worker × 2 workers = 12 GiB of concurrent HtoD traffic exceeded what the PCIe bus could sustain without introducing queuing delays.

Input Knowledge Required

To fully understand this message, a reader needs:

Output Knowledge Created

This message produced several critical pieces of knowledge:

  1. Phase 9 dual-worker throughput: 41.0 s/proof — a regression from the Phase 8 baseline of 37.4 s/proof.
  2. Validation of the OOM fix: All 5 proofs completed with 0 failures, confirming that the memory-aware allocator and pool trimming logic work correctly under load.
  3. Evidence of PCIe bandwidth contention as the new bottleneck: The gap between kernel-level GPU times (~1,616 ms per partition) and system-level throughput (41.0 s/proof) indicates that PCIe transfers, CPU-side processing, or daemon scheduling are now the limiting factors.
  4. A boundary condition for the optimization strategy: The pre-staging approach that works brilliantly for single-worker mode introduces serialization overhead that negates its benefits in dual-worker mode.

The Thinking Process Visible in the Reasoning

While the message itself is just a bash command and its output, the reasoning behind it is visible in the surrounding conversation. The assistant had just spent messages 2441–2463 debugging OOM failures, diagnosing CUDA memory pool interactions, and implementing the memory-aware allocator. The decision to run with -c 5 -j 3 (5 proofs, concurrency 3) was deliberate — it matches the production configuration used for the Phase 8 baseline, enabling a direct comparison.

The assistant's thinking, visible in the preceding messages, shows a methodical debugging process: measuring free VRAM, calculating allocation sizes, tracing the interaction between async and sync memory pools, and implementing a fallback path. The pivot after this benchmark to examining detailed timing logs (in [msg 2465]) shows that the assistant treats unexpected results as data, not failures.

Conclusion

This single bash command — one line in a conversation spanning thousands of messages — encapsulates a critical moment in the optimization of a high-performance GPU proving engine. It validated that the kernel-level optimizations work (all proofs passed), but it also revealed that the optimization strategy had reached a new frontier. The GPU was no longer the bottleneck; PCIe bandwidth contention and CPU-side serialization now dominated.

The message is a reminder that in systems optimization, the most valuable results are often the ones that contradict your assumptions. The 41.0 s/proof throughput was worse than the baseline, but it pointed the way to the next layer of optimization: PCIe bandwidth management, buffer reuse across partitions, and potentially a rethinking of the dual-worker architecture itself. The assistant would go on to investigate these issues in the following messages, but this benchmark marked the moment when the team learned that Phase 9's gains, while real at the kernel level, required a different approach to translate into system-level improvements under multi-worker concurrency.