When the Benchmark Breaks: Diagnosing OOM Failures in Phase 9's Dual-Worker GPU Pipeline

The Message

In the middle of an intensive optimization campaign targeting the cuzk SNARK proving engine for Filecoin PoRep, the assistant issued the following benchmark command and received a devastating result:

[assistant] Daemon is ready. Now run the benchmark:
[bash] /home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch -t porep --c1 /data/32gbench/c1.json -c 5 -j 3 2>&1 | tee /tmp/cuzk-phase9-bench.log

=== Batch Benchmark ===
proof type:  porep
count:       5
concurrency: 3

  [1/5] FAILED — 39.5s (prove=0 ms, queue=0 ms)
  [2/5] FAILED — 75.4s (prove=0 ms, queue=0 ms)
  [3/5] FAILED — 110.0s (prove=0 ms, queue=0 ms)
  [4/5] FAILED — 100.3s (prove=0 ms, queue=0 ms)
  [5/5] FAILED — 95.4s (prove=0 ms, queue=0 ms)

Every single proof failed. The prove=0 ms timestamps are particularly telling — they indicate that the proving engine itself never successfully completed a single partition's GPU work. The queue times (39.5s, 75.4s, 110.0s, etc.) show that proofs were being submitted and eventually timing out, but the actual GPU computation never produced a result. This was not a gradual degradation or a flaky test — it was a complete, systematic failure of the Phase 9 optimization under the dual-worker configuration.

Context: What Led to This Moment

This message represents the culmination of a multi-phase optimization campaign documented across segments 21 through 26 of the cuzk proving engine project. The assistant had been systematically working through a series of optimization proposals, each targeting a specific bottleneck in the Groth16 proof generation pipeline.

Phase 8 had established a baseline with a dual-worker GPU interlock design that achieved 13–17% throughput improvement over the original single-worker architecture. A subsequent TIMELINE analysis (Segment 25) identified two root causes of remaining GPU utilization dips: non-pinned host memory causing PCIe transfer stalls, and hard synchronization stalls in the Pippenger MSM's per-batch sync() calls.

Phase 9 was designed to address both issues. The assistant implemented two changes:

  1. Change 1 (Tier 1): Moved the 6 GiB of a/b/c polynomial uploads out of the GPU mutex by pinning host memory with cudaHostRegister, allocating device buffers, and issuing async cudaMemcpyAsync transfers on a dedicated stream with event-based synchronization. The idea was to overlap data transfer with the other worker's GPU compute time.
  2. Change 2 (Tier 3): Eliminated per-batch hard sync stalls in the Pippenger MSM by introducing double-buffered host result buffers (res_buf[2], ones_buf[2]) and deferring the sync() call to the next iteration, allowing GPU compute to overlap with DtoH transfers. The single-worker benchmark had been spectacularly successful: NTT+MSM time dropped from ~2430 ms to ~690 ms (a 71.6% reduction), tail MSM from ~125 ms to ~82 ms (34.4% reduction), and overall GPU time per partition from ~3746 ms to ~1450–1900 ms (50–61% reduction). System throughput improved from 37.4 s/proof to 32.1 s/proof, a 14.2% gain. Emboldened by these results, the assistant launched the full production benchmark with gpu_workers_per_device=2 and concurrency=3 — the intended dual-worker configuration that would stress-test the optimization under real-world conditions. Message 2417 is the record of that benchmark launch and its catastrophic failure.

Why This Message Matters

This message is a pivotal moment in the optimization campaign. It represents the collision between a theoretically sound optimization and the messy realities of GPU memory management under concurrent access. The single-worker benchmark had validated the kernel-level improvements, but it had completely failed to expose the resource contention issue that would emerge under dual-worker operation.

The message is also a masterclass in debugging methodology. The assistant does not panic, does not revert changes, does not blame the tooling. Instead, it immediately pivots to investigation. In the very next message ([msg 2418]), the assistant checks the daemon logs and finds the critical clue: prestage_setup=fallback err=2 — a CUDA memory allocation failure (error code 2 is cudaErrorMemoryAllocation). The pre-staging path had fallen back, and then the original path also failed with OOM.

The Root Cause Analysis

The assistant's subsequent analysis ([msg 2419]) reveals a subtle but critical flaw in the Phase 9 design. The pre-staging allocation happened before the GPU mutex was acquired, meaning both workers could attempt to allocate their 6 GiB of device memory simultaneously. On a 16 GiB GPU, two concurrent 6 GiB allocations (12 GiB total) would consume nearly all available VRAM, leaving nothing for the CUDA kernels themselves.

The assistant's thinking process is remarkable for its clarity and systematic reasoning:

"The problem is clear: both workers try to pre-stage simultaneously. With 2 gpu_workers_per_device, worker 0 allocates 6 GiB for pre-staging, then worker 1 also tries to allocate 6 GiB — that's 12 GiB just for pre-staging on a 16 GiB GPU, leaving nothing for the rest."

The assistant then walks through the memory accounting carefully:

"On a 16 GiB GPU: - Worker doing CUDA kernels: needs ~7.5 GiB peak - Worker pre-staging: needs ~6 GiB That's 13.5 GiB — it fits! The problem is when both workers try to pre-stage at the same time (before either holds the mutex)."

This is a crucial insight. The optimization was designed assuming that pre-staging would overlap with the other worker's kernel execution (7.5 + 6 = 13.5 GiB, which fits). But the actual failure mode was both workers pre-staging simultaneously (6 + 6 = 12 GiB, plus then kernel memory on top), which exceeds capacity.

The assistant considers several fixes: a separate lightweight mutex for allocation, using the main mutex with a try_lock pattern, or moving pre-staging inside the mutex. Each option is evaluated for its correctness and performance implications.

Assumptions and Their Violations

The Phase 9 design rested on several assumptions that proved incorrect under dual-worker load:

  1. Assumption: Pre-staging outside the mutex is safe. The design assumed that because pre-staging only allocates memory and initiates async transfers, it could run concurrently with another worker's kernel execution. This was true in principle but failed to account for the case where both workers reach the pre-staging phase simultaneously.
  2. Assumption: Memory fragmentation is manageable. The CUDA memory pool model (cudaMallocAsync/cudaFreeAsync) does not necessarily release freed memory back to the synchronous cudaMalloc pool. After a fallback from the pre-staging path, the freed memory might not be available for subsequent allocations, compounding the OOM issue.
  3. Assumption: Single-worker results generalize. The spectacular single-worker gains created an implicit expectation that dual-worker performance would be proportionally better. The complete failure in dual-worker mode was a stark reminder that concurrency introduces new failure modes that single-threaded testing cannot expose.

The Design Pivot

The assistant's response to the failure is instructive. Rather than abandoning the optimization or scaling back to single-worker mode, the assistant redesigns the pre-staging approach:

"Solution: Do pre-staging inside the mutex acquisition flow. Specifically: 1. Acquire the GPU mutex 2. While holding it, allocate device buffers + issue async uploads 3. The uploads run on the copy engine while we immediately start the compute kernels"

This is a significant architectural shift. The original design aimed to overlap pre-staging with another worker's kernel time. The new design accepts that pre-staging must be serialized, but retains the benefits of pinned memory and async transfers. The key insight is that on a single GPU with a single copy engine, true overlap between workers is limited anyway — the real gains come from avoiding bounce buffers and blocking transfers.

The assistant also acknowledges a mistake in the original specification:

"The key insight from the spec was wrong about the overlap — on a single GPU, there's only one copy engine, and it can overlap with compute but only for the same worker's streams."

This self-correction is a hallmark of effective engineering: the willingness to revise one's mental model when evidence contradicts it.

Input and Output Knowledge

To understand this message, the reader needs knowledge of:

The Broader Significance

This message is a microcosm of the entire optimization campaign. It demonstrates the iterative, hypothesis-driven approach that characterizes the work: propose an optimization, implement it, test it, analyze the results, and refine. The failure is not a setback but a data point — one that reveals a deeper truth about the system's behavior under concurrent load.

The assistant's response to failure is particularly noteworthy. There is no frustration, no blame, no hasty revert. Instead, there is a calm, methodical diagnosis followed by a redesigned approach that preserves the core benefits of the optimization while addressing the newly discovered constraint. This is the essence of systems engineering: understanding that every optimization introduces trade-offs, and that the only way to discover those trade-offs is to test under realistic conditions.

The message also highlights the gap between kernel-level and system-level performance. The individual GPU kernel times were dramatically improved, but the system-level throughput under concurrency revealed a bottleneck that the kernel metrics could not predict. This is a recurring theme in high-performance computing: optimizing a component in isolation can produce misleading results, and end-to-end testing under realistic load is essential for validating real-world gains.

Conclusion

Message 2417 is the moment when Phase 9's theoretical elegance collided with physical reality. The benchmark's complete failure was not a sign of a bad optimization, but rather an indication that the optimization's assumptions needed refinement. The assistant's subsequent diagnosis and redesign — moving pre-staging inside the mutex, adding memory-aware allocation, and freeing buffers early — would eventually produce a working dual-worker implementation with validated throughput improvements.

In the broader narrative of the cuzk optimization campaign, this message represents the critical transition from "does it work in isolation?" to "does it work under load?" — a transition that every production system must eventually make. The failure was not the end of Phase 9, but the beginning of its maturation into a robust, production-ready optimization.