The Hypothesis That Failed: Debugging a GPU OOM at the Crossroads of Concurrency and Memory Management

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, every millisecond counts and every byte of VRAM is precious. The SUPRASEAL_C2 Groth16 proving pipeline for Filecoin's Proof-of-Replication (PoRep) operates at the edge of hardware limits, routinely allocating 12+ GiB of GPU memory for a single partition's polynomial data. When the Phase 9 PCIe transfer optimization was implemented to reduce GPU idle time by overlapping host-to-device transfers with computation, the expected throughput gains were immediately visible — but so was a perplexing out-of-memory (OOM) failure that resisted straightforward diagnosis. Message [msg 2444] captures a pivotal moment in this debugging journey: the moment when a carefully formed hypothesis about concurrency-induced OOM was decisively disproven by a single benchmark run.

The Message

The message itself is deceptively simple — a bash command launching a benchmark with reduced concurrency parameters, followed by its output:

/home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch -t porep \
  --c1 /data/32gbench/c1.json -c 3 -j 1 2>&1 | tee /tmp/cuzk-phase9-gw1-bench.log

The benchmark configuration is: concurrency=1 (one proof at a time), gpu_workers_per_device=1 (one GPU worker thread), and count=3 (three proofs total). The output is unambiguous:

=== Batch Benchmark ===
proof type:  porep
count:       3
concurrency: 1

  [1/3] FAILED — 40.7s (prove=0 ms, queue=0 ms)
  [2/3] FAILED — 43.3s (prove=0 ms, queue=0 ms)
  [3/3] FAILED — 41.3s (prove=0 ms, queue=0 ms)

=== Batch Summary ===
total time:    125.3s
completed:     0
failed:        3

Zero completions. Three failures. Each taking roughly 40 seconds — the time it takes to exhaust GPU memory and panic. The prove=0 ms and queue=0 ms fields are particularly telling: the proofs never reached the proving stage; they failed during setup or memory allocation.

The Context: Phase 9 and the Dual-Worker OOM

To understand why this message was written, we must trace back through the preceding chain of reasoning. The Phase 9 optimization ([chunk 26.0]) targeted two root causes of GPU idle gaps identified in the Phase 8 baseline. Change 1 (Tier 1) pinned host memory with cudaHostRegister and issued async cudaMemcpyAsync transfers on a dedicated stream, moving the 6 GiB of a/b/c polynomial uploads out of the GPU mutex. Change 2 (Tier 3) eliminated per-batch hard sync stalls in the Pippenger MSM by double-buffering host result buffers and deferring the sync() call, allowing GPU compute to overlap with device-to-host transfers.

Initial testing with gpu_workers_per_device=2 (the intended production configuration) revealed immediate OOM failures. The assistant's analysis in [msg 2440] and [msg 2443] traced the problem through multiple layers:

  1. Host registration conflicts: With two workers processing partitions of the same proof, both tried to cudaHostRegister the same host memory pages simultaneously. The second worker received cudaErrorHostMemoryAlreadyRegistered (error 712).
  2. Fallback path failure: When host registration failed, the worker fell back to the original allocation path (gpu.Dmalloc + dev_ptr_t), which should have worked — 12 GiB of allocations on a 16 GiB GPU with 14.3 GiB free.
  3. CUDA memory pool fragmentation: The cudaMallocAsync/cudaFreeAsync memory pools used by the pre-staging path did not release freed memory back to the synchronous cudaMalloc pool used by the fallback path, causing subsequent allocations to fail even after cleanup.
  4. Leaked memory cascading: After the first OOM panic, leaked GPU allocations accumulated, making each subsequent failure worse. The assistant confirmed this with nvidia-smi, showing 10 GiB of leaked memory after the failed runs.## The Hypothesis Being Tested The assistant's reasoning in the preceding messages had converged on a specific theory: the OOM was caused by dual-worker concurrency. The logic chain went as follows: - With gpu_workers_per_device=2, two workers enter generate_groth16_proofs_c simultaneously. - Both attempt cudaHostRegister on the same host memory pages (since they process different partitions of the same proof, sharing the same provers[0].a/b/c pointers). - The second worker gets error 712 and falls back to the non-prestaged path. - Meanwhile, the first worker holds the GPU mutex and has pre-staged 12 GiB of VRAM. - When the first worker finishes and releases the mutex, the second worker acquires it and tries to allocate — but the CUDA memory pools are fragmented, or the first worker's cleanup hasn't fully completed, or the async deallocation thread hasn't run. The natural next step in this diagnostic chain was to eliminate the concurrency variable entirely. If the OOM was truly caused by dual-worker interactions — host registration conflicts, mutex contention, or simultaneous 12 GiB allocations — then running with a single GPU worker should make the problem disappear. The assistant's hypothesis predicted that gpu_workers_per_device=1 would succeed. Message [msg 2444] is the execution of that experiment. The assistant crafted a minimal configuration file (/tmp/cuzk-phase9-gw1.toml) that set gpu_workers_per_device = 1, killed the old daemon, started a fresh one, and launched the benchmark with concurrency=1 and -j 1 (one job at a time). Every variable that could introduce concurrency was stripped away: one proof at a time, one GPU worker, one partition processed sequentially.

The Result That Changed Everything

The benchmark failed. All three proofs failed. The OOM was not caused by dual-worker concurrency.

This result is the article's central dramatic moment because it falsified the prevailing hypothesis. The assistant had spent multiple messages tracing through CUDA memory pool interactions, analyzing cudaHostRegister error codes, computing MSM bucket allocation sizes, and verifying gpu_ptr_t reference counting — all under the assumption that the concurrency between two workers was the root cause. Message [msg 2444] proved that assumption wrong.

The implications are profound for the debugging trajectory:

  1. The OOM is intrinsic to single-worker operation, not a concurrency artifact. Something in the Phase 9 code path itself leaks or misallocates GPU memory, even when only one worker thread exists.
  2. The 40-second failure cadence (each proof taking ~40-43 seconds to fail) is consistent with the time required to allocate and exhaust GPU memory through repeated failed attempts, not a quick error return.
  3. The prove=0 ms field indicates the failure occurs before the proving computation begins — during the setup phase where device buffers are allocated and polynomial data is uploaded.

Assumptions and Their Collapse

The assistant made several assumptions that this result challenged:

Assumption 1: The fallback path should work. The assistant computed that 12 GiB of allocations (4 GiB for d_a + 8 GiB for d_bc) should fit in 14.3 GiB of free VRAM. This assumed that cudaMalloc and cudaMallocAsync draw from the same memory pool — an assumption that CUDA's async memory management explicitly violates. The cudaMallocAsync memory pool is separate from the synchronous cudaMalloc pool, and memory freed via cudaFreeAsync may not be available for a subsequent cudaMalloc call.

Assumption 2: Cleanup is complete. The assistant carefully traced the destructor chains — gpu_ptr_t reference counting, msm_t bucket deallocation, dev_ptr_t destruction — and concluded that worker 0's cleanup should leave the GPU in a clean state. But this assumed that CUDA's internal memory tracking is deterministic and that all cudaFree calls return memory to a pool accessible by subsequent allocations. The fragmentation between async and sync pools violated this assumption.

Assumption 3: The OOM is a concurrency problem. This was the most consequential assumption. The assistant had observed that with gpu_workers_per_device=2, the first partition succeeded and the second failed. The natural inference was that the two workers interfered with each other. Message [msg 2444] disproved this by showing that even one worker fails — meaning the OOM is a property of the Phase 9 code path itself, not of concurrent execution.

Input Knowledge Required

To understand this message, a reader needs knowledge spanning multiple domains:

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. A negative result that constrains the search space: The OOM is not a concurrency bug. This eliminates an entire class of hypotheses (mutex contention, double registration, simultaneous allocation) and forces the investigation toward single-threaded memory management issues.
  2. A precise failure signature: Three proofs, each failing in ~40-43 seconds, with prove=0 ms. This timing signature becomes a diagnostic fingerprint — any fix that changes this pattern can be evaluated against this baseline.
  3. A validated experimental methodology: The assistant demonstrated a clean experimental design — isolate the variable, eliminate confounders, run a controlled test. The configuration change from gpu_workers_per_device=2 to gpu_workers_per_device=1 was the right experiment, even though it disproved the hypothesis.
  4. A forcing function for deeper investigation: By ruling out concurrency, this message forces the next round of investigation to focus on the actual memory allocation patterns within the single-worker path. The user's response in [msg 2445] — "Immediate OOM - perhaps we want to memory-manage the early copies" — shows that the message successfully communicated the problem's nature and prompted a new direction.

The Broader Significance

This message exemplifies a pattern common in systems optimization work: the most valuable experiments are often the ones that disprove your hypothesis. The assistant could have continued debugging the dual-worker interaction for hours, adding more logging, tracing CUDA API calls, and analyzing thread interleaving. Instead, it designed a clean experiment that quickly established the OOM was not a concurrency problem — saving time and focusing effort on the actual root cause.

The message also illustrates the importance of controlling for concurrency when debugging GPU memory issues. GPU programming introduces unique failure modes — memory pool fragmentation, asynchronous allocation semantics, and context-level state corruption — that can mimic concurrency bugs. The assistant's instinct to reduce gpu_workers_per_device to 1 was the correct diagnostic maneuver, even though it didn't produce the expected success.

For the broader Phase 9 optimization effort, this message marks a turning point. The single-worker OOM failure meant that the pre-staging logic itself — the cudaHostRegister calls, the async transfer setup, the memory-aware allocator — had a fundamental flaw that needed to be addressed before any dual-worker throughput gains could be realized. The subsequent conversation (captured in [chunk 26.1]) shows the assistant pivoting to examine the detailed timing logs and eventually achieving successful dual-worker operation, but only after the single-worker path was fixed.

Conclusion

Message [msg 2444] is a masterclass in diagnostic discipline. Faced with a confusing OOM failure that appeared to be caused by dual-worker concurrency, the assistant designed and executed a clean experiment that eliminated concurrency as a variable. The result — three failures in a single-worker configuration — disproved the prevailing hypothesis and redirected the investigation toward the actual memory management issues in the Phase 9 code path.

The message is brief — a single bash command and its output — but it carries immense diagnostic weight. It demonstrates that in complex systems debugging, the ability to formulate testable hypotheses and execute controlled experiments is more valuable than any amount of code tracing or static analysis. Sometimes the most important thing you can learn is that you were wrong about what the problem was.