Diagnosing a Silent OOM: How Untracked Pinned Memory Crashed the CuZK Prover on an RTX 5090

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proving, memory is the ultimate constraint. Every byte counts, and when a system operates within a cgroup memory limit, the margin between success and catastrophic failure can be razor-thin. This article examines a pivotal diagnostic message ([msg 4103]) in an opencode coding session where an AI assistant, after successfully fixing a bash script bug and watching Phase 1 of a benchmark complete flawlessly, discovered that the RTX 5090 instance had gone completely dark—silently killed by the cgroup OOM killer. The message represents a critical inflection point: the moment when a superficial bug fix gave way to a deep architectural understanding of why the system was crashing under memory pressure.

The Scene: A Benchmark on the Edge

The context leading up to this message is essential for understanding its significance. The assistant had been working on the CuZK proving engine—a high-performance GPU-accelerated system for generating Filecoin Proof-of-Replication (PoRep) proofs. The system uses a sophisticated pipeline architecture where proof synthesis (CPU-bound) feeds partitions to GPU workers for proving. A key component is the "pinned pool," a pre-allocated set of CUDA pinned memory buffers used for efficient GPU data transfer.

The assistant had just resolved a frustrating bash script bug in benchmark.sh where a subtle interaction between set -euo pipefail and an if ! cmd | tee pipeline pattern caused exit codes to be captured incorrectly, masking OOM crashes and preventing the timed benchmark phase from running (<msg id=4089-4098>). After deploying the fix and initiating a benchmark run with --skip-warmup, Phase 1 (5 warmup proofs) completed successfully in 518 seconds. The system was humming: memory usage peaked at 341 GiB out of a 342 GiB cgroup limit—already at 99.7% capacity, but holding.

Then Phase 2 began. The first proof submitted, and immediately the daemon log showed the pinned pool being drained—all 24 buffers checked out across 8 partitions. Then silence. The benchmark client reported a "transport error" and "broken pipe." The daemon was gone. And then, as the assistant discovered in the messages immediately preceding [msg 4103], the instance itself became unreachable: ssh: connect to host ssh6.vast.ai port 17008: Connection refused.

The Diagnostic Message: Piecing Together the Evidence

Message [msg 4103] is the assistant's synthesis of this crisis. It begins with a stark observation:

"The instance is completely unreachable. It was likely OOM-killed at the container level (the cgroup OOM killer kills the entire container when memory limit is hit)."

This opening line is significant because it demonstrates the assistant's understanding of the Linux cgroup memory architecture. Unlike a process-level OOM kill (where the kernel's OOM killer selects a victim process), the cgroup OOM killer operates at the container boundary: when the sum of all processes in the cgroup exceeds the memory limit, the entire cgroup is killed. This is why the SSH connection dropped—the container itself ceased to exist.

The assistant then reconstructs the memory arithmetic that led to the crash:

"342 GiB cgroup limit. Budget set to 331 GiB. During Phase 1, memory peaked at 341 GiB (already at 100%). Phase 2 started, more synthesis tasks launched, pinned pool was being drained (8 partitions × 3 buffers checked out = 24 buffers, ~93 GiB pinned). The combination of SRS (44 GiB pinned) + PCE (26 GiB heap) + pinned pool + synthesis working memory exceeded the cgroup limit."

This calculation reveals the fundamental flaw in the memory budget system. The MemoryBudget component tracks heap allocations and per-partition working memory, but it has a critical blind spot: the pinned pool's cudaHostAlloc buffers are invisible to it. When a partition completes, its per-partition budget reservation is released, but the pinned pool retains the physical memory. This creates a systematic over-commitment where the budget thinks memory is available when it is not.

The root cause statement crystallizes the insight:

"331 GiB budget is too high for a 342 GiB cgroup machine. The kernel/driver overhead + pinned pool (which is NOT tracked in the budget) pushes total memory usage over the limit."

The Thinking Process: From Symptom to Root Cause

What makes this message particularly valuable is the visible reasoning process. The assistant did not jump to conclusions. Instead, it methodically assembled evidence from multiple sources over several preceding messages:

  1. Memory readings from Phase 1: The cgroup usage at 341 GiB out of 342 GiB was a warning sign that the budget was too aggressive. With only 1 GiB of headroom, any additional allocation would trigger the OOM killer.
  2. The daemon log: The last log entries showed the pinned pool being drained—free_remaining=0—indicating that all pre-allocated pinned buffers were in use. This was the moment of maximum memory pressure.
  3. The architectural knowledge: The assistant understood that the pinned pool uses cudaHostAlloc for GPU DMA buffers, which are allocated from the CUDA driver's memory pool. This memory is charged against the process's RSS and therefore counts toward the cgroup limit, but it is not tracked by the application-level MemoryBudget.
  4. The timing: The crash occurred at the transition from Phase 1 to Phase 2, when the pipeline was being re-primed. Phase 1 had 5 proofs at concurrency 4; Phase 2 launched 10 proofs at the same concurrency, but the pipeline state was different—the pinned pool was already fully checked out from the tail end of Phase 1, and new synthesis tasks were competing for the same limited pool. The assistant's conclusion—that the budget must be reduced or the pinned pool must be properly integrated—is not just a tactical fix but a strategic realization. The system cannot safely operate at 331 GiB on a 342 GiB machine because the untracked pinned pool creates an invisible memory tax that can spike at any moment.

Assumptions and Their Validity

The message rests on several key assumptions, most of which are well-founded:

Assumption 1: The cgroup OOM killer was responsible. Given the complete disappearance of the container (not just the daemon process), this is the most parsimonious explanation. The alternative—a network-level failure or SSH tunnel collapse—is less likely given that the crash coincided precisely with peak memory pressure.

Assumption 2: The pinned pool memory is not tracked in the budget. This is confirmed by the architectural analysis in the following chunk ([chunk 30.1]), where the assistant reads the full memory.rs file and confirms that MemoryBudget has no visibility into cudaHostAlloc allocations. The pinned pool operates independently, allocating and freeing buffers without consulting the budget.

Assumption 3: The calculation of ~93 GiB for 24 pinned buffers. This is based on the partition buffer size for 32 GiB proofs. Each partition requires three pinned buffers (input, output, and scratch), and with 8 partitions active simultaneously, 24 buffers × ~3.9 GiB each ≈ 93 GiB. This is a reasonable estimate.

Assumption 4: The budget of 331 GiB is too aggressive. This is the core insight and is demonstrably correct: the system crashed because the combination of tracked and untracked memory exceeded the cgroup limit. A lower budget would have provided more headroom, though at the cost of reduced parallelism.

One potential limitation of the analysis is that it does not account for kernel slab allocations, driver overhead, or other system-level memory consumers. The cgroup limit of 342 GiB includes all of these, not just the application's allocations. The assistant's calculation of SRS (44 GiB) + PCE (26 GiB) + pinned pool (~93 GiB) + synthesis working memory may undercount the true memory footprint, but the conclusion remains valid: the budget is too high.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

  1. Linux cgroup memory management: The distinction between process-level OOM kills and cgroup-level container kills, and how memory.usage_in_bytes and memory.limit_in_bytes work.
  2. CUDA memory architecture: The difference between pageable memory and pinned memory (cudaHostAlloc), and why pinned memory is necessary for GPU DMA transfers but consumes host-side RAM that counts toward RSS.
  3. The CuZK proving pipeline: How proof synthesis (CPU-bound partition generation) feeds GPU workers, the role of the pinned pool in buffer management, and the distinction between SRS (Structure Reference String—permanent pinned memory), PCE (Pre-Compiled Constraint Evaluator—heap memory), and per-partition working memory.
  4. The MemoryBudget system: How the application tracks heap allocations, the RAII MemoryReservation guards, and the permanent vs. working memory split.
  5. The vast.ai environment: Docker containers with cgroup memory limits, SSH tunneling for remote access, and the specific hardware configuration of the RTX 5090 instance (342 GiB cgroup limit).

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. Root cause identification: The crash is definitively diagnosed as a cgroup OOM kill caused by untracked pinned pool memory exceeding the budget.
  2. Architectural blind spot: The MemoryBudget system has a fundamental flaw—it cannot see pinned pool allocations. This is not a bug but a design limitation that must be addressed.
  3. Quantitative bounds: The system cannot safely operate at 331 GiB on a 342 GiB machine. The required headroom depends on the pinned pool utilization, which varies with the number of concurrent partitions.
  4. Actionable direction: The fix is not to reduce the budget arbitrarily (which would harm performance) but to integrate the pinned pool with the budget so that the system dynamically adapts to available memory.
  5. Todo update: The assistant updates its task list to reflect the new understanding, shifting from "fix benchmark.sh syntax" to "diagnose OOM: pinned pool memory not tracked in budget."

The Broader Significance

This message represents a classic systems debugging pattern: a superficial symptom (the benchmark crashing) leads to a superficial fix (the bash script bug), which then reveals a deeper architectural issue (the memory accounting blind spot). The assistant's willingness to abandon the quick fix of capping the pinned pool at 40% of the budget—which the user correctly rejected as "unprincipled" and "catastrophically harmful to performance" ([chunk 30.1])—demonstrates the discipline required for proper systems engineering.

The message also highlights the fragility of operating at the edge of memory limits. The system was running at 99.7% of the cgroup limit during Phase 1, leaving virtually no headroom for transient spikes. In production systems, such tight margins are a recipe for disaster. The proper solution—integrating the pinned pool with the budget so that the system can dynamically throttle parallelism based on actual available memory—is a significant engineering undertaking, but it is the only way to achieve both safety and performance.

Conclusion

Message [msg 4103] is a masterclass in diagnostic reasoning under pressure. Faced with a completely unreachable instance and a failed benchmark, the assistant methodically assembled evidence from memory readings, daemon logs, and architectural knowledge to identify the root cause: the pinned pool's invisible memory footprint was pushing the system over the cgroup limit. The message transforms a frustrating crash into a clear engineering challenge, setting the stage for the architectural redesign that follows in the subsequent chunk. It is a reminder that in complex systems, the most important debugging tool is not a debugger but a deep understanding of how every component consumes resources—and what happens when those resources run out.