The Phantom 108 GiB: Tracing a Memory Accounting Bug in a CUDA Pinned Pool

Introduction

In the course of building a high-performance GPU proving system for Filecoin, a team found themselves chasing a particularly insidious bug: a daemon that would crash with a "broken pipe" error during GPU processing, but only on memory-constrained machines. The crash looked like an OOM kill, but the memory budget said there should be plenty of headroom. The discrepancy between what the budget thought was allocated and what the operating system actually had pinned in RAM was the difference between a working system and a silent, unrecoverable crash.

This article examines a single message from the opencode coding session ([msg 3984]) in which the assistant—an AI agent working alongside a human developer—diagnoses the root cause of this crash. The message is a remarkable piece of systems-level debugging: the assistant walks through the entire memory lifecycle of a GPU proof pipeline, identifies a fundamental accounting mismatch in a pinned memory pool, quantifies the hidden memory consumption, and evaluates several possible fixes. The reasoning in this message is not just a diagnosis—it is a deep exploration of how memory budgets, GPU pipelines, and pinned allocations interact in a real-world CUDA application, and how a seemingly reasonable optimization (reusing pinned buffers via a pool) can create a catastrophic blind spot in memory accounting.

The Scene: A Deployment Gone Wrong

The immediate trigger for this message is a failed deployment. Earlier in the session, the assistant had been deploying GPU instances on vast.ai, a marketplace for cloud GPU rentals. The goal was to test a new cgroup-aware memory detection system on a machine with approximately 256 GiB of RAM—a constrained environment that would stress-test the memory budgeting logic. However, the deployment pipeline went awry: instead of the targeted 256 GiB RTX 5090 machine, the system auto-deployed a different offer, landing on an RTX PRO 4000 instance with 342 GiB of RAM (see [msg 3968] and [msg 3970]).

This instance ran its benchmark and promptly died. The benchmark output showed a "broken pipe" error during GPU processing of partition 6, and the daemon was killed. The vast.ai manager recorded the kill reason as "bench_rate 0.0 below min_rate 20.0"—the benchmark produced zero proofs before dying. The assistant's first reaction, captured in the reasoning section of [msg 3984], is to recognize this as an OOM kill: "The 'broken pipe' error strongly suggests the cuzk daemon process was terminated while handling the GPU workload — most likely an OOM kill given the tight memory constraints on that instance."

But the interesting thing is that the memory budget should have been sufficient. The cgroup-aware detection had correctly identified the container limit as 342 GiB, applied a 10 GiB safety margin, and set the budget to 331 GiB. The system reported 266 GiB in use during the crash—well within the 331 GiB budget. So why did it crash?

The Surface-Level Diagnosis: Slow Synthesis and the PI Pacer

The assistant's reasoning begins with a surface-level observation: the synthesis was taking 232 seconds per partition, which is suspiciously slow. On a normal 755 GiB machine, synthesis runs in 20–60 seconds. The RTX PRO 4000 instance has 128 AMD EPYC cores at 50% GPU fraction, suggesting it is a shared host with CPU contention. The assistant connects this to the PI pacer system (a proportional-integral controller for GPU dispatch, developed earlier in the session—see [msg 3985] onward for the pacer's implementation):

"Since synthesis is 4x slower here, the PI pacer would try to dispatch 4x more partitions concurrently to keep the GPU fed. With bench_concurrency=4 and max_parallel_synthesis=18, that could mean 40 partitions in flight, each needing 14 GiB of working memory."

This is a crucial insight. The PI pacer is designed to maintain a target GPU queue depth by adjusting the dispatch rate based on feedback. When synthesis is slow, the pacer compensates by dispatching more partitions concurrently, trying to keep the GPU busy. But on a memory-constrained machine, this creates a feedback loop: slow synthesis → more concurrent partitions → higher memory pressure → even slower synthesis (due to swapping/thrashing) → even more partitions dispatched. The 331 GiB budget would only support around 18 concurrent partitions at 14 GiB each, but the pacer might be trying to dispatch far more.

However, the assistant quickly realizes that this is only part of the story. Even with 40 partitions in flight, the budget should prevent allocations beyond 331 GiB. The fact that the system crashed despite the budget suggests a deeper problem: the budget is not accounting for all the memory that is actually allocated.

The Core Discovery: PinnedPool Accounting Mismatch

This is where the assistant's reasoning takes a sharp turn from surface-level symptoms to deep architectural analysis. The key insight is stated directly:

"The pinned pool is the culprit—it's allocating outside the budget, causing OOM on constrained machines."

To understand this, we need to understand the memory lifecycle of a GPU proof in this system. The assistant walks through it step by step:

  1. Budget acquisition: The system acquires 14 GiB from the MemoryBudget for a partition's working set.
  2. Synthesis: The partition runs CPU-side synthesis. The a/b/c buffers (approximately 13 GiB) are allocated as pinned memory via cudaHostAlloc. If a pinned buffer is available in the PinnedPool (from a previous partition that finished GPU work), it is reused. If not, a new cudaHostAlloc call is made.
  3. GPU queue: After synthesis completes, the partition is queued for GPU processing.
  4. GPU prove_start: The GPU picks up the partition. The a/b/c buffers are released from the partition's ownership—but crucially, they are returned to the PinnedPool for reuse, not freed via cudaFreeHost.
  5. Budget release: The budget reservation for the a/b/c buffers (13 GiB) is released. The problem is in step 5. When the budget releases the 13 GiB reservation, it marks that memory as available for new allocations. But the actual RSS (Resident Set Size) of the process does not decrease, because the pinned buffers are still allocated in the PinnedPool. The assistant quantifies this:
"Budget-tracked RSS: working_set minus a/b/c = ~1 GiB per partition in GPU. Actual RSS: pinned buffer still allocated = ~12.5 GiB per partition in GPU."

This is the accounting mismatch. The budget thinks it has 13 GiB of free space per partition that has moved to the GPU queue. In reality, that 13 GiB is still pinned in RAM, sitting in the pool waiting for reuse. The budget can therefore allocate more partitions than the system can actually hold, because it is counting the same memory twice.

Quantifying the Hidden Memory

The assistant then does the math to quantify the scale of the problem:

"With 8 partitions in the GPU queue, the budget thinks it's only using 8 GiB, but the pinned pool is actually holding 108 GiB of untracked memory. That's a massive discrepancy—the budget freed 13 GiB per partition, but those buffers never actually left the system."

Let me verify this: 8 partitions × 12.5 GiB per partition = 100 GiB (the assistant says 108 GiB, which may account for some overhead or a slightly different per-partition pinned size). On a machine with 342 GiB total RAM, this 100+ GiB of hidden memory is catastrophic.

The assistant then builds a full picture of the memory layout on the crashed machine:

"Looking at the full machine with 342 GiB total, I've got 44 GiB pinned and budget-tracked for SRS, 26 GiB heap for PCE, and this hidden 108 GiB in the pinned pool that the budget doesn't see."

Adding these up: 44 GiB (SRS) + 26 GiB (PCE) + 108 GiB (hidden pinned pool) = 178 GiB of accounted-for memory. But the budget only "sees" 44 + 26 + 8 = 78 GiB (the 8 GiB being the ~1 GiB per partition that remains budget-tracked after the a/b/c release). The remaining 100 GiB is invisible to the budget. When the budget tries to allocate more partitions, it thinks it has 331 - 78 = 253 GiB of headroom, when in reality it has 331 - 178 = 153 GiB. And that's before accounting for kernel overhead, glibc arenas, page tables, GPU driver allocations, and the transient SRS loading spike.

The assistant's conclusion is stark: "On a 342 GiB cgroup limit, that's a guaranteed OOM."

Tracing the History: How Did We Get Here?

The assistant also reflects on the history of this code. Earlier in the session, the PinnedPool had budget integration that was removed because it caused double-counting:

"Looking back at the original fix, the problem was that PinnedPool was double-counting—it tried to acquire from the budget for a/b/c buffers, but those were already reserved by the partition's working memory. So the budget integration was removed entirely."

This is a classic software engineering dilemma. The first attempt at budget integration caused double-counting (the partition reserved the memory, and then the pool tried to reserve it again). The fix was to remove the pool's budget integration entirely. But this created a new problem: the pool's allocations became completely invisible to the budget. The pendulum swung from double-counting to zero-counting, and neither was correct.

The correct accounting, as the assistant realizes, is more nuanced:

"The right approach would be to count pooled buffers against the budget when they're idle, then let the partition's existing a/b/c reservation cover them when they're checked out, and return them to the pool (and budget) when the partition releases them."

In other words, the budget should track the memory throughout its lifecycle:

Evaluating the Fixes

The assistant considers several possible fixes, ranging from simple to comprehensive:

1. Cap the PinnedPool Size

"The simplest solution is just to cap the pinned pool size based on available memory or the maximum number of buffers that could ever be in flight at once—which is bounded by the GPU queue depth plus synthesis workers."

This is a pragmatic fix. If the pool can hold at most N buffers, and N is chosen based on the maximum number of partitions that could be in flight simultaneously, then the pool's growth is bounded. When the pool is full, new allocations fall back to regular heap memory (non-pinned), which avoids exhausting the budget. However, this loses the performance benefit of pinned memory for those fallback allocations.

2. Don't Release Budget on Pool Return

"Or even simpler: don't release a/b/c from the budget when they go into the pool, only release them when they're actually freed with cudaFreeHost, so the budget correctly reflects that the memory is still allocated."

This is a minimal change that fixes the accounting without restructuring the pool. The budget reservation for a/b/c would persist until the buffer is actually freed, not when it returns to the pool. This means the budget would correctly reflect the true RSS. However, it would also mean that the budget appears more constrained than it needs to be during steady-state operation (when buffers are cycling through the pool), potentially underutilizing memory.

3. Budget-Aware Pool Cache

"The real fix would be treating the pinned pool as a budget-aware cache that tracks memory throughout its lifecycle—when buffers are checked out, they're already accounted for in the synthesis budget; when they return to the pool after GPU work, the budget reservation stays; and only when the pool needs to shrink under budget pressure do we actually free buffers and release their budget."

This is the most comprehensive solution, treating the pool as a proper cache with eviction under pressure. It's similar to how the SRS and PCE caches handle permanent allocations. But the assistant acknowledges that "it's a more involved change."

Assumptions and Potential Blind Spots

The assistant's reasoning is thorough, but there are several assumptions worth examining:

Assumption 1: The crash was an OOM kill. The assistant assumes that the "broken pipe" error during GPU processing was caused by the OOM killer terminating the cuzk daemon. This is plausible but not proven. A broken pipe could also result from a GPU driver crash, a CUDA API failure, or a network issue between the benchmark client and the daemon. The assistant does not check the system logs (e.g., dmesg or journalctl) for OOM killer evidence. However, the circumstantial evidence—266 GiB used out of 342 GiB, the known accounting mismatch, and the fact that larger machines with more headroom do not crash—makes OOM the most likely cause.

Assumption 2: The pinned pool grows unbounded. The assistant says "it grows unbounded and sits outside the memory budget." But is it truly unbounded? The pool reuses buffers; it doesn't allocate new ones for every partition. The pool size is bounded by the maximum number of buffers ever created, which is itself bounded by the maximum number of concurrent partitions. If the system reaches steady state where the number of partitions in flight equals the pool size, no new allocations occur. The problem is that the first wave of partitions creates N buffers, and those N buffers persist even after the partitions finish. The pool size is bounded by N, but N can be large if the PI pacer dispatches many partitions before the GPU processes them.

Assumption 3: The 10 GiB safety margin is insufficient. The assistant later discovers (in subsequent messages) that the safety margin needs to account for kernel overhead (glibc arenas, page tables, GPU driver allocations) and the transient SRS loading spike. The 10 GiB margin was chosen arbitrarily and proved insufficient on constrained machines. This is not a flaw in the reasoning but an empirical discovery that follows from the analysis.

Assumption 4: The PI pacer is making things worse. The assistant suggests that the PI pacer's response to slow synthesis (dispatching more partitions) creates a positive feedback loop with memory pressure. This is a plausible hypothesis, but the assistant does not have data on the actual dispatch rate or GPU queue depth at the time of the crash. The pacer might have been operating within normal parameters, and the crash might have been caused purely by the pinned pool accounting mismatch without any pacer amplification.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

  1. CUDA pinned memory: cudaHostAlloc allocates host memory that is page-locked (pinned), allowing GPU DMA transfers without copying through pageable buffers. Pinned memory is not swappable and counts against the process's RSS.
  2. The MemoryBudget system: A software component that tracks and limits memory allocations to prevent exceeding a configured budget. It uses reservations (acquire/release) to track how much memory is in use.
  3. The PinnedPool: A reuse cache for pinned memory buffers. Instead of freeing pinned memory after GPU processing (which is expensive), buffers are returned to the pool for reuse by future partitions.
  4. The proof pipeline: The sequence of stages a proof goes through: synthesis (CPU), GPU queue, GPU processing, completion. Each stage has different memory requirements and different budget tracking.
  5. cgroup memory limits: Linux cgroups v2 (memory.max) and v1 (memory.limit_in_bytes) constrain container memory. The system must read these limits rather than /proc/meminfo to correctly determine available memory inside a container.
  6. The PI pacer: A proportional-integral controller that adjusts GPU dispatch rate to maintain a target queue depth, compensating for varying synthesis times.
  7. The vast.ai environment: A GPU rental marketplace where instances have varying RAM, GPU models, and cgroup constraints. Instances can be "fractional" (a slice of a larger machine with a cgroup limit).

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. A precise diagnosis of the OOM crash: The root cause is the accounting mismatch between the MemoryBudget and the PinnedPool. The budget releases reservations when buffers return to the pool, but the actual RSS does not decrease.
  2. A quantified model of hidden memory: The assistant calculates that each partition in the GPU queue contributes ~12.5 GiB of untracked pinned memory, and with 8 partitions this creates 100+ GiB of invisible allocations.
  3. A taxonomy of possible fixes: From simple caps to comprehensive budget-aware caching, the assistant evaluates the trade-offs of each approach.
  4. A historical understanding: The assistant traces how the code arrived at this state—the earlier double-counting bug led to the removal of budget integration, which created the current zero-counting bug.
  5. A connection between the PI pacer and memory pressure: The assistant identifies a potential feedback loop where slow synthesis causes the pacer to dispatch more partitions, increasing memory pressure and further slowing synthesis.
  6. A decision framework: The assistant weighs the complexity of each fix against its effectiveness, ultimately leaning toward the simpler approach (capping the pool or deferring budget release) for immediate deployment, while acknowledging that the comprehensive fix (budget-aware cache) is the correct long-term solution.

The Thinking Process: A Window into Systems Debugging

The reasoning section of [msg 3984] is remarkable for its structure. It reads like a systems debugging session transcribed in real time, with all the false starts, corrections, and deepening insights preserved.

The assistant begins with the surface-level observation (slow synthesis, broken pipe) and immediately jumps to the OOM hypothesis. Then it backtracks and re-examines the pipeline code, walking through each step of the memory lifecycle. This is where the key insight emerges: "When synthesis finishes and the a/b/c move into pinned memory, the partition releases its 13 GiB budget reservation, but the pinned buffer itself never gets freed—it just returns to the pool for reuse."

Notice the "Actually, wait" moment in the reasoning: "Actually, wait. Let me re-read the code." This is the assistant catching itself, realizing that the initial hypothesis (slow synthesis + PI pacer) is insufficient, and diving deeper into the actual memory accounting. This self-correction is a hallmark of good debugging—the willingness to question your own assumptions and re-examine the evidence.

The assistant then quantifies the problem, building a concrete model of the memory layout. It calculates the hidden 108 GiB, adds up the SRS and PCE allocations, and arrives at the conclusion that the system is operating far closer to the limit than the budget indicates.

Finally, the assistant considers solutions. It starts with the most complex (budget-aware cache), then simplifies: "Actually, the simplest solution is just to cap the pinned pool size." This progression from complex to simple is another hallmark of good engineering—start with the ideal solution, then simplify until you have something you can implement and deploy quickly.

Conclusion

Message [msg 3984] captures a pivotal moment in the debugging of a production GPU proving system. The assistant takes a cryptic "broken pipe" error and traces it to a fundamental accounting mismatch in a pinned memory pool, quantifying the hidden memory consumption and evaluating multiple fix strategies.

The deeper lesson here is about the dangers of invisible state in memory management. The PinnedPool was designed as a performance optimization—reusing pinned buffers avoids the expensive cudaFreeHost/cudaHostAlloc cycle. But this optimization created a blind spot: memory that was "freed" from the budget's perspective was still physically allocated in RAM. The budget and reality diverged, and on a memory-constrained machine, that divergence was fatal.

This is a cautionary tale for any system that manages memory through budgets, pools, or caches. When you introduce a reuse layer between allocation and freeing, you must ensure that your accounting tracks the physical reality, not just the logical lifecycle. The budget must see what the kernel sees.

The assistant's analysis in this message set the stage for the fixes that follow in subsequent messages: the memprobe utility to empirically measure hidden overhead, the OOM recovery loop in the benchmark script, and the eventual restructuring of the PinnedPool to be budget-aware. But the foundational insight—that the pinned pool was creating phantom memory that the budget couldn't see—was born in this single message, in a moment of careful reasoning about a broken pipe on a machine far away.