The Pinned Pool Paradox: Tracing an OOM Crash Through Memory Accounting Mismatches

Introduction

In the high-stakes world of GPU-accelerated proving for Filecoin, memory management is not merely an optimization—it is a matter of survival. When a system operates within a cgroup memory limit of 342 GiB, every gigabyte counts. Message 4106 in this opencode session captures a pivotal moment of architectural diagnosis: the assistant has just witnessed a container-level OOM kill during a benchmark run, and is now working to understand why the memory budget system failed despite apparently correct configuration. This message is a turning point—the moment when a superficial fix (capping the pinned pool) is rejected in favor of a deep understanding of the memory accounting mismatch that caused the crash.

The Message

The assistant writes:

Now I need to understand the relationship better. Let me look at how many buffers the pool actually grows to:

>

The pool grows by 3 buffers (a, b, c) per concurrent partition synthesis. With max_parallel_synthesis=18, that's up to 18 × 3 = 54 buffers × 3.88 GiB = 210 GiB of pinned memory. But the budget already accounts for 14 GiB per partition of working memory. The pinned a/b/c buffers (11.6 GiB) are a subset of that 14 GiB.

>

The issue: when partitions complete, their budget reservations are released (14 GiB freed in budget), and their pinned buffers return to the pool (11.6 GiB stays allocated). So the budget thinks it has 14 GiB free, but actually only 14 - 11.6 = 2.4 GiB of real memory was freed.

>

The pool naturally caps at peak concurrent partitions. On the 342 GiB machine, during Phase 1 warmup, many partitions run concurrently (the pacer dispatches aggressively to fill the GPU queue). The pool grew, and then when Phase 2 starts, those buffers are reused. But the total pinned pool memory (which is real physical memory) is NOT freed between phases.

>

The best approach: Add a max pool size parameter and free excess buffers on checkin. The max should be based on max_gpu_queue_depth + gpu_workers_per_device (since that's the max number of partitions that need pinned buffers simultaneously — those in the GPU queue + those on GPU). Partitions still being synthesized haven't finished yet, so their buffers are checked out but will be checked in after GPU proving.

>

Actually wait — buffers are checked out during synthesis and stay checked out until AFTER GPU proving (released by release_abc). So the peak is: - Partitions being synthesized: have buffers checked out - Partitions in GPU queue (synth_done, waiting for GPU): have buffers checked out - Partitions on GPU (prove_start): have buffers checked out, released during prove_start

>

So peak = max_parallel_synthesis (all synthesizing) is the worst case. But we already cap concurrent synthesis with this config.

>

The key insight: the pool's total allocation (including checked-out buffers) should be tracked and capped. When a buffer is checked in and the pool is over its cap, the buffer should be freed immediately rather than kept.

>

Let me implement a max_pool_buffers cap on PinnedPool:

This message is the culmination of a long debugging chain that began with a "transport error" / "broken pipe" during a benchmark run on an RTX 5090 vast.ai instance ([msg 4099]). The assistant had already fixed a bash script bug that was masking the real issue ([msg 4094]), but the underlying memory pressure remained. The instance became unreachable, confirming a cgroup-level OOM kill ([msg 4103]).

Why This Message Was Written: The Reasoning and Motivation

The assistant's motivation in this message is diagnostic clarity leading to a principled fix. The chain of reasoning is worth unpacking in detail.

Step 1: Quantifying the problem. The assistant begins by calculating the worst-case growth of the pinned pool. With max_parallel_synthesis=18, the pool can theoretically hold 54 buffers (3 per partition for the a, b, and c vectors), totaling approximately 210 GiB of pinned CUDA memory. This is a staggering amount—over 60% of the total cgroup limit of 342 GiB. The assistant is establishing the scale of the problem before proposing a solution.

Step 2: Identifying the accounting mismatch. The core insight emerges: the memory budget system reserves approximately 14 GiB per partition for working memory, and the pinned a/b/c buffers (11.6 GiB) are a subset of that reservation. However, when a partition completes, the budget reservation is released (the budget thinks 14 GiB is now free), but the pinned buffers persist in the pool. The actual physical memory freed is only 14 - 11.6 = 2.4 GiB. The budget therefore systematically overestimates available memory by the size of the pinned pool's retained buffers.

This is the root cause of the OOM crash. The budget believes it has headroom, but the pinned pool holds onto physical memory invisibly, eventually pushing total usage over the cgroup limit.

Step 3: Rejecting the naive cap. The assistant initially considers capping the pool based on max_gpu_queue_depth + gpu_workers_per_device, reasoning that only partitions in the GPU queue or actively on GPU need pinned buffers simultaneously. But then the assistant corrects itself: "Actually wait — buffers are checked out during synthesis and stay checked out until AFTER GPU proving." This self-correction is crucial. The peak number of concurrently checked-out buffers is actually max_parallel_synthesis (all partitions being synthesized), because buffers are allocated at synthesis time and only released after GPU proving. A cap based on GPU queue depth would be too aggressive and could starve the pipeline.

Step 4: Arriving at the principled solution. The assistant settles on tracking the pool's total allocation (including checked-out buffers) and freeing excess buffers on checkin when the pool exceeds a configurable maximum. This is a clean solution: it bounds the pool's physical memory footprint without introducing arbitrary caps on concurrent synthesis, and it leverages the existing checkin path as a natural point for memory reclamation.

How Decisions Were Made

The decision-making in this message is iterative and self-correcting. The assistant moves through several candidate approaches:

  1. Calculate worst-case pool size — establishes the upper bound (210 GiB).
  2. Identify the accounting mismatch — the budget releases reservations but pinned memory persists.
  3. Propose a cap based on GPU queue depth — initially seems reasonable.
  4. Self-correct — realizes buffers are held longer than expected (from synthesis through GPU proving).
  5. Refine to max_parallel_synthesis as the true peak — but notes this is already config-capped.
  6. Final design: track and cap total pool allocation, free excess on checkin — a principled, bounded solution. The decision to implement a max_pool_buffers parameter rather than a percentage-based cap or a hard-coded limit reflects a preference for configurability. The assistant does not hardcode a value; instead, it creates a mechanism that can be tuned per deployment.

Assumptions Made

Several assumptions underpin the assistant's reasoning:

Assumption 1: The pinned pool is the dominant unaccounted memory consumer. The assistant assumes that the pinned pool's 210 GiB worst-case allocation is the primary reason the budget fails. This is reasonable given the evidence—the daemon log showed the pool being drained to zero free buffers at the moment of the crash ([msg 4101]). However, other factors like kernel/driver overhead (~6 GiB) and fragmentation are acknowledged but not deeply analyzed.

Assumption 2: The per-partition budget reservation of 14 GiB accurately covers the a/b/c vectors. The assistant states that "the pinned a/b/c buffers (11.6 GiB) are a subset of that 14 GiB." This assumes the budget's 14 GiB per-partition reservation was designed to include the a/b/c allocations, and that 11.6 GiB of pinned memory is indeed within that 14 GiB envelope. If the budget reservation is smaller or structured differently, the math changes.

Assumption 3: Freeing excess buffers on checkin is safe and sufficient. The assistant assumes that freeing a pinned buffer with cudaFreeHost when it is checked in (and the pool is over cap) will not cause performance regressions. This is plausible but unverified—if the pool frequently allocates and frees buffers, the overhead of repeated cudaHostAlloc calls could impact latency.

Assumption 4: The pool's allocation is bounded by max_parallel_synthesis. The assistant correctly notes that buffers are checked out during synthesis and held until GPU proving completes. This means the peak pool size is max_parallel_synthesis × 3 buffers, not max_gpu_queue_depth × 3 buffers. This self-correction prevented a buggy implementation.

Mistakes or Incorrect Assumptions

The most notable mistake in this message is the initial proposal to cap the pool based on max_gpu_queue_depth + gpu_workers_per_device. The assistant catches this error within the same message, but it's worth examining why the first instinct was wrong.

The error stems from conflating when buffers are needed with when buffers are held. The GPU queue depth determines how many partitions are waiting for GPU processing, but buffers are allocated before the queue—during synthesis—and held through GPU proving. The correct bound is therefore the number of concurrently synthesizing partitions, not the queue depth.

This mistake is understandable given the complexity of the pipeline state machine. The assistant's self-correction demonstrates the value of thinking through the full lifecycle of a buffer: allocation at synthesis start, checkout during synthesis, hold through GPU queue, release during GPU prove, checkin after GPU prove.

Another subtle issue: the assistant says "the pool's total allocation (including checked-out buffers) should be tracked and capped." But checked-out buffers are, by definition, in use—they cannot be freed. The cap can only be enforced on checked-in buffers (those available for reuse). The assistant's design of "free excess on checkin" correctly targets the reusable pool, not the checked-out buffers. However, the phrase "total allocation (including checked-out buffers)" is slightly misleading—the cap applies to the pool's free buffer count, not the total allocated buffers including those checked out.

Input Knowledge Required

To understand this message, a reader needs knowledge in several domains:

CUDA memory management: Understanding that cudaHostAlloc allocates pinned (page-locked) host memory, which enables faster GPU transfers but consumes physical RAM that cannot be swapped. The distinction between regular heap memory and pinned memory is central to the problem.

Memory budgeting in proving systems: The concept of a "memory budget" that reserves memory for different phases of computation (permanent allocations like SRS parameters, working memory for synthesis, GPU memory). The budget system uses RAII-style MemoryReservation guards that are released when a partition completes.

Pipeline architecture: The proving pipeline involves multiple stages—synthesis (CPU), GPU queue (waiting for GPU), GPU proving (GPU compute), and result collection. Buffers flow through these stages, and their lifetimes span multiple stages.

Cgroup memory limits: Linux cgroups enforce memory limits at the container level. When total memory (including pinned allocations, heap, GPU memory, kernel overhead) exceeds the limit, the OOM killer terminates the entire container. This is distinct from a process-level OOM.

The specific codebase: Familiarity with PinnedPool, MemoryBudget, MemoryReservation, and the partition synthesis/proving lifecycle in the cuzk proving engine is necessary to fully grasp the implementation details.

Output Knowledge Created

This message generates several important outputs:

1. A root cause diagnosis: The OOM crash is traced to a fundamental accounting mismatch—the memory budget releases per-partition reservations when partitions complete, but the pinned pool retains the physical memory invisibly. This is the key insight that guides the fix.

2. A quantitative model of pool growth: The assistant establishes that the pool can grow to 210 GiB (54 buffers × 3.88 GiB) under worst-case conditions, and that the budget's per-partition reservation of 14 GiB only partially covers this (11.6 GiB of the 14 GiB is pinned, leaving only 2.4 GiB of truly freed memory per partition).

3. A design for the fix: The assistant decides to add a max_pool_buffers parameter to PinnedPool and free excess buffers on checkin. This is a principled solution that bounds the pool's physical footprint without arbitrary caps on synthesis concurrency.

4. A corrected understanding of buffer lifetimes: The assistant corrects its own initial mistake about when buffers are held, establishing that the peak pool size is bounded by max_parallel_synthesis (not GPU queue depth).

5. An edit to the source code: The message concludes with an edit to pinned_pool.rs, implementing the max_pool_buffers cap.

The Thinking Process

The thinking process visible in this message is remarkably transparent and iterative. The assistant does not present a polished final analysis; instead, it walks through its reasoning step by step, including the false starts and self-corrections.

The process begins with a quantitative question: "how many buffers the pool actually grows to." This is a concrete, measurable starting point. The assistant calculates the worst case (54 buffers, 210 GiB) and immediately connects this to the budget system.

The key insight—that budget reservations are released while pinned memory persists—emerges from juxtaposing two facts: (a) the budget accounts for 14 GiB per partition, and (b) the pinned pool holds 11.6 GiB per partition. The assistant realizes that the delta (2.4 GiB) is the only memory truly freed when a partition completes.

The self-correction is particularly revealing. The assistant first proposes a cap based on GPU queue depth, then pauses with "Actually wait—" and re-examines the buffer lifecycle. This moment of reflection prevents a flawed implementation. The assistant traces the full path: buffers are checked out during synthesis, held through the GPU queue, and only released during GPU proving. Therefore, the peak is max_parallel_synthesis, not queue depth.

The final decision—to track and cap total pool allocation, freeing excess on checkin—is a natural consequence of this analysis. The assistant has identified that the pool's unchecked growth is the problem, and the checkin path is the natural point to enforce a limit.

Conclusion

Message 4106 represents a pivotal moment in the debugging journey. The assistant moves from observing a symptom (OOM crash) to understanding the root cause (accounting mismatch between budget reservations and pinned pool persistence). The message is a model of systematic debugging: quantify the problem, identify the mismatch, propose a solution, self-correct, and arrive at a principled design.

The fix implemented in this message—a configurable max_pool_buffers cap with excess buffer freeing on checkin—is not a silver bullet. It addresses the immediate OOM issue but raises questions about performance under tight memory budgets. If the pool frequently allocates and frees buffers, the overhead could impact throughput. The true test will come in the next benchmark run.

But as a piece of engineering analysis, this message is exemplary. It demonstrates the value of understanding the full lifecycle of resources, the danger of invisible state (pinned memory not tracked in the budget), and the importance of self-correction in the reasoning process. The assistant's willingness to question its own initial proposal ("Actually wait—") is perhaps the most important lesson for any engineer debugging a complex system.