The Invisible Memory: Diagnosing a Budget Accounting Mismatch in CUDA Pinned Pool Allocation

Introduction

In the high-stakes world of GPU-accelerated proof generation for Filecoin, memory management is not merely a matter of efficiency—it is a matter of survival. When a production daemon running on an RTX 5090 instance with 342 GiB of cgroup memory was killed by the OOM killer during a benchmark run, the crash appeared at first to be a simple case of insufficient headroom. But as the assistant dug deeper into the logs, a far more subtle and architectural problem emerged: the CUDA pinned memory pool was growing invisibly outside the memory budget system, systematically over-committing physical memory until the container was terminated. Message [msg 4105] captures the pivotal moment when the assistant synthesized the full picture of this accounting mismatch, calculated its catastrophic potential, and began reasoning toward a fix.

This article examines that single message in depth: the reasoning that led to the diagnosis, the assumptions that shaped the proposed solution, the input knowledge required to understand the problem, and the output knowledge that the message produced. It also explores the tension between quick fixes and principled architectural changes—a tension that would define the subsequent conversation.

Context: The Crash That Revealed the Blind Spot

The story begins with a production benchmark on a vast.ai RTX 5090 instance (C.32897009). The assistant had been iteratively improving a GPU pipeline dispatch system, implementing a PI-controlled pacer with EMA feed-forward, adding synthesis throughput caps, and deploying production Docker images. The benchmark was running with a memory budget of 331 GiB on a machine with a 342 GiB cgroup limit—a tight but seemingly workable configuration.

Phase 1 of the benchmark (5 warmup proofs) completed successfully in 518 seconds. But Phase 2 (the timed run of 10 proofs) failed almost immediately with a "transport error" and "broken pipe." The daemon had vanished. The instance became unreachable. The container had been OOM-killed.

The initial investigation in [msg 4100] and [msg 4101] revealed the proximate cause: the pinned memory pool had exhausted its free buffers, and the daemon log ended abruptly at the moment of peak pressure. But the root cause was more subtle. The assistant traced the chain of events:

The Moment of Synthesis: Message 4105

Message [msg 4105] opens with a striking calculation:

Now I understand the full picture. The pool comment on line 129 says "The pool is naturally bounded by synthesis_concurrency × 3 buffers" but synthesis_concurrency here means max_parallel_synthesis which defaults to 18. So the pool can grow to 18 × 3 × 3.88 GiB = 209 GiB of pinned memory!

This is the moment of reckoning. The pool's own documentation claimed it was "naturally bounded," but the assistant realized that the bound was not a limit at all—it was a formula for unbounded growth. With max_parallel_synthesis defaulting to 18, the pool could theoretically consume 209 GiB of pinned memory, all of it invisible to the budget system. On a machine with 342 GiB total, that left dangerously little room for everything else.

The assistant then traced the exact failure mechanism:

The problem is clear: the memory budget tracks "per-partition working memory" reservations, but the pinned pool holds onto the CUDA pinned memory even after the partition's budget reservation is released. So the budget thinks it has freed 14 GiB when a partition completes, but the pinned pool still holds 3×3.88=11.6 GiB of actual physical CUDA pinned memory.

This is the heart of the accounting mismatch. The budget system operates on a reservation model: each partition reserves its working memory before starting, and releases it when done. But the pinned pool operates on a reuse model: buffers are allocated once and kept forever, returned to a free list rather than freed. The two models are fundamentally incompatible. The budget sees a release and assumes the memory is available for other uses. The pool sees a check-in and assumes the memory is available for future partitions. Both are correct from their own perspective—but together, they create a systematic over-commit.

The Reasoning Process: Three Options Weighed

Having identified the root cause, the assistant enumerated three possible fixes:

  1. Cap the pinned pool size — Limit total pinned memory based on max_gpu_queue_depth or a config parameter.
  2. Track pinned pool in budget — Make the pool's allocations visible to the budget system, though this was explicitly removed previously due to double-counting.
  3. Free excess pool buffers — When buffers are checked in and the pool exceeds a threshold, free them with cudaFreeHost. The assistant noted that options 1 and 3 are "essentially the same: cap the pool size and free excess." It then attempted to calculate the right cap:
The right cap is based on how many partitions can be actively using pinned memory at once: - max_gpu_queue_depth partitions in the GPU queue (synthesized, waiting for GPU) - Plus ~2 partitions actively on GPU (one per worker) - Total: (max_gpu_queue_depth + gpu_workers) × 3 buffers

>

With defaults: (8 + 2) × 3 = 30 buffers × 3.88 GiB = ~116 GiB. Still a lot!

This calculation reveals the assistant's mental model of the pipeline: partitions flow from synthesis through a GPU queue to GPU workers, and pinned buffers are needed at every stage. The cap must cover the worst-case number of partitions simultaneously holding pinned buffers.

But then the assistant paused and reconsidered:

Actually, let me reconsider. The budget system already accounts for per-partition memory. The real issue is: 1. Budget reserves 14 GiB per PoRep partition for working memory 2. Pinned pool allocates 3×3.88=11.6 GiB of ACTUAL CUDA-pinned memory per partition 3. When the partition completes and its budget reservation is released, the 11.6 GiB of pinned memory stays in the pool 4. The budget now thinks it has 14 GiB free, but really only 14-11.6=2.4 GiB is truly freed

This numbered breakdown is the clearest articulation of the problem. Each partition's budget reservation of 14 GiB is supposed to cover all working memory, including the a/b/c vectors that are now allocated as pinned pool buffers. But because the pool retains those buffers after the reservation is dropped, the budget's accounting diverges from physical reality. The budget sees 14 GiB freed; the system sees only 2.4 GiB truly available.

The Double-Counting Paradox

The assistant recognized that this was the inverse of a previously encountered problem:

So the double-counting issue the comment mentions is real but backwards: the per-partition budget covers the a/b/c allocation, BUT when those allocations are pinned pool buffers, they persist after the budget reservation is dropped.

The comment in the pinned pool code had warned about double-counting: if the pool's allocations were tracked in the budget, and the per-partition reservations also accounted for a/b/c memory, the same memory would be counted twice. But the assistant realized that the current code had the opposite problem: by not tracking the pool at all, the budget under-counted the memory that was actually in use. The pool's persistence created a blind spot where physical memory was consumed but not accounted for.

This insight is crucial. It means that any correct fix must either:

Assumptions and Their Consequences

The assistant made several assumptions in this message that would prove significant:

Assumption 1: A cap is the simplest fix. The assistant concluded that "the simplest fix: cap the pool to a maximum total size and free excess on checkin." This assumed that a cap would be easy to implement, easy to configure, and would not harm performance. In the subsequent conversation, the user would reject this assumption, arguing that a cap is "unprincipled" and would "catastrophically harm performance on memory-constrained systems where pinned memory is the dominant operational memory."

Assumption 2: The cap should be based on GPU queue depth. The assistant calculated the cap as (max_gpu_queue_depth + gpu_workers) × 3 buffers. This assumed that the GPU queue depth is the right proxy for peak pinned buffer demand. But this ignores the fact that partitions being synthesized also hold pinned buffers—and synthesis concurrency can be much higher than GPU queue depth. The assistant's own earlier calculation showed that with max_parallel_synthesis=18, the pool could grow to 54 buffers, far more than the 30 buffers the cap would allow.

Assumption 3: Freeing excess buffers on check-in is safe. The assistant assumed that calling cudaFreeHost on excess buffers when they're checked in would not cause performance issues. But cudaFreeHost is a synchronous CUDA API call that can block, and frequent allocation/deallocation of large pinned memory regions could fragment the CUDA memory arena. The assistant did not consider these costs.

Assumption 4: The pool's growth is the sole problem. The assistant focused entirely on the pinned pool, but the OOM crash likely involved multiple factors: kernel/driver overhead (~6 GiB), PCE heap (~26 GiB), SRS (~44 GiB), and the pool. The budget's 331 GiB on a 342 GiB machine left only 11 GiB of headroom—too little for any accounting error. The pool was the trigger, but the insufficient safety margin was the underlying vulnerability.

Input Knowledge Required

To understand this message, a reader needs knowledge spanning several domains:

CUDA memory model: Understanding that cudaHostAlloc allocates pinned (page-locked) host memory, which cannot be paged out by the OS and is necessary for high-speed GPU transfers. This memory is physically resident and counts toward the process's RSS, making it visible to cgroup OOM killers.

The cuzk proving pipeline: Knowledge that each partition of a PoRep proof requires a/b/c vectors (~3.88 GiB each, three per partition), that synthesis produces these vectors, and that GPU proving consumes them. The pipeline has stages: synthesis → GPU queue → GPU execution, and pinned buffers are needed at each stage.

The memory budget system: Understanding that the MemoryBudget uses a reservation model with RAII guards (MemoryReservation), splitting memory into permanent (SRS, PCE) and working (per-partition) categories. Reservations are acquired before work begins and released when it completes.

The pinned pool implementation: The pool maintains a free list of pre-allocated pinned buffers. Partitions check out buffers (a, b, c) when they start synthesis and check them back in after GPU proving. The pool grows by allocating new buffers when the free list is empty, and never shrinks.

The crash context: The RTX 5090 instance had 342 GiB cgroup limit, 331 GiB budget, and was running PoRep-32g proofs with 10 partitions each. The daemon was killed during Phase 2 of the benchmark after Phase 1 had already stressed memory to 341 GiB.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

Quantified pool growth potential: The calculation that the pool can grow to 209 GiB (18 × 3 × 3.88 GiB) is a concrete, alarming number that reframes the problem from a minor accounting issue to a systemic risk.

The accounting mismatch model: The four-step breakdown of how budget reservations interact with pool persistence provides a clear mental model that can be applied to any similar system where resource pools and budget systems coexist.

The double-counting inversion: The insight that the previous double-counting concern (which led to removing pool tracking) was the opposite of the actual problem is a valuable lesson in how accounting errors can flip depending on allocation lifetimes.

Three solution options: The enumeration of cap, track, and free provides a structured decision space for the fix, even though the specific choice (cap) would later be rejected.

The pipeline capacity model: The calculation of (max_gpu_queue_depth + gpu_workers) × 3 buffers as the peak pinned buffer demand is a useful formula for sizing, even if it underestimates the true peak.

The Thinking Process: From Crash to Root Cause

The reasoning visible in this message is a masterclass in systems debugging. The assistant moves through several cognitive stages:

Stage 1: Quantification. The first thing the assistant does is calculate the worst-case pool size: 18 × 3 × 3.88 GiB = 209 GiB. This number is shocking—it's more than half the total system memory. By putting a number on the problem, the assistant transforms an abstract concern into a concrete crisis.

Stage 2: Tracing the failure path. The assistant reconstructs the exact sequence: Phase 1 allocated many buffers, the pool grew, Phase 2 started, more synthesis tasks launched, the pool was drained (free_remaining hit 0), and the combination of SRS + PCE + pool + working memory exceeded the cgroup limit. This narrative connects the code's behavior to the observed crash.

Stage 3: Identifying the mismatch. The key insight is that the budget and the pool have different lifetime models: budget reservations are transient, pool allocations are persistent. The assistant articulates this as: "the budget thinks it has freed 14 GiB when a partition completes, but the pinned pool still holds 3×3.88=11.6 GiB of actual physical CUDA pinned memory."

Stage 4: Exploring the solution space. The assistant enumerates three options and evaluates each. This structured approach shows a disciplined engineering mindset: don't just grab the first fix, consider the alternatives.

Stage 5: Settling on a direction. The assistant chooses the cap-and-free approach, reasoning that it's the simplest. This decision reflects a pragmatic trade-off: a simple fix that addresses the immediate problem, even if it's not the most architecturally pure solution.

Stage 6: Implementation planning. The assistant immediately moves to implementation, reading the config file to understand how to add the new parameter. This shows a bias toward action—having understood the problem, the assistant wants to fix it.

The Unresolved Tension: Quick Fix vs. Principled Architecture

The most interesting aspect of this message is what happens after it. The assistant's proposed cap-and-free approach is a quick fix: it addresses the symptom (unbounded pool growth) without addressing the root cause (the accounting mismatch between the budget and the pool). The cap is an arbitrary limit that must be tuned for each deployment, and it introduces a new failure mode: if the cap is set too low, partitions will fail to allocate pinned buffers and the pipeline will stall.

The user's subsequent rejection of this approach (in the following chunk) is a defense of architectural integrity: the pinned pool and memory manager must be "properly integrated to collaborate on memory management, avoiding thrashing while maximizing parallelism." The user insists on a principled solution where the pool is aware of the budget and participates in the reservation system, rather than being externally constrained by an arbitrary cap.

This tension—between quick fixes that solve today's crash and architectural changes that prevent tomorrow's—is a recurring theme in systems engineering. The assistant's message [msg 4105] represents the quick-fix mindset: identify the problem, calculate the numbers, implement the simplest correction. The user's response represents the architectural mindset: the system must be correct by design, not by configuration.

Both perspectives are valid. The quick fix would have stopped the OOM crashes immediately. The architectural fix would prevent a whole class of similar crashes in the future. The conversation that follows this message is the negotiation between these two approaches—a negotiation that ultimately leads to a deeper understanding of the budget system, the pool's role, and how they can be properly integrated.

Conclusion

Message [msg 4105] is a turning point in the debugging journey. It is the moment when scattered observations—a crashed daemon, a drained pool, a budget that thought it had free memory—coalesce into a coherent model of the failure. The assistant's calculation that the pool could grow to 209 GiB of invisible pinned memory is the kind of insight that changes how you think about a system. It reveals that the problem is not a minor configuration issue but a fundamental architectural flaw: two subsystems (budget and pool) with incompatible lifetime models, operating on the same physical memory without coordination.

The message also illustrates the power of quantified reasoning. Before the calculation, "the pool might grow too large" is a vague concern. After the calculation, "the pool can consume 209 GiB of invisible memory" is a concrete crisis that demands action. The assistant's ability to trace from a crash log to a specific code comment to a worst-case arithmetic calculation is exactly the kind of systems thinking that distinguishes effective debugging from guesswork.

Whether the cap-and-free approach was the right fix is a question the message itself does not answer. But the message's true value lies not in the solution it proposes but in the understanding it achieves. By the end of [msg 4105], the assistant has a complete, quantified, and articulated model of the accounting mismatch—and that understanding is the foundation for whatever fix comes next.