The Blind Spot in Memory Accounting: Diagnosing a Pinned Pool OOM in CuZK

Introduction

In any high-performance computing system, memory management is the invisible hand that keeps everything running. When that hand misses a finger, the entire system can collapse. Message 4104 in this opencode session captures a pivotal diagnostic moment: the assistant, after watching a benchmark crash on a 342 GiB RTX 5090 instance, traces the root cause to a fundamental accounting blind spot in the CuZK proving engine's memory budget system. The pinned memory pool—a performance optimization designed to accelerate GPU transfers—was silently consuming tens of gigabytes outside the budget's awareness, systematically over-committing the system until the cgroup OOM killer terminated the entire container. This article examines that diagnostic message in depth, exploring the reasoning, assumptions, and architectural insights it reveals.

The Scene: A Crash After a Successful Warmup

The context leading into message 4104 is a story of near-success. The assistant had been iterating on a GPU pipeline dispatch system for CuZK, a zero-knowledge proof engine for Filecoin. After fixing a bash script bug in benchmark.sh that had masked earlier crashes (see <msg id=4095-4103>), the assistant deployed a fresh benchmark run on a vast.ai RTX 5090 instance. Phase 1—a warmup of 5 proofs—completed successfully in 518 seconds. The pipeline was working, the GPU workers were busy, and the system appeared healthy.

Then Phase 2 began. The first proof started synthesizing, and within minutes the daemon vanished. The benchmark client reported a "transport error" / "broken pipe." When the assistant tried to SSH back into the instance, it was gone: Connection refused. The container had been killed.

The assistant's initial suspicion was an OOM (Out of Memory) kill at the container level. The cgroup memory limit was 342 GiB, and the budget had been set to 331 GiB. During Phase 1, memory usage had peaked at 341 GiB—already at 100% of the limit. Phase 2 launched more synthesis tasks, and the system tipped over. But why? The budget was set to 331 GiB, leaving 11 GiB of headroom. Where did the extra memory come from?

The Core Diagnosis: A Missing Accounting Entry

Message 4104 opens with the assistant's synthesized understanding:

OK, the core issue is now clear. The instance was killed because of a combination of factors:

>

1. Pinned pool memory is NOT tracked in the budget — by design (the comment says "pinned memory replaces heap a/b/c allocations that are already accounted for in per-partition budget reservations"). But this assumption is wrong...

This is the crux of the diagnosis. The CuZK proving engine uses a MemoryBudget system to track and limit memory usage, preventing the process from exceeding the cgroup limit and getting OOM-killed. The budget accounts for permanent allocations (like the SRS parameters, ~44 GiB) and per-partition working memory (heap allocations for synthesis). However, the pinned memory pool—a separate subsystem that allocates cudaHostAlloc buffers for fast GPU transfers—operates outside this budget.

The design rationale, as the assistant reconstructs it, was that pinned memory "replaces heap a/b/c allocations that are already accounted for in per-partition budget reservations." The idea is that when a partition uses a pinned buffer, it's not additional memory—it's a substitute for heap memory that was already reserved. If the budget already reserved ~14 GiB per partition for heap working memory, and the pinned pool just replaces some of that heap with pinned memory, then the total memory usage shouldn't exceed the reservation.

Why the Assumption Was Wrong

The assistant identifies three reasons this assumption fails in practice:

First, the pinned pool holds onto buffers after synthesis completes. When a partition finishes synthesizing, its pinned buffers are returned to the pool for reuse, but the underlying cudaHostAlloc memory is never freed. The pool grows monotonically to accommodate peak concurrent demand. On this run, 8 partitions were simultaneously synthesizing, each holding 3 pinned buffers of ~3.88 GiB, totaling ~93 GiB of pinned memory. This memory persists even after the partitions that allocated it have completed.

Second, the per-partition budget reservations are released when a partition completes. The budget uses RAII-style MemoryReservation guards: when a partition starts, it reserves ~14 GiB; when it finishes, the reservation is released. But the pinned pool retains the physical memory. The budget sees the reservation released and thinks memory is available, while the pinned pool is still holding tens of gigabytes. This creates a systematic over-commit cycle: each new wave of partitions sees budget available, allocates more pinned buffers, the pool grows, and the budget remains blind to the accumulation.

Third, multiple proofs can be in-flight simultaneously. The pipeline architecture deliberately overlaps synthesis and GPU proving to maximize throughput. This means multiple proofs' partitions can be active at once, each contributing to the pinned pool. The pool's total allocation is a function of peak concurrency, not per-partition working set.

The Memory Budget Breakdown

The assistant performs a concrete accounting exercise to illustrate the mismatch:

On a 342 GiB machine with 331 GiB budget: - SRS: ~44 GiB (pinned, counted in budget as permanent) - PCE: ~26 GiB (heap) - Kernel/driver overhead: ~6 GiB - Available for working memory: 342 - 44 - 26 - 6 = ~266 GiB - But budget thinks it has 331 - 44 = ~287 GiB for working memory - The pinned pool grows outside budget tracking → OOM

This breakdown reveals two problems. First, the budget of 331 GiB is already dangerously close to the cgroup limit of 342 GiB—only 11 GiB of headroom. Second, the budget's calculation of available working memory (287 GiB) is 21 GiB higher than the true available memory (266 GiB), because it doesn't account for PCE heap and kernel overhead. And this is before the pinned pool enters the picture. The pinned pool's ~93 GiB is entirely invisible to the budget, making the actual memory pressure far worse than the budget realizes.

The assistant's arithmetic is devastatingly clear: the system was operating at 99% of the cgroup limit during Phase 1, with no headroom for the additional allocations that Phase 2 would trigger. The OOM kill was not a random failure—it was an inevitability given the accounting mismatch.

The Thinking Process: From Symptoms to Root Cause

What makes message 4104 particularly valuable is the visible reasoning process. The assistant doesn't just state the conclusion; it walks through the logic step by step, showing how each piece of evidence fits together.

The message begins with the assistant's own summary of the "core issue," then immediately qualifies it with a parenthetical note about the design rationale: "by design (the comment says...)." This shows the assistant has read the source code and understands the intended behavior, not just the observed behavior. The assistant is distinguishing between the design intent and the actual consequences—a crucial analytical skill.

The assistant then enumerates three bullet points explaining why the design assumption fails, each building on the previous one. The first point identifies the temporal mismatch (pool holds buffers after completion). The second identifies the accounting mismatch (budget releases reservations while pool retains memory). The third identifies the concurrency dimension (multiple proofs amplify the effect).

The numerical breakdown that follows is the synthesis: it takes the abstract accounting problem and makes it concrete with actual numbers from the crashed instance. This transforms a theoretical concern into a proven root cause.

Finally, the assistant lists four possible fixes, ranging from the most principled (track pinned pool in the budget) to the most pragmatic (reduce the budget). The message ends with the assistant beginning to read the pinned_pool.rs source file—a concrete next step toward implementing a fix.

Input Knowledge Required

To fully understand message 4104, the reader needs several pieces of context:

CuZK architecture knowledge: The CuZK proving engine has a pipeline architecture where proofs are broken into partitions, each partition undergoes synthesis (CPU computation to generate witness data), and then GPU proving. The system uses a MemoryBudget to track allocations and prevent OOM.

Pinned memory concepts: CUDA pinned memory (cudaHostAlloc) allocates host memory that is page-locked, allowing fast DMA transfers to the GPU. Unlike regular heap memory, pinned memory is not swappable and is typically allocated at a higher cost. The pinned pool is a reuse mechanism to avoid repeated allocation/deallocation.

Budget accounting model: The MemoryBudget tracks permanent memory (SRS parameters, loaded at startup and never freed) and working memory (per-partition allocations that are reserved when a partition starts and released when it finishes). The budget is configured as a fraction of the cgroup memory limit, with a safety margin.

cgroup OOM behavior: Linux cgroups (v1 or v2) enforce memory limits by killing processes that exceed the limit. The OOM killer targets the entire cgroup, not individual processes. When the CuZK daemon was killed, the entire container went down, making SSH impossible.

The previous debugging session: The assistant had just fixed a bash script bug in benchmark.sh (line 346) that was masking earlier crashes. That fix allowed Phase 1 to complete, revealing the true OOM issue underneath.

Output Knowledge Created

Message 4104 produces several forms of knowledge:

A confirmed root cause: The pinned pool memory is not tracked in the budget, causing systematic over-commit. This is the definitive diagnosis that explains the crash.

A quantitative model: The memory breakdown (SRS + PCE + kernel overhead + pinned pool vs. budget) provides a concrete framework for understanding memory pressure on any instance, not just this one.

A taxonomy of possible fixes: The four options listed (track in budget, cap pool size, reduce budget, free buffers) define the solution space. Each has different trade-offs in complexity, performance impact, and correctness.

A design critique: The message identifies a flaw in the original design assumption—that pinned memory "replaces" heap allocations and therefore doesn't need separate tracking. This is a valuable architectural insight that applies beyond this specific system.

A debugging methodology: The assistant's approach—gathering evidence (log tails, memory stats, process state), forming a hypothesis, testing it against the numbers, and then reading the source code to confirm—is a model of systematic debugging.

Assumptions and Potential Limitations

The assistant's diagnosis makes several assumptions that deserve scrutiny:

The assumption that the pinned pool grows monotonically: The assistant states "the total pool size only grows (it never shrinks)." This is likely true given the current implementation (the pool reuses buffers but never frees them), but it's worth verifying. If the pool did shrink under some conditions, the diagnosis would need refinement.

The assumption that kernel/driver overhead is ~6 GiB: This is an estimate, not a measured value. The actual overhead could vary depending on GPU driver version, kernel configuration, and other factors. The 6 GiB figure is plausible but unverified.

The assumption that the budget is the right level of abstraction for tracking: The assistant's preferred fix is to "track pinned pool memory in the budget." This assumes the budget can accurately predict pinned pool allocation sizes, which depends on the pool's allocation strategy and the number of concurrent partitions. If the pool allocates lazily or in variable sizes, tracking may be complex.

The assumption that the cgroup OOM killed the container: The instance became unreachable, which is consistent with an OOM kill, but there could be other explanations (network failure, SSH daemon crash, Docker restart). The assistant reasonably infers OOM from the memory pressure evidence, but this is a post-hoc inference rather than a direct observation.

Conclusion

Message 4104 is a masterclass in diagnostic reasoning. It takes a complex system failure—a container that vanished under load—and traces it to a specific accounting blind spot: the pinned memory pool operating outside the budget's awareness. The assistant's analysis is clear, quantitative, and grounded in both source code understanding and empirical observation. The message transforms a frustrating crash into a well-defined engineering problem with a known solution space.

The deeper lesson is about the fragility of performance optimizations. The pinned pool was introduced to accelerate GPU transfers by avoiding the CUDA bounce buffer bottleneck. It succeeded at that goal—but it created an invisible memory liability that grew silently until it overwhelmed the system. The budget system, designed to prevent OOM, was rendered ineffective because it didn't account for this optimization. This tension between performance and safety is a recurring theme in systems engineering, and message 4104 captures it perfectly.

The assistant's next step—reading the pinned_pool.rs source—promises a deeper dive into the implementation details. But message 4104 itself stands as a complete diagnostic arc: from symptom to root cause, from confusion to clarity, from crash to actionable fix. It is the kind of message that every engineer hopes to write when faced with a mysterious failure, and it serves as a model for systematic debugging in complex systems.