The Safety Margin That Wasn't: Diagnosing an OOM Kill in a GPU Memory Manager

Message Overview

The subject message, <msg id=2369>, is an assistant message in an opencode coding session where the AI is building and debugging a budget-based memory manager for the cuzk GPU proving engine. In this message, the assistant discovers that a production-scale benchmark has crashed with an OOM (Out of Memory) kill, and performs forensic analysis to understand why a carefully designed memory budget system failed to prevent the daemon from being killed by the kernel.

The message contains two parts: an Agent Reasoning block where the assistant thinks through the failure, and a bash tool call that executes diagnostic commands on the remote machine to confirm the hypothesis. The reasoning reveals a subtle and instructive failure mode: the safety_margin configuration parameter was set to a mere 5 GiB, which was catastrophically insufficient to account for co-located processes (specifically Curio, using ~87 GiB of shared memory) and kernel overhead. The budget system correctly limited cuzk's own allocations, but it had no visibility into memory consumed by other processes on the same machine.

The Message in Full

## Agent Reasoning The daemon crashed — transport error means the connection was broken, likely OOM-killed at ~500 GiB RSS. The machine has 755 GiB total but some is used by other processes (curio was using ~87 GiB shared memory). The kernel killed cuzk when it approached the available memory.

>

The budget says 750 GiB but the actual system available memory is only ~528 GiB (from earlier free -h). The problem is safety_margin = "5GiB" is too low when there are other significant processes running (like Curio).

>

Also, with 30 partitions all starting simultaneously at 14 GiB each = 420 GiB working set + 44 GiB SRS + 26 GiB PCE = 490 GiB.

>

Adding Curio's ~87 GiB brings the total to around 577 GiB, which is dangerously close to the 755 GiB limit and triggers OOM with kernel overhead factored in. The safety margin of 5 GiB doesn't account for these other processes, so I need to either increase it substantially to around 250 GiB to cover Curio and OS overhead, or set an explicit budget cap at 500 GiB instead.

>

OOM killed — RSS hit 498 GiB with Curio using another ~87 GiB. The safety_margin=5GiB doesn't account for Curio. Let me check the OOM kill and restart with a proper safety margin:

>

[bash] ssh -p 40612 root@141.0.85.211 'dmesg | grep -i "oom\|kill" | tail -5; echo "==="; tail -5 /tmp/cuzk-memtest.log; echo "==="; ps aux | grep curio | grep -v grep | awk "{print \$6/1048576 \" GiB\", \$11}"'

Context: The Memory Manager Journey

To understand this message, one must appreciate the arc of the preceding work. The assistant had spent several segments designing and implementing a unified memory management system for cuzk, a GPU-based zero-knowledge proving engine for Filecoin. The old system used a static partition_workers limit that was fragile and did not account for varying proof sizes or SRS/PCE memory footprints. The new system introduced a MemoryBudget with admission control: each component (SRS loading, PCE extraction, partition synthesis) would acquire a reservation from the budget before allocating memory, and the budget would block until sufficient memory was available.

The system had three key parameters: total_budget (the total memory available to cuzk), safety_margin (memory reserved for other processes and OS), and eviction_min_idle (how long SRS/PCE data must be idle before eviction). The budget could be set to "auto", which would detect total system memory and subtract the safety margin.

In the immediately preceding messages, the assistant had been debugging a different problem: with a tight 100 GiB budget, only one partition synthesized at a time because the SRS (44 GiB) + PCE (26 GiB) baseline left only ~30 GiB for working sets, and a race condition in SRS pre-acquisition caused three concurrent proofs to each reserve 44 GiB simultaneously. The user had complained that "there is only one synthesis running at a time, previously the machine was handling 10 or 8."

The assistant's response was to switch from the artificially tight 100 GiB budget to "auto" mode, which detected 755 GiB of total RAM and, with a 5 GiB safety margin, set the budget to 750 GiB. The result was dramatic: all 30 partitions started within one second. But then the daemon crashed.

The Reasoning Process: A Step-by-Step Reconstruction

The Agent Reasoning block reveals a sophisticated diagnostic thought process. Let me trace through it step by step.

Step 1: Recognizing the symptom. The assistant sees "transport error" and "stream closed because of a broken pipe" in the benchmark output. This is not a normal error — it means the server process (cuzk daemon) disappeared abruptly. The assistant correctly infers an OOM kill: "The daemon crashed — transport error means the connection was broken, likely OOM-killed at ~500 GiB RSS."

Step 2: Recalling prior knowledge. The assistant remembers that "curio was using ~87 GiB shared memory" from earlier investigation. This is crucial context — the machine is not dedicated to cuzk alone. Curio, another component of the Filecoin proving stack, is also running and consuming significant memory.

Step 3: Performing the budget arithmetic. The assistant calculates: 30 partitions × 14 GiB each = 420 GiB working set, plus 44 GiB SRS, plus 26 GiB PCE = 490 GiB total for cuzk. Adding Curio's ~87 GiB brings the total to ~577 GiB. The machine has 755 GiB total, but the assistant recalls that free -h showed only ~528 GiB available — the discrepancy between total RAM and available RAM is due to kernel overhead, filesystem cache, and other system processes.

Step 4: Identifying the root cause. The safety margin of 5 GiB was intended to reserve memory for other processes, but it was orders of magnitude too small. Curio alone consumed ~87 GiB, and kernel overhead consumed additional memory. The budget system correctly limited cuzk's own allocations to 750 GiB, but that number was based on total system RAM minus a trivial safety margin, not on actual available memory after accounting for co-located processes.

Step 5: Formulating the fix. The assistant identifies two possible solutions: increase the safety margin to ~250 GiB (to cover Curio + OS overhead), or set an explicit budget cap at ~500 GiB. Both approaches would prevent cuzk from consuming memory that belongs to other processes.

Assumptions and Their Failure Modes

This message is particularly instructive because it reveals several assumptions that turned out to be incorrect:

Assumption 1: The safety margin covers all non-cuzk memory. The assistant assumed that a 5 GiB safety margin would be sufficient to account for other processes. This assumption failed because the margin was not based on actual measurement of co-located process memory consumption. The assistant had earlier seen Curio's memory usage but did not incorporate it into the safety margin calculation.

Assumption 2: Auto-detection of total memory is sufficient. The "auto" mode detects total system RAM via /proc/meminfo and subtracts the safety margin. But total RAM is not the same as available RAM. The kernel reserves memory for itself, for page cache, for slab allocator, and for other purposes. The assistant had seen free -h output showing only ~528 GiB available out of 755 GiB total, but did not adjust the safety margin to account for this gap.

Assumption 3: The budget system is self-contained. The memory manager was designed as if cuzk were the only significant memory consumer on the machine. In reality, the machine runs Curio, system services, and the kernel itself. The budget system has no mechanism to discover or account for memory consumed by other processes.

Assumption 4: 30 partitions can start simultaneously without issue. The assistant enabled synthesis_concurrency = 4 and the budget allowed all 30 partitions to dispatch within one second. But dispatching 30 partitions simultaneously means all 30 working set reservations (420 GiB total) are acquired at once, even though the GPU can only process a few at a time. The budget system did not enforce any ordering or throttling beyond the raw numeric limit.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

  1. The cuzk proving engine architecture: cuzk is a GPU-based zero-knowledge proof generator for Filecoin. It uses a pipeline with synthesis (constraint system generation) and GPU proving (Groth16). Proofs are split into partitions for parallel processing.
  2. The memory budget system: A recently implemented admission control system where components acquire reservations from a MemoryBudget before allocating memory. The budget tracks used and available amounts and blocks if insufficient memory remains.
  3. The SRS/PCE concepts: SRS (Structured Reference String) is a ~44 GiB cryptographic parameter loaded once and shared across proofs. PCE (Pre-Compiled Constraint Evaluator) is a ~26 GiB cached constraint evaluation that accelerates synthesis.
  4. The deployment environment: A remote machine with 755 GiB RAM, an RTX 5090 GPU, and 64 CPU cores. The machine also runs Curio, another component of the Filecoin proving infrastructure.
  5. OOM killer behavior: The Linux kernel's Out-Of-Memory killer terminates processes when system memory is exhausted. It selects a victim based on a heuristic that considers memory usage, process priority, and other factors. OOM kills typically manifest as abrupt process termination without a clean shutdown.
  6. Linux memory accounting: The difference between total RAM, available RAM (as reported by free -h), and RSS (Resident Set Size). Available memory accounts for reclaimable page cache and slab, which can be freed under pressure, but may not be sufficient to prevent OOM.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. A confirmed failure mode: The budget-based memory manager works correctly for isolating cuzk's own memory usage, but it cannot prevent OOM if the safety margin does not account for co-located processes. This is a fundamental architectural insight: a per-process memory budget is insufficient without system-wide memory visibility.
  2. A quantitative failure analysis: The exact arithmetic of the OOM is documented: 490 GiB (cuzk) + 87 GiB (Curio) + kernel overhead ≈ 577+ GiB, exceeding the 755 GiB total and triggering the OOM killer. This analysis can be reused for capacity planning.
  3. Two concrete fix options: The assistant identifies increasing the safety margin to ~250 GiB or setting an explicit cap at ~500 GiB. Both approaches are simple configuration changes that would prevent the OOM without code modifications.
  4. A diagnostic procedure: The bash command demonstrates how to check for OOM kills (dmesg | grep -i "oom\|kill"), inspect daemon logs for the last messages before death, and measure Curio's memory consumption. This procedure is reusable for future debugging.
  5. A design requirement: The message implicitly defines a requirement for the memory manager: it must either discover and account for other processes' memory usage, or require the operator to specify a safety margin based on measured system conditions. The "auto" mode is insufficient without a properly sized safety margin.

The Thinking Process: What the Reasoning Reveals

The Agent Reasoning block is particularly valuable because it shows the assistant's diagnostic methodology. The thinking is structured as a series of logical deductions:

  1. Symptom → Cause: Transport error → process death → OOM kill (deduction based on error pattern)
  2. Cause → Mechanism: OOM kill → memory exhaustion → cuzk RSS + Curio RSS > available memory (arithmetic verification)
  3. Mechanism → Root Cause: Memory exhaustion → safety margin too small → margin doesn't account for Curio (identification of the design flaw)
  4. Root Cause → Fix: Safety margin too small → increase to 250 GiB or set explicit 500 GiB cap (formulation of solution) The reasoning also shows the assistant integrating multiple data sources: the benchmark error message, the RSS trace showing ~500 GiB peak, the earlier free -h output showing ~528 GiB available, and the knowledge that Curio uses ~87 GiB. This multi-source synthesis is a hallmark of effective debugging. Notably, the assistant does not blame the budget system itself. The budget worked correctly — it allowed 30 partitions to dispatch because 750 GiB was available in the budget. The failure was in the configuration of the budget, not in its implementation. This is an important distinction: the code was correct, but the parameters were wrong for the deployment environment.

Broader Implications

This message illustrates a classic systems engineering challenge: the gap between a component's internal resource management and the system-wide resource reality. The memory manager was designed as a self-contained allocator, but memory is a shared resource. No amount of internal budget tracking can prevent OOM if the budget does not account for external consumers.

The lesson extends beyond this specific case. Any resource management system that uses a "total budget minus safety margin" model must ensure that the safety margin is based on actual measurement of co-located resource consumption, not on an arbitrary small value. The assistant's instinct to set safety_margin = "5GiB" was reasonable for a dedicated machine, but the machine was not dedicated.

The message also demonstrates the importance of understanding the difference between "total memory" and "available memory" in Linux. The "auto" mode used total RAM (755 GiB) as its baseline, but the kernel's available memory calculation already accounts for reclaimable caches and kernel structures. Using total RAM as the budget baseline double-counts memory that is not actually available for application allocation.

Conclusion

Message <msg id=2369> is a turning point in the memory manager development. It reveals that the budget system, while architecturally sound, requires careful configuration to match the deployment environment. The assistant's reasoning demonstrates a mature debugging methodology: trace the symptom to its root cause, verify with arithmetic, and formulate a concrete fix. The message also serves as a cautionary tale about the dangers of assuming a single-process view of memory in a multi-process environment.

The fix is straightforward — increase the safety margin — but the insight is profound: a memory budget is only as good as its awareness of the system it runs in. The assistant's next steps will likely involve either documenting this requirement, adding a mechanism to discover co-located process memory usage, or both. The OOM kill was a failure, but it was a learned failure, and the knowledge it produced will make the system more robust.