The OOM Recovery Loop: A Single Decision That Made a Production System Resilient

Introduction

In the middle of a debugging session that had already spanned hours of investigation into out-of-memory (OOM) crashes on GPU-proving instances, a single message from the AI assistant crystallized the team's strategy into executable code. Message [msg 4009] reads, in its entirety:

Now I need to wrap the core benchmark logic in an OOM-recovery retry loop. The key insight: if the daemon dies (exit 137 = OOM kill), we increase the safety margin by 50% and restart. Let me restructure benchmark.sh:

On its surface, this is a straightforward engineering statement: the assistant is about to edit a shell script. But the message represents a critical inflection point in a much longer debugging journey. It is the moment when speculative analysis about kernel overhead, page tables, glibc arenas, and pinned memory accounting gave way to a pragmatic, self-healing mechanism. This article examines that single message in depth: the reasoning that led to it, the decisions it encodes, the assumptions it rests on, and the knowledge it both consumes and produces.

The Context: Why This Message Was Written

To understand why message [msg 4009] exists, one must understand the crisis that preceded it. The team had deployed a CUDA-based zero-knowledge proving system (cuzk) onto vast.ai GPU instances for production benchmarking. On a 342 GiB cgroup-limited machine, the daemon was crashing with OOM kills during GPU processing. The 10 GiB safety margin—a simple heuristic subtracting 10 GiB from the cgroup limit to arrive at a memory budget—was proving empirically insufficient.

The investigation (visible in the assistant's extensive reasoning in [msg 3999]) had traced the problem to multiple overlapping causes. The CUDA pinned memory pool (PinnedPool) operated outside the MemoryBudget accounting system: pinned buffers returned to the pool after GPU work were never freed from actual RSS, creating a massive accounting discrepancy. Kernel and driver overhead—glibc malloc arenas consuming ~1.1 GiB across 18 threads, thread stacks adding 144 MiB, kernel slab caches, and GPU driver allocations—ate into the available memory in ways the simple safety margin did not capture. And during SRS (Structured Reference String) loading, a transient spike from simultaneous mmap of the 44 GiB parameter file and cudaHostAlloc of the in-memory representation temporarily doubled the memory footprint.

The user had proposed a two-pronged strategy in [msg 3997]: "maybe we should have a program before benchmark which tries to actually allocate memory, and in benchmark runs also recover from OOMs and retry with 50% overhead a few times." The assistant had already implemented the first prong—a memprobe C utility that empirically measures available memory by allocating 1 GiB chunks via mmap and memset until it nears the cgroup limit ([msg 4000]). Message [msg 4009] is the execution of the second prong: the OOM recovery loop in benchmark.sh.

The Decision: What the Message Actually Does

The message itself is an edit operation on /tmp/czk/docker/cuzk/benchmark.sh. The assistant states its design principle with remarkable concision: "if the daemon dies (exit 137 = OOM kill), we increase the safety margin by 50% and restart."

This encodes several decisions:

Exit code 137 as the signal. On Linux, when the OOM killer terminates a process, the process receives SIGKILL (signal 9). A process killed by SIGKILL exits with code 128 + 9 = 137. The assistant chose this as the trigger for recovery. This is a deliberate choice: not all crashes are OOM kills. A segfault (signal 11, exit code 139) or an abort (signal 6, exit code 134) would indicate a different class of bug, one unlikely to be resolved by reducing memory pressure. By keying specifically on exit code 137, the assistant ensures the recovery loop only fires when the root cause is genuinely memory exhaustion.

Geometric reduction of the safety margin. The 50% increase in safety margin is a multiplicative, not additive, adjustment. If the initial safety margin is 10 GiB, a failure increases it to 15 GiB, then to 22.5 GiB, then to 33.75 GiB. This geometric progression is a standard control-theory approach: it converges rapidly to a safe operating point while requiring only a small number of retries. The assistant's todo list (visible in [msg 3999]) specified "up to 3 retries," which means the safety margin could grow from 10 GiB to 33.75 GiB in the worst case—a 3.4× increase that would almost certainly find a stable budget on any machine that can run the workload at all.

Restart, not graceful degradation. The loop restarts the entire daemon rather than trying to dynamically adjust memory allocation at runtime. This is a pragmatic choice: the daemon's memory allocation patterns are complex, involving pinned CUDA memory, synthesis buffers, and GPU queue depths. Trying to hot-reconfigure these mid-flight would be fragile. A clean restart with a more conservative budget is simpler and more reliable.

The Assumptions Embedded in the Design

Every engineering decision rests on assumptions, and this message is no exception.

Assumption 1: OOM is the only failure mode that matters. The recovery loop only triggers on exit code 137. This assumes that all production crashes on memory-constrained instances are OOM kills, not logic bugs, hardware errors, or network failures. The team's empirical data supported this: the 342 GiB instance was crashing with "transport error" and "broken pipe" messages that were downstream symptoms of the OOM kill (the daemon dies, the benchmark client loses its connection). But the assumption could fail if a different class of bug emerges.

Assumption 2: Reducing the budget is sufficient to prevent recurrence. The loop assumes that the same workload will succeed with a smaller memory budget. This is not guaranteed: some workloads have hard minimum memory requirements (e.g., the SRS must fit in memory regardless of the safety margin). If the budget drops below the irreducible minimum, the loop will retry forever (or until its retry limit) without success. The 3-retry cap is a practical safeguard against this infinite-loop scenario.

Assumption 3: The safety margin is the right knob to turn. The assistant chose to increase the safety margin (i.e., reduce the memory budget available to the daemon) rather than reducing concurrency, synthesis parallelism, or GPU queue depth. This assumes that memory pressure is a continuous function of the budget—that reducing the budget by X% reduces peak memory usage by roughly X%. In reality, memory usage can be lumpy: the SRS allocation is a fixed 44+ GiB regardless of the safety margin, and reducing the budget below that threshold guarantees failure. The geometric increase in safety margin makes this assumption safer: it quickly moves the budget well below any plausible fixed allocation.

Assumption 4: The daemon's exit code is reliably propagated. The benchmark script must correctly capture the daemon's exit code. If the script uses a shell construct that masks the exit code (e.g., piping through tee or running in a subshell), the recovery loop could miss OOM kills. The assistant's edit presumably handles this, but the assumption is implicit in the design.

Input Knowledge Required

To understand message [msg 4009], a reader needs substantial domain knowledge:

Linux OOM behavior. The significance of exit code 137 (128 + SIGKILL) is not obvious without knowing that the OOM killer sends signal 9. A reader unfamiliar with Linux signal handling would see "exit 137" as an arbitrary magic number.

The cgroup memory model. The entire memory budgeting system is built around cgroup v2 limits. The assistant had previously rewritten detect_system_memory() to read memory.max (cgroup v2) and memory.limit_in_bytes (cgroup v1). The OOM recovery loop operates within this model: it adjusts the safety margin that is subtracted from the cgroup limit to produce the daemon's memory budget.

CUDA pinned memory. The crash's root cause involved cudaHostAlloc allocations that were not properly accounted for by the MemoryBudget system. Understanding why the OOM killer fires requires knowing that pinned memory is non-evictable and counts against the cgroup limit immediately upon allocation.

The benchmark architecture. The benchmark.sh script launches a cuzk-daemon process, then runs proofs against it via a client. The daemon is the memory-intensive component. The recovery loop must kill any remaining daemon process from the failed attempt before restarting, to avoid accumulating memory from multiple instances.

Output Knowledge Created

Message [msg 4009] produces a modified benchmark.sh that embodies a specific operational pattern. The output knowledge includes:

A reusable OOM recovery pattern. The loop structure—detect specific failure, geometrically increase safety margin, restart, cap retries—is a general pattern applicable to any memory-constrained service. It could be extracted and reused for other daemons or workloads.

Empirical calibration data. Each OOM recovery event produces a data point: "on machine with X GiB cgroup limit, safety margin of Y GiB was insufficient; margin of Z GiB succeeded." Over time, these data points build a empirical model of the true overhead on different instance types, potentially allowing the team to set instance-specific default safety margins.

A safety net for the memprobe. The memprobe utility ([msg 4000]) measures available memory before the daemon starts, but it cannot account for the daemon's runtime overhead. The OOM recovery loop acts as a second line of defense: if the memprobe's estimate is too optimistic, the loop catches the resulting OOM and adjusts. The two mechanisms together provide both proactive (memprobe) and reactive (OOM loop) memory safety.

The Thinking Process

The reasoning behind message [msg 4009] is not in the message itself—it is in the preceding message [msg 3999], which contains an extensive "Agent Reasoning" block. This block reveals the assistant's thought process as it works through the problem:

The assistant starts by restating the user's suggestion, then immediately begins quantifying the problem. It computes the overhead from glibc malloc arenas (~1.1 GiB for 18 threads), thread stacks (144 MiB), and kernel slab caches. It estimates total overhead at 3-5 GiB beyond the 10 GiB safety margin. These are not precise measurements—they are educated guesses based on standard Linux memory behavior—but they serve to bound the problem.

The assistant then considers the SRS loading path in detail, working through the C++ code's memory allocation patterns. It traces through the cudaHostAlloc call, the mmap of the parameter file, and the potential for a compressed-vs-uncompressed point format mismatch. It catches itself speculating ("Rather than getting lost in the internals") and redirects to the practical solution. This self-correction is a key reasoning pattern: the assistant recognizes when analysis is becoming unproductive and pivots to action.

The todo list at the end of [msg 3999] shows the assistant's execution plan: write memprobe.c, add it to the Dockerfile, update benchmark.sh with OOM recovery, update entrypoint.sh. Message [msg 4009] is the third step in this plan. The assistant is methodically working through a checklist, and the message's brevity reflects its focus on execution rather than analysis.

Conclusion

Message [msg 4009] is a study in concise engineering communication. In two sentences, it states the problem, the solution, and the mechanism. But the depth behind those sentences is considerable: hours of debugging, careful analysis of Linux memory management, understanding of CUDA pinned memory behavior, and a pragmatic philosophy of building self-healing systems. The OOM recovery loop it introduces is not a fix for the root cause—the pinned memory accounting mismatch remains—but it is a robust operational band-aid that allows the system to function in production while the deeper issues are addressed. Sometimes the best engineering is not the most elegant fix, but the one that keeps the system running long enough to find the elegant fix.