The Memory Probe and OOM Recovery: A Turning Point in Production Reliability

Introduction

In the course of building a high-performance GPU-accelerated proof generation system for Filecoin, a team encountered a recurring and frustrating problem: the cuzk daemon would crash with an out-of-memory (OOM) kill on certain cloud instances, even when the memory budget appeared to be correctly calculated. The crashes were intermittent, workload-dependent, and extremely difficult to reproduce in controlled environments. After weeks of debugging—tracing pinned memory pool accounting, investigating GPU dispatch pacing, and implementing cgroup-aware memory detection—the team arrived at a critical juncture. A single message from the AI assistant, message 3999 in the conversation, crystallized the problem and proposed a two-pronged solution that would fundamentally change how the system handled memory constraints.

This article examines that message in depth: the reasoning that led to it, the decisions it embodies, the assumptions it makes, and the knowledge it creates. The message is a masterclass in practical systems engineering—balancing deep technical investigation with pragmatic, deployable solutions.

Context: The OOM Crash That Wouldn't Go Away

The conversation leading up to message 3999 takes place within a larger project to build cuzk, a CUDA-accelerated zero-knowledge proof generation system for Filecoin's proof-of-replication (PoRep) protocol. The system operates on rented GPU cloud instances from vast.ai, where memory configurations vary wildly. Some instances have 961 GiB of RAM; others have as little as 256 GiB. The system must work reliably across this spectrum.

Earlier in the session (Segment 24-29 of the conversation), the team had already solved several memory-related problems:

  1. Pinned memory pool: They implemented a CUDA pinned memory pool to accelerate H2D transfers, reducing transfer times from ~4 GB/s to near PCIe bandwidth.
  2. GPU dispatch pacing: They replaced a naive semaphore-based dispatch with a PI controller that dynamically adjusted GPU queue depth based on memory pressure.
  3. Cgroup-aware memory detection: They rewrote detect_system_memory() to read cgroup v2 (memory.max) and v1 (memory.limit_in_bytes) limits, preventing the system from using host RAM values that were meaningless inside Docker containers.
  4. Safety margin: They added a 10 GiB safety margin between the cgroup limit and the memory budget to account for kernel overhead. Despite all these improvements, a crash occurred on a 342 GiB instance (RTX PRO 4000) during a benchmark run. The crash happened during GPU processing—specifically during NTT/MSM operations—and the daemon died with a broken pipe error, consistent with an OOM kill. The immediate preceding messages show the team trying to understand why. In message 3985, the user states "We track memory/pinned memory in memory manager," pushing back against a hypothesis that the pinned pool was untracked. In message 3986, the assistant works through the budget math: 44 GiB for SRS, 14 GiB per partition for synthesis, up to 18 parallel syntheses, PCE extraction transients—all fitting within the 331 GiB budget (342 GiB minus 10 GiB safety). The assistant considers GPU memory exhaustion (the RTX PRO 4000 has only 24 GiB VRAM) but concludes the crash was host-side OOM because a GPU OOM would return a CUDA error code, not kill the process. In message 3987, the user offers a cryptic hint: "Maybe in SRS-mode it a bit weird." This triggers a deep investigation into how the SRS (Structured Reference String) is loaded. The assistant traces the code path through srs_manager.rs, supraseal_params.rs, and into the C++ groth16_srs.cuh file, discovering that SRS loading involves a simultaneous mmap of the 44 GiB file plus a cudaHostAlloc for the in-memory representation. The assistant speculates about point format size mismatches (compressed vs. uncompressed BLS12-381 points) and a conditional compilation flag (H_IS_STD__VECTOR) that affects whether the h component is allocated inside or outside the pinned buffer. Then, in message 3997-3998, the user delivers the key insight:
"The hunch about 10gb being low also sounds right, maybe we should have a program before benchmark which tries to actually allocate memory, and in benchmark runs also recover from OOMs and retry with 50% overhead a few times."

This is the spark that message 3999 runs with.

The Subject Message: A Plan Takes Shape

Message 3999 is the assistant's response to the user's suggestion. It is a long, multi-threaded reasoning trace that does several things simultaneously:

  1. Validates the user's intuition about the 10 GiB safety margin being insufficient.
  2. Breaks down the hidden memory overhead that the safety margin must cover.
  3. Designs two complementary solutions: a memory probe (memprobe) and an OOM recovery loop.
  4. Considers implementation constraints (no Python or C compiler in the runtime Docker image).
  5. Explores the SRS loading path to understand whether the budget accounting is fundamentally wrong.
  6. Decides on a pragmatic approach rather than continuing to chase speculative bugs. Let us examine each of these threads in detail.

The Hidden Memory Overhead: Why 10 GiB Was Never Enough

The assistant's reasoning begins with a concrete breakdown of the memory overhead that the 10 GiB safety margin was supposed to cover. This is one of the most valuable parts of the message—a quantitative analysis of where memory goes in a containerized CUDA application.

The assistant enumerates:

The SRS Loading Investigation: A Detour into the Abyss

A significant portion of the reasoning is devoted to understanding how the SRS (Structured Reference String) is loaded into memory. This investigation was triggered by the user's comment about SRS mode being "a bit weird," and it represents a fork in the road: should the team chase a potential budget accounting bug in SRS loading, or should they build the measurement and recovery infrastructure that would make such bugs survivable?

The assistant traces the SRS loading code through multiple layers:

  1. The Rust srs_manager.rs reserves budget equal to the file size (44 GiB) and calls SuprasealParameters::new().
  2. SuprasealParameters::new() calls SRS::try_new() in the supraseal-c2 crate.
  3. SRS::try_new() calls the C++ create_SRS() function via FFI.
  4. create_SRS() in groth16_srs.cuh mmaps the file, then calls cudaHostAlloc to allocate pinned memory, then copies data from the mmap to the pinned buffer. The assistant identifies two potential accounting mismatches: First mismatch: Transient double allocation. During SRS loading, both the mmap (44 GiB) and the cudaHostAlloc (unknown size total) exist simultaneously. The budget only accounts for one of these. If the mmap pages remain resident during the copy, the system could be using ~88 GiB against a 44 GiB budget entry. However, the assistant correctly notes that mmap-backed pages can be reclaimed under memory pressure since they're backed by disk, and the copy operation is short-lived. Second mismatch: Point format expansion. The assistant speculates that the on-disk format might use compressed BLS12-381 points (48 bytes for G1, 96 bytes for G2) while the in-memory representation uses uncompressed coordinates (96 bytes for G1, 192 bytes for G2). If true, loading a 44 GiB file could require 88 GiB of pinned memory. The assistant walks through the field arithmetic: each Fp element is 6 × 64-bit limbs = 48 bytes, so an affine G1 point (two coordinates) is 96 bytes, and an affine G2 point (four coordinates) is 192 bytes. But then the assistant corrects itself: "the Filecoin params files are actually stored uncompressed, so both the on-disk and in-memory sizes should be 96 bytes per point." The assistant also discovers a conditional compilation flag H_IS_STD__VECTOR that affects whether the h component is allocated inside or outside the pinned buffer. If h is a separate heap allocation, the pinned buffer is smaller than the total SRS memory footprint. This investigation is thorough but ultimately inconclusive. The assistant cannot determine the exact value of total without running the code or examining the C++ template parameters. More importantly, the assistant makes a critical strategic decision: "Rather than getting lost in the internals, I should focus on the user's practical suggestions: building a memprobe tool to measure actual memory usage and adding OOM recovery to the benchmark. These will give me empirical data about what's happening instead of speculating about mmap behavior or heap allocation quirks." This decision defines the entire message. The assistant chooses measurement over speculation, resilience over perfect accounting.

The Two-Pronged Solution: memprobe and OOM Recovery

The assistant designs two complementary mechanisms:

1. The Memory Probe (memprobe)

The memprobe is a small C program that:

2. OOM Recovery in benchmark.sh

The OOM recovery mechanism is a retry loop wrapped around the benchmark execution:

The Todo List: From Reasoning to Action

The message concludes with a structured todo list that translates the reasoning into concrete actions:

  1. Write memprobe.c — allocate 1 GiB chunks to find real usable memory
  2. Add memprobe to Dockerfile.cuzk (compile in builder, copy to runtime)
  3. Update benchmark.sh with OOM recovery + retry with increasing safety margin
  4. Update entrypoint.sh to use memprobe results This todo list is notable for what it doesn't include: there is no item for fixing the SRS budget accounting, no item for investigating the H_IS_STD__VECTOR flag, no item for measuring the exact total allocation size in create_SRS(). The assistant has made a deliberate choice to prioritize robustness over correctness—or rather, to achieve correctness through robustness.

Assumptions Made in This Message

The assistant makes several assumptions, some explicit and some implicit:

  1. The OOM kill is host-side, not GPU-side. This is based on the observation that GPU OOM returns a CUDA error code, while a process kill (broken pipe) indicates host OOM. This assumption is reasonable but not proven—a GPU driver crash could also kill the process.
  2. Exit code 137 reliably indicates OOM. In practice, exit code 137 is 128 + 9 (SIGKILL), which the OOM killer delivers. But any SIGKILL produces this exit code, including manual kill -9 or systemd unit timeouts.
  3. The safety margin should be a percentage of the budget, not a fixed value. The user suggested "50% overhead," which the assistant interprets as increasing the safety margin by 50% of its current value (from 10 GiB to 15 GiB) on each retry. This is a heuristic with no theoretical justification.
  4. Memory overhead scales linearly with allocation size. The assistant's overhead breakdown (page tables, arenas, stacks) assumes these are proportional to the number of threads and allocations. In reality, some overheads are fixed (kernel slab) and some are super-linear (page table fragmentation).
  5. The memprobe won't trigger the OOM killer. The assistant recognizes this risk and considers using mmap without MAP_POPULATE to avoid instant OOM. But even incremental allocation could trigger the OOM killer if the system is close to the limit.
  6. The runtime Docker image has no C compiler. This is verified knowledge from the conversation context—the assistant checked installed packages earlier.
  7. The SRS budget accounting is approximately correct. Despite the deep investigation into SRS loading, the assistant ultimately assumes the 44 GiB budget is close enough and that any discrepancy will be caught by the OOM recovery mechanism.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. CUDA pinned memory: Understanding that cudaHostAlloc allocates host memory that is page-locked for fast GPU transfers, and that this memory is not swappable.
  2. Linux cgroups: Understanding that containers have memory limits enforced by cgroup v1 or v2, and that /proc/meminfo reports host RAM, not container limits.
  3. BLS12-381 elliptic curve: Understanding point sizes (G1: 48 bytes compressed, 96 bytes uncompressed; G2: 96 bytes compressed, 192 bytes uncompressed) and Montgomery representation (6 × 64-bit limbs per field element).
  4. Filecoin proof structure: Understanding that PoRep proofs involve SRS parameters (~44 GiB), partition synthesis (~14 GiB per partition), and GPU proving phases.
  5. The cuzk pipeline: Understanding the flow from partition synthesis → GPU queue → GPU proving, and the role of the PI pacer in managing dispatch.
  6. mmap behavior: Understanding that MAP_PRIVATE file-backed pages are reclaimable under pressure, while MAP_POPULATE forces pages into memory immediately.
  7. glibc memory allocation: Understanding malloc arenas and thread-specific memory pools.
  8. Docker multi-stage builds: Understanding that the assistant plans to compile memprobe in the builder stage and copy the binary to the runtime stage.

Output Knowledge Created

This message creates several forms of knowledge:

  1. A quantitative model of memory overhead in containerized CUDA applications: page tables, malloc arenas, thread stacks, kernel slab, GPU driver overhead. This model is approximate but actionable.
  2. A design for a memory probe utility: A C program that empirically measures usable memory by attempting allocations up to the cgroup limit. The design includes considerations for avoiding OOM during the probe itself.
  3. A design for OOM recovery: A retry loop with exponential safety margin increases, integrated into the benchmark script.
  4. A strategic decision framework: The message demonstrates when to stop investigating and start building. The assistant explicitly chooses empirical measurement over further speculation, establishing a pattern for future debugging.
  5. A todo list that bridges reasoning and action: The structured todo list translates the analysis into concrete, sequential implementation steps.
  6. An understanding of the SRS loading path: While the assistant doesn't resolve the budget accounting question, the message documents the mmap + cudaHostAlloc double allocation and the H_IS_STD__VECTOR conditional, creating knowledge that could be useful for future optimization.

Mistakes and Incorrect Assumptions

Several aspects of the reasoning are incorrect or incomplete:

  1. The SRS point format speculation is resolved incorrectly. The assistant initially thinks compressed points could cause a 2× memory expansion, then corrects itself by asserting that Filecoin params files are stored uncompressed. This is actually debatable—the Filecoin params format uses a specific serialization that may not match either simple compressed or uncompressed categories. The assistant lacks definitive knowledge of the format.
  2. The overhead analysis omits significant categories. The assistant calculates page tables, arenas, and stacks, but omits: - File page cache: The SRS file, once mmap'd and read, may leave pages in the page cache even after munmap. - CUDA driver context memory: The CUDA driver allocates its own memory for context management, symbol tables, and JIT caches. - Container infrastructure: The container runtime itself (containerd, runc) consumes memory. - Double accounting: The 44 GiB SRS budget might not account for the mmap during loading.
  3. The OOM recovery design assumes the crash is always due to budget exhaustion. If the crash is caused by a memory leak, infinite loop, or GPU driver bug, reducing the budget won't help—the system will crash again at the same point. The retry loop could enter an infinite regression.
  4. The 50% safety margin increase is arbitrary. The user suggested "50% overhead," but the assistant doesn't question whether 50% is appropriate. On a 342 GiB machine, increasing the safety margin from 10 GiB to 15 GiB (50% increase) might not be enough; the actual overhead could be 20+ GiB.
  5. The memprobe design doesn't account for fragmentation. The probe allocates 1 GiB contiguous chunks, but the actual workload allocates many small buffers (thread stacks, arenas, individual CUDA allocations). Fragmentation could cause OOM at a lower total allocation than the probe measures.
  6. The assistant assumes the runtime image has no C compiler but doesn't verify the builder image has one. Docker multi-stage builds typically use a full compiler toolchain in the builder, but this is not guaranteed.

The Thinking Process: A Window into Practical Engineering

The most valuable aspect of message 3999 is the thinking process itself. The assistant's reasoning exhibits several characteristics of expert engineering:

Iterative refinement: The assistant starts with a simple idea (memory probe + OOM recovery) and progressively refines it through consideration of constraints (no compiler in runtime, MAP_POPULATE risk, timing issues).

Depth-first investigation with bounded commitment: The assistant goes deep into SRS loading internals but recognizes when the investigation is no longer productive and pivots to the practical solution. This is a critical skill—knowing when to stop digging.

Quantitative reasoning: The assistant doesn't just say "the safety margin is too low"; it calculates specific overheads (page tables: 192 MiB, arenas: 1.1 GiB, stacks: 144 MiB) and compares them to the 10 GiB margin.

Risk awareness: The assistant identifies specific risks (MAP_POPULATE triggering OOM, memprobe giving optimistic baseline) and designs around them.

Trade-off articulation: The assistant explicitly compares memprobe (simpler, but incomplete) with OOM recovery (more robust, but more complex) and decides to implement both.

System-level thinking: The assistant traces memory through multiple layers—kernel, glibc, CUDA driver, application code—and understands how they interact.

The Broader Significance

Message 3999 represents a turning point in the project's approach to reliability. Earlier messages focused on perfecting the memory budget accounting—making the theoretical model match reality. This message accepts that the theoretical model will never be perfect and builds resilience instead.

This shift from "prevent all failures" to "survive failures gracefully" is characteristic of mature systems engineering. The memprobe provides a better baseline; the OOM recovery provides a safety net for everything the baseline misses. Together, they form a defense-in-depth strategy for memory management.

The message also demonstrates the value of the human-AI collaboration pattern in this conversation. The user provides the high-level insight ("10 GiB is too low, let's measure and retry"), and the assistant fleshes out the details, considers edge cases, and produces an implementation plan. The assistant's deep investigation into SRS loading, while ultimately not leading to a fix, provides confidence that the simpler approach is sufficient—the assistant has verified that there isn't a trivial bug that would make the memprobe approach miss the real problem.

Conclusion

Message 3999 is a remarkable artifact of practical systems engineering. It begins with a user's intuition about an insufficient safety margin, proceeds through a deep technical investigation of SRS memory allocation, arrives at a strategic decision to prioritize measurement and resilience over perfect accounting, and produces a concrete implementation plan with two complementary mechanisms.

The message's reasoning is not flawless—it makes several assumptions that later turn out to be incomplete, and its overhead analysis omits important categories. But it is precisely this imperfection that makes the message instructive. The assistant demonstrates how to make decisions under uncertainty: quantify what you can, acknowledge what you cannot, build safety nets for the unknowns, and keep moving forward.

The memprobe and OOM recovery mechanisms that result from this message will go on to be deployed on real vast.ai instances, where they will successfully prevent crashes on memory-constrained nodes and provide empirical data about actual memory overhead. The 256 GiB instance that was running at 99% of its cgroup limit with zero headroom will be stabilized. The 342 GiB instance that crashed during GPU processing will complete its benchmarks.

But more than the specific solutions, the message embodies a philosophy: in complex systems, you cannot account for everything. Measure what you can, build resilience for what you cannot, and let empirical data guide your next move.