Living on the Edge: How Empirical Measurement and OOM Recovery Tamed Memory Death in a GPU Proving Pipeline

Introduction

In the life of a production system, there are moments when everything hangs in the balance—when a single allocation, a single page fault, a single kernel data structure expansion can push a process from "surviving" to "dead." For the CuZK GPU-accelerated proof generation engine, that moment arrived repeatedly on memory-constrained vast.ai instances. The daemon would be running, processing proofs, and then—silently—the kernel's out-of-memory killer would strike. A broken pipe. A dead process. A benchmark that produced zero proofs before being terminated.

This article synthesizes the work captured in Segment 29, Chunk 1 of an opencode coding session—a sustained debugging effort that traced these OOM crashes from their surface symptoms down to a fundamental accounting mismatch in a CUDA pinned memory pool, and then built a two-pronged defense of empirical measurement and adaptive recovery that transformed the system from fragile to resilient. The chunk spans the moment of diagnosis through deployment and empirical validation on live production instances.

The Phantom 108 GiB: Tracing the Root Cause

The investigation began with a crash on a 342 GiB cgroup-limited instance (an RTX PRO 4000 machine on vast.ai). The cuzk daemon had been running a benchmark when it died with a "broken pipe" error during GPU processing—the classic signature of an OOM kill. What made this crash particularly frustrating was that the memory budget should have been sufficient. The cgroup-aware memory detection had correctly identified the container limit as 342 GiB, applied a 10 GiB safety margin, and set the budget to 331 GiB. The system reported 266 GiB in use during the crash—well within the 331 GiB budget. So why did it crash?

The assistant's initial hypothesis, developed in [msg 3984], pointed to the CUDA pinned memory pool (PinnedPool). The reasoning was detailed and compelling: when a partition's synthesis completed and the a/b/c vectors were released, the MemoryBudget system freed their 13 GiB reservation. But the pinned buffers themselves were not freed—they were returned to the PinnedPool for reuse. The budget thought the memory was available; the kernel knew it was still allocated. With eight partitions in the GPU queue, this created a staggering 108 GiB of hidden, untracked memory.

The user corrected this assumption in [msg 3985] with a single, devastating sentence: "We track memory/pinned memory in memory manager." This twelve-word correction dismantled the entire hypothesis and forced a complete re-examination. If pinned memory was tracked, the accounting discrepancy the assistant had spent paragraphs constructing did not exist. The OOM must have a different cause.

This pivot—captured in [msg 3986]—is a masterclass in structured debugging under uncertainty. The assistant did not discard its analysis; it systematically re-traced the entire pipeline flow from first principles, enumerating each step: budget acquisition, vector allocation, GPU queue entry, budget release, pool return. The user's follow-up hint in [msg 3987]—"Maybe in SRS-mode it a bit weird"—redirected the investigation toward the Structured Reference String (SRS) loading path, where a transient double-allocation (simultaneous mmap of the 44 GiB file and cudaHostAlloc of the in-memory representation) created a temporary 88 GiB peak against a budget that only accounted for 44 GiB.

From Analysis to Action: The Memprobe and OOM Recovery

The investigation into SRS loading internals was thorough but ultimately inconclusive. The assistant traced the code through srs_manager.rs, supraseal_params.rs, and into the C++ groth16_srs.cuh file, discovering a conditional compilation flag (H_IS_STD__VECTOR) that affected allocation paths and speculating about point format size mismatches. But rather than getting lost in the internals, the assistant made a critical strategic decision, articulated in [msg 3999]: "Rather than getting lost in the internals, I should focus on the user's practical suggestions: building a memprobe tool to measure actual memory usage and adding OOM recovery to the benchmark."

This decision defines the entire chunk. The assistant chose measurement over speculation, resilience over perfect accounting.

The user's message at [msg 3998] had crystallized the way forward: "The hunch about 10gb being low also sounds right, maybe we should have a program before benchmark which tries to actually allocate memory, and in benchmark runs also recover from OOMs and retry with 50% overhead a few times." From this seed, the assistant designed two complementary mechanisms.

The Memory Probe (memprobe)

The memprobe utility, written as a 133-line C program in [msg 4000], is elegant in its simplicity. Rather than trying to parse kernel memory accounting structures or interrogate /proc filesystem entries, it simply probes the available memory by allocating 1 GiB chunks via mmap with MAP_ANONYMOUS and touching each page via memset until it nears the cgroup limit. It stops 2 GiB before the limit to avoid triggering the OOM killer on itself.

The assistant considered several design alternatives. A bash-based approach using tmpfs was rejected because it would test tmpfs limits, not RSS. The stress and stress-ng tools were absent from the runtime Docker image. Python was unavailable. C, compiled in the Docker builder stage and copied to the runtime image, was the only viable option.

The key limitation was explicitly acknowledged: the probe runs before the daemon starts, measuring only the empty container's memory capacity. Once the daemon is running, additional overhead from page tables for CUDA pinned memory, GPU driver allocations, thread stacks, and glibc arenas would shrink the available headroom. The memprobe provides an optimistic baseline—valuable but not sufficient on its own.

The OOM Recovery Loop

The OOM recovery mechanism, implemented in benchmark.sh ([msg 4009]), is a retry loop wrapped around the benchmark execution. When the cuzk daemon is killed with exit code 137 (128 + SIGKILL, the signal delivered by the OOM killer), the script detects this, reduces the memory budget by 10%, waits 30 seconds for kernel memory reclaim, and retries the benchmark up to three times.

The assistant recognized that this recovery loop is "actually the more reliable safety mechanism" because it tests under real workload conditions. The memprobe provides a baseline; the OOM recovery provides a safety net for everything the baseline misses. Together, they form a defense-in-depth strategy for memory management.

Integration and Deployment

The integration of these two mechanisms required changes across four files, captured in the commit at [msg 4015] and [msg 4016]. The Dockerfile.cuzk was updated to compile memprobe.c in the builder stage and copy the binary to the runtime image (6 insertions). The entrypoint.sh was updated to run memprobe after memcheck and use the more conservative of the two budget estimates (29 insertions). The benchmark.sh was restructured with the OOM recovery loop (319 insertions, significant deletions). And memprobe.c itself was added as a new file (133 insertions).

The commit message, quoted in [msg 4016], reads:

cuzk: add memprobe utility and benchmark OOM recovery

>

- memprobe.c: tiny C program that allocates 1 GiB chunks via mmap+memset to find actual usable memory. Accounts for kernel overhead (page tables, slab, driver) that consumes cgroup budget invisibly. Stops 2 GiB before cgroup limit. - benchmark.sh: wrap benchmark in OOM recovery loop. If daemon is OOM-killed (exit 137), reduce budget by 10% and retry up to 3 times. Waits 30s between retries for kernel memory reclaim. - entrypoint.sh: run memprobe after memcheck, use the more conservative of the two budget estimates. - Dockerfile.cuzk: compile memprobe in builder stage, copy to runtime.

The Docker image was built and pushed to the registry in [msg 4017] and [msg 4018]. But the team faced a practical challenge: the 256 GiB instance (32897009) was already running a benchmark with the old scripts. Restarting the container would interrupt the benchmark and waste hours of parameter download time.

The solution, executed in [msg 4020] and [msg 4021], was a surgical hot-patch. The assistant extracted the memprobe binary from the newly built Docker image using docker create and docker cp, then deployed it to the running instance via scp. This avoided restarting the container while still getting the diagnostic tool onto the live system.

The 99% Machine: Empirical Validation

The moment of truth arrived in [msg 4022]. The assistant deployed memprobe to the live 256 GiB instance and captured the output:

{
  "cgroup_limit_bytes": 367003697152,
  "usage_before_bytes": 342817820672,
  "allocated_bytes": 15032385536,
  "allocated_gib": 14,
  "usage_peak_bytes": 364346327040,
  "usage_after_bytes": 350478913536,
  "overhead_bytes": 6496120832,
  "overhead_gib": 6.0,
  "probe_chunks": 14
}

The data was stark. The instance was operating at 99% of its cgroup limit (340 GiB / 342 GiB) before the probe even started. Only 14 GiB of additional memory could be allocated. Six gigabytes of kernel/driver overhead—page tables, slab allocator caches, GPU driver allocations, glibc arena metadata—were consuming cgroup budget invisibly. The 10 GiB safety margin was being eaten alive by overhead that standard tools could not detect.

The assistant's commentary in [msg 4023] captured the tension: "The machine is running at 99% of its cgroup limit during benchmark." This was not a system that had failed—it was a system that was surviving, but on a knife's edge.

The follow-up check in [msg 4024] confirmed the daemon was still alive, with RSS at 340 GiB against a 342 GiB limit and a budget of 331 GiB. The daemon log showed a clean initialization: "memory budget initialized total_budget_gib=331," "CUDA pinned memory pool initialized," "starting pipeline." No errors, no warnings. The system was running, but the 9 GiB gap between the budget (331 GiB) and actual RSS (340 GiB) was the PinnedPool accounting discrepancy in action—visible, measurable, and dangerous.

In [msg 4025], the assistant drilled deeper, querying both the cgroup memory controller and the daemon's internal status API. The cgroup reported memory.current at 320.6 GiB against a memory.max of 342 GiB. The daemon's status API showed used_bytes at 321.8 GiB with only 9.2 GiB available. Both GPU workers were in "proving" state—the pipeline was fully saturated. The assistant's assessment: "The machine is currently surviving! RSS is 340/342 GiB which is extremely tight but hasn't OOM'd yet."

Assumptions, Limitations, and the Path Forward

The memprobe and OOM recovery approach rests on several assumptions that deserve examination.

The memprobe measures anonymous mmap, not CUDA pinned memory. The probe allocates via mmap with MAP_ANONYMOUS, which is a reasonable proxy for general memory pressure but does not capture the specific overhead of CUDA pinned allocations or GPU driver state. The 6 GiB overhead figure is a lower bound—the real overhead during GPU proving could be higher.

Exit code 137 reliably indicates OOM. In practice, exit code 137 is 128 + 9 (SIGKILL), which the OOM killer delivers. But any SIGKILL produces this exit code, including manual kill -9 or systemd unit timeouts. The recovery loop could retry on non-OOM failures, potentially masking other bugs.

The 10% budget reduction per retry is a heuristic. There is no theoretical justification for 10% rather than 5% or 20%. On a 342 GiB machine, three 10% reductions give a maximum adjustment of ~27%. This is gentle enough to avoid collapsing performance on the first retry but might be insufficient if the overshoot is large.

The 30-second wait between retries is a guess. Kernel memory reclaim after an OOM kill can complete in seconds on some systems and take minutes on others, especially with large pinned allocations. A fixed 30-second wait is a compromise.

The approach addresses symptoms, not the root cause. The memprobe and OOM recovery loop do not fix the PinnedPool accounting issue. They build a safety net around it. The pool still operates outside the budget system; the RSS still diverges from the budget. A future workload change could invalidate the memprobe's measurements.

Conclusion

The work captured in this chunk represents a fundamental shift in how the CuZK proving engine approaches memory management. The team moved from a model of perfect accounting—where the budget should exactly match reality—to a model of empirical measurement and adaptive resilience. The memprobe provides data where there was only guesswork. The OOM recovery loop provides a safety net where there was only fatal failure.

The empirical validation on the live 256 GiB instance confirmed the severity of the problem: 6 GiB of invisible kernel overhead, 99% cgroup utilization, zero headroom. The system was surviving, but barely. The memprobe and OOM recovery loop gave it a fighting chance.

In the broader narrative of the CuZK project, this chunk represents the moment when the system learned to survive in the wild—on memory-constrained GPU instances where every gigabyte counts and the kernel is always watching. The commit that codified this survival ([msg 4016]) is not just a set of code changes; it is the crystallized artifact of a debugging odyssey, encoding hard-won knowledge about kernel memory accounting, cgroup limits, and the gap between what a budget thinks is allocated and what the kernel knows is real.## References

[1] "The Phantom 108 GiB: Tracing a Memory Accounting Bug in a CUDA Pinned Pool" — Analysis of the PinnedPool accounting mismatch at [msg 3984].

[2] "The Pivot: How a Single Line Redirected an OOM Investigation" — The user's correction at [msg 3985] that dismantled the initial hypothesis.

[3] "Debugging Memory Accounting in a GPU Proving System: Tracing the OOM Root Cause Through Budget Tracking and Pinned Pool Dynamics" — The re-examination at [msg 3986].

[16] "The Memory Probe and OOM Recovery: A Turning Point in Production Reliability" — The strategic decision at [msg 3999] to build memprobe and OOM recovery.

[17] "The Memprobe: A Single File That Captured a Debugging Odyssey" — The writing of memprobe.c at [msg 4000].

[32] "The Commit That Saved the Stack: How a 374-Line Change Tamed OOM Crashes on Memory-Constrained GPU Nodes" — The commit at [msg 4015].

[33] "The Commit That Codified Survival: How a Git Message Captured the Battle Against Silent Memory Death" — The commit message at [msg 4016].

[39] "Empirical Validation: The Memprobe Utility Exposes Hidden Kernel Overhead on a Live Vast.ai Instance" — The memprobe deployment at [msg 4022].

[40] "The 99% Machine: A Live Probe into Memory Pressure on GPU Instances" — The analysis at [msg 4023].

[41] "Living on the Edge: Diagnosing Memory Pressure at 99% Cgroup Capacity" — The status check at [msg 4024].

[42] "Living on the Edge: A Real-Time Memory Pressure Snapshot in CuZK's GPU Proving Pipeline" — The deep dive at [msg 4025].