The Commit That Saved the Stack: How a 374-Line Change Tamed OOM Crashes on Memory-Constrained GPU Nodes

git add docker/cuzk/memprobe.c docker/cuzk/benchmark.sh docker/cuzk/entrypoint.sh Dockerfile.cuzk && git diff --cached --stat

At first glance, message [msg 4015] appears to be nothing more than a routine commit command—a developer staging four files and checking the diff stat before pushing. The numbers are unremarkable: 374 insertions, 113 deletions, four files touched. But this message is the culmination of an intense debugging session that traced a production OOM crash through multiple layers of abstraction, from CUDA pinned memory pools to Linux cgroup limits to glibc arena fragmentation. It represents the moment when a two-pronged defensive strategy—a custom memory-probing utility and an adaptive OOM recovery loop—was committed to the repository, marking the transition from investigation to deployment.

The Crisis That Preceded the Commit

To understand why this message was written, one must understand the crisis that preceded it. The CuZK proving engine had been deployed on vast.ai GPU instances, and while large instances (961 GiB) ran smoothly, a 342 GiB instance was crashing with OOM kills during GPU processing. The crash manifested as a broken pipe during GPU work—the daemon was silently murdered by the kernel's out-of-memory killer.

The initial investigation (documented in the preceding messages) had already ruled out the obvious suspects. The detect_system_memory() function had been rewritten to be cgroup-aware, correctly reading memory.max from cgroup v2 and memory.limit_in_bytes from cgroup v1, rather than naively trusting /proc/meminfo which reports the host's full RAM inside Docker containers. The safety margin had been increased from 5 GiB to 10 GiB. Yet the 342 GiB instance still crashed.

The root cause, as the assistant and user jointly identified, was subtle and multi-layered. The CUDA pinned memory pool (PinnedPool) was operating outside the MemoryBudget system entirely. When pinned buffers were returned to the pool after GPU work completed, they were never freed from actual RSS (Resident Set Size). The pool accumulated buffers, the RSS grew, and the kernel's OOM killer eventually stepped in. Compounding this were kernel and driver overheads that the simple safety margin model didn't account for: glibc memory arena fragmentation, page tables, GPU driver allocations, and a transient spike during simultaneous SRS (Structured Reference String) loading that combined mmap with cudaHostAlloc. The 10 GiB safety margin was empirically insufficient for constrained instances.

What the Commit Actually Contains

The git diff --cached --stat output in message [msg 4015] reveals the precise scope of the response:

docker/cuzk/memprobe.c (133 insertions): A new C utility that allocates 1 GiB chunks via mmap/memset until it approaches the cgroup memory limit. This provides a data-driven safety margin that accounts for all hidden kernel and driver overhead—glibc arenas, page tables, GPU driver memory, and any other allocations that the kernel forces into RSS but that aren't visible to the application's memory tracking.
docker/cuzk/benchmark.sh (319 insertions, significant deletions): The benchmark script was restructured with an OOM recovery loop. If the cuzk daemon is killed with exit code 137 (the signal sent by the OOM killer), the budget is reduced by 10% and the benchmark is retried, up to three times. This transforms a fatal crash into a graceful degradation path.
docker/cuzk/entrypoint.sh (29 insertions): The container entrypoint was updated to run memprobe after memcheck and use its empirically derived budget estimate, which is more conservative than the theoretical calculation.
Dockerfile.cuzk (6 insertions): The Docker build was updated to compile memprobe.c during the build phase and copy the binary into the runtime image. The diff stat tells a story of surgical precision: 374 lines of new code and 113 lines of deleted or modified code across four files, each addressing a different layer of the OOM problem.

The Thinking Process Behind the Commit

The reasoning visible in this message is the reasoning of a developer who has reached a natural commit point after an intensive debugging session. The assistant's thought process, as revealed in the preceding messages, followed a clear arc:

Hypothesis formation: The OOM crash on the 342 GiB instance was not a simple configuration error but a systemic issue involving the pinned memory pool, kernel overhead, and the safety margin calculation.
Root cause analysis: The assistant traced the problem to three interacting factors: the PinnedPool not accounting to MemoryBudget, kernel/driver overhead invisible to /proc/meminfo, and transient SRS loading spikes.
Strategy design: Rather than attempting a complex fix to the PinnedPool accounting (which would require deep changes to the Rust CUDA runtime), the assistant chose a defensive approach: measure empirically and recover gracefully.
Implementation: The memprobe utility was written in C for maximum reliability—no runtime dependencies, no Go garbage collector interference, just raw mmap and memset loops. The benchmark script was restructured with a retry loop. The entrypoint was updated to integrate both.
Commit readiness: Message [msg 4015] represents the moment when all four pieces were ready to be committed together as a coherent unit. The git add command stages them, the git diff --cached --stat provides a final review before the commit is sealed.

Assumptions Made and Their Validity

Several assumptions underpin this commit:

Assumption 1: The OOM crashes are caused by memory pressure, not by bugs in the proving logic. This assumption was validated by the crash signature (exit code 137, the OOM killer signal) and by the fact that larger instances with more memory ran successfully.

Assumption 2: A 10% budget reduction per retry is a reasonable degradation strategy. This is a heuristic—too aggressive a reduction might waste GPU capacity, too conservative might not prevent the next OOM. The assumption is that the OOM is caused by a modest overshoot, not a catastrophic leak.

Assumption 3: mmap/memset allocation closely approximates the memory pressure that the actual CU workload will produce. This is a simplifying assumption—real GPU proving involves CUDA allocations, pinned memory, and file-backed mappings that behave differently from anonymous mmap pages. The memprobe utility measures one specific kind of memory pressure and extrapolates to the general case.

Assumption 4: The three-retry limit is sufficient. If the budget needs more than three 10% reductions to stabilize, the benchmark will still fail. The assumption is that three retries cover the likely range of overshoot.

Input Knowledge Required

To understand this message fully, one needs knowledge of:

Linux OOM behavior: The kernel's out-of-memory killer selects a process to kill when the system runs out of memory. Exit code 137 (128 + 9, where 9 is SIGKILL) is the signature of an OOM kill.
cgroup memory limits: Docker containers run under cgroup constraints. /proc/meminfo reports host RAM, not the container's limit, which is why cgroup-aware detection was necessary.
CUDA pinned memory: cudaHostAlloc allocates page-locked (pinned) host memory for fast GPU transfers. Pinned memory is not subject to RLIMIT_MEMLOCK on modern NVIDIA drivers, but it still counts toward RSS.
glibc memory allocation: glibc uses memory arenas that can fragment and retain freed memory, increasing RSS beyond what the application believes it has allocated.
The CuZK proving pipeline: The benchmark measures proof generation throughput. The daemon's memory usage pattern includes SRS loading (large mmap files), GPU work buffers (pinned memory), and synthesis concurrency.

Output Knowledge Created

This message creates several forms of knowledge:

A reproducible safety margin methodology: The memprobe utility provides a portable, language-independent way to measure the true usable memory of a cgroup-constrained container, accounting for all kernel overhead.
An empirical validation: When deployed to the live 256 GiB instance, memprobe revealed that the machine was operating at 99% of its cgroup limit (340 GiB / 342 GiB), with only 14 GiB of additional allocatable space and 6 GiB of kernel/driver overhead. This confirmed that the new adaptive safety margin was essential.
A recovery pattern: The OOM recovery loop in benchmark.sh establishes a pattern that can be applied to other services: instead of crashing fatally on OOM, detect the signal, reduce resource consumption, and retry.
A deployment artifact: The committed code becomes part of the Docker image, deployed to all vast.ai instances. The next time a memory-constrained node runs a benchmark, it will benefit from both the empirical safety margin and the recovery loop.

Mistakes and Limitations

The most significant limitation of this approach is that it addresses the symptom (OOM crashes) rather than the root cause (the PinnedPool operating outside MemoryBudget). The commit does not fix the accounting issue in the Rust CUDA runtime—it builds a safety net around it. This is a pragmatic choice: fixing the PinnedPool accounting would require deep changes to the proving engine's memory management, with risk of introducing new bugs. The defensive approach gets the system stable faster.

A secondary limitation is that memprobe measures anonymous mmap memory, not CUDA pinned memory. Pinned memory may behave differently under pressure—the GPU driver might allocate additional kernel-side structures that mmap doesn't capture. The empirical validation on the 256 GiB instance showed 6 GiB of overhead, but this number may vary across GPU models, driver versions, and kernel configurations.

The 10% reduction step is also a heuristic. If the actual overshoot is 50%, three retries won't be enough. The assumption is that the OOM is caused by a modest overshoot, but this hasn't been proven across all instance sizes.

Why This Message Matters

Message [msg 4015] is the inflection point where investigation becomes deployment. The preceding messages were about understanding the problem—reading logs, examining code, forming hypotheses. This message is about locking in the solution. The git add and git diff --cached --stat commands are the developer's ritual of final review before committing to history.

For the reader following the conversation, this message provides a clean summary of what was accomplished: four files, 374 new lines, 113 deleted lines, spanning a C utility, two shell scripts, and a Dockerfile. The diff stat is the scorecard of a successful debugging session.

The commit that followed this message would be pushed to the repository, built into a Docker image, and deployed to production vast.ai instances. On those instances, the memprobe utility would run at container startup, measuring the true usable memory with all kernel overhead accounted for. The benchmark script would wrap the daemon in a recovery loop, ready to catch any OOM kill and adapt. The 342 GiB instance that had been crashing would now have a fighting chance.

In the broader narrative of the CuZK proving engine's development, this commit represents the moment when the system learned to survive in the wild—on memory-constrained GPU instances where every gigabyte counts and the kernel is always watching.