Chunk 29.1

In this chunk, the investigation into the 342 GiB instance crash (OOM/broken pipe during GPU processing) narrowed down the root cause. The assistant and user identified that the CUDA pinned memory pool (`PinnedPool`) was operating outside the `MemoryBudget` system—pinned buffers returned to the pool after GPU work were never freed from actual RSS, creating a massive accounting discrepancy. Combined with kernel/driver overhead (glibc arenas, page tables, GPU driver allocations) and the transient SRS loading spike (simultaneous mmap + `cudaHostAlloc`), the 10 GiB safety margin was empirically insufficient for constrained instances. To solve this, a two-pronged strategy was implemented. A `memprobe` utility was written in C that allocates 1 GiB chunks via `mmap`/`memset` until it nears the cgroup limit, providing a data-driven safety margin that accounts for hidden kernel overhead. The `benchmark.sh` script was enhanced with an OOM recovery loop: if the cuzk daemon is killed (exit code 137), the budget is reduced by 10% and the benchmark is retried up to three times. The `entrypoint.sh` was updated to run `memprobe` after `memcheck` and use the more conservative budget estimate. The Docker image was rebuilt and pushed with these changes. The updated scripts and `memprobe` binary were deployed to the live 256 GiB instance (32897009) currently running a benchmark. Running `memprobe` on this instance provided stark empirical validation of the problem: the machine was operating at 99% of its cgroup limit (340 GiB / 342 GiB), with only 14 GiB of additional allocatable space and 6 GiB of kernel/driver overhead. The instance was surviving the benchmark but with zero headroom, confirming that the new adaptive safety margin and OOM recovery logic are essential for reliable operation on memory-constrained vast.ai nodes.

Living on the Edge: How Empirical Measurement and OOM Recovery Tamed Memory Death in a GPU Proving Pipeline 2368 words

Message Articles