Segment 29

In this sub-session, we completed and committed all outstanding changes, including rewriting `detect_system_memory()` to be cgroup-aware (reading cgroup v2/v1 limits) so that the memory budget is derived from the container's actual limit rather than host RAM, preventing OOM kills. We fixed two bugs in `memcheck.sh` (GPU JSON parsing and pinning detection) and hardened the entrypoint script. The updated Docker image was built, pushed, and deployed on two real vast.ai instances, where the cgroup-aware budgeting worked correctly. Subsequently, we investigated a crash on a 342 GiB instance and identified that the pinned memory pool operated outside the memory budget, combined with kernel/driver overhead and SRS loading spikes, making the 10 GiB safety margin insufficient. To address this, we developed a `memprobe` C utility that empirically measures available memory up to the cgroup limit, and added an OOM recovery loop to `benchmark.sh` that reduces the budget by 10% on OOM and retries up to three times. The `entrypoint.sh` was updated to run `memprobe` for a more conservative budget. Running `memprobe` on a live 256 GiB instance confirmed the machine was at 99% of its cgroup limit with zero headroom, validating the need for these adaptive measures.

Taming the OOM Killer: How Cgroup-Aware Memory Detection, Empirical Probing, and Adaptive Recovery Made a GPU Proving Engine Survive in the Wild 3950 words

Chunks

• From Host Blindness to Container Awareness: How Cgroup-Aware Memory Detection Rescued a GPU Proving Engine from OOM Kills
  2103 words
• Living on the Edge: How Empirical Measurement and OOM Recovery Tamed Memory Death in a GPU Proving Pipeline
  2368 words