Chunk 29.0

In this chunk, we finalized and committed all outstanding changes, including the new Docker scripts, memcheck utility, vast-manager improvements, and the safety margin increase. The core Rust `detect_system_memory()` function was rewritten to be cgroup-aware: it now reads cgroup v2 (`memory.max`) and v1 (`memory.limit_in_bytes`) limits and returns the minimum of host RAM and the cgroup constraint. This prevents the previous behavior where `/proc/meminfo` reported the host's full RAM inside Docker containers, causing massive over-allocation and OOM kills. We also fixed two bugs discovered during live testing: the GPU JSON parsing in `memcheck.sh` (which split GPU names on spaces, producing invalid JSON) and the pinning detection (which incorrectly flagged low `ulimit -l` as a failure, even though CUDA's `cudaHostAlloc` bypasses `RLIMIT_MEMLOCK` via the NVIDIA kernel driver). The entrypoint script was hardened to handle `jq` parse errors gracefully. The updated Docker image was built and pushed, then deployed and verified on two real vast.ai instances: a 961 GiB cgroup-limited machine (RTX 4090, 2003 GiB host) and a 342 GiB cgroup-limited machine (RTX 5090, 503 GiB host). On both, the memory budget was correctly derived from the cgroup limit rather than host RAM, and the system operated within safe memory bounds. The 342 GiB instance's benchmark failed with a transport error (likely a network issue), but the memory detection and budgeting worked as intended. The 961 GiB instance completed its benchmark successfully, achieving ~63.9 proofs/hour with no OOM kills. The cgroup-aware memory detection and the deployment pipeline are now validated on production-like vast.ai nodes.

From Host Blindness to Container Awareness: How Cgroup-Aware Memory Detection Rescued a GPU Proving Engine from OOM Kills 2103 words

Message Articles