Chunk 8.0

This session focused on resolving the critical OOM (Out of Memory) failures that plagued low-RAM GPU instances during the cuzk PoRep proving benchmark. The initial blocking issue—the BC Canada instance (125GB RAM) being killed during warmup—was traced to two distinct problems: the daemon using too many partition workers during the initial PCE extraction, and the benchmark concurrency being too high for the available system memory. The Norway instance (1x RTX 4090, 500GB RAM) successfully completed its benchmark, but at 41.32 proofs/hour it fell below the 50 proofs/hour minimum, validating the need for multi-GPU setups and triggering a lifecycle bug where the manager failed to destroy underperforming instances. Two major code fixes were implemented and deployed. First, `benchmark.sh` was refactored to detect the absence of a PCE cache and start the daemon with `partition_workers=2` for the warmup proof, preventing the memory spike of simultaneous partition synthesis. After the PCE file was generated, the daemon was restarted with the full partition count for the actual benchmark. Second, `entrypoint.sh` was rewritten to dynamically scale benchmark concurrency based on available RAM and GPU count, replacing the hardcoded `concurrency=5` with a formula that reserves 100GB overhead and estimates 6GB per partition worker per proof. Additionally, a lifecycle bug was fixed in the vast-manager's `handleBenchDone` endpoint so that instances failing the benchmark are immediately destroyed via `vastai destroy`, preventing wasted expenditure. The session culminated in the successful deployment of two new instances running the hardened Docker image. A Czechia instance (2x RTX 3090, 251GB RAM) correctly auto-configured to `partition_workers=10` and `concurrency=2`, while a Belgium instance (2x A40, 2TB RAM) configured to `partition_workers=16` and `concurrency=6`. The Belgium instance completed its warmup without issue and began its benchmark, validating the OOM fix and the hardware-aware pipeline. The system is now in a robust state, with automated lifecycle management, dynamic hardware configuration, and proper handling of benchmark failures, ready for sustained proving operations.

The OOM Wars: A Distributed GPU Proving System's Journey from Crash to Self-Tuning Infrastructure 2742 words

Message Articles