Segment 8
The session began by tackling the critical OOM failures that plagued low-RAM GPU instances during the cuzk PoRep proving benchmark. The root cause was traced to the daemon using too many partition workers during initial PCE extraction and excessive benchmark concurrency. Two major fixes were implemented: benchmark.sh was refactored to detect missing PCE cache and start the daemon with partition_workers=2 for the warmup proof, and entrypoint.sh was rewritten to dynamically scale benchmark concurrency based on available RAM and GPU count. A lifecycle bug in vast-manager's handleBenchDone endpoint was also fixed to immediately destroy underperforming instances. After deploying the hardened image, new instances in Czechia (2x RTX 3090, 251GB RAM) and Belgium (2x A40, 2TB RAM) were created. However, both failed in distinct ways—Belgium hit the benchmark timeout, Czechia suffered a gRPC transport error. Tactical fixes increased the timeout to 45 minutes, added a post-restart warmup proof, and refined partition worker logic for ~256GB machines. Despite these, new instances continued to fail, revealing that real-world proving performance cannot be reliably predicted from hardware specs alone. This prompted a fundamental strategic shift toward a data-driven, experimental system to automatically discover optimal hardware. The implementation included a new host_perf database table to track benchmark results per host, an API to search Vast.ai offers filtered by GPU/RAM/price while overlaying known host performance, a deploy endpoint, and foundational UI code, setting the stage for a robust, self-tuning deployment pipeline.
From OOM to Oracle: How a Distributed GPU Proving System Learned to Learn