The OOM Wars: A Distributed GPU Proving System's Journey from Crash to Self-Tuning Infrastructure

Introduction

In the span of roughly 120 messages spanning a single coding session, an AI assistant and its human counterpart transformed a broken, crash-prone GPU proving pipeline into a self-adapting, hardware-aware deployment system. The journey—documented across messages 1028 through 1151 of an opencode conversation—reads like a case study in distributed systems debugging: OOM kills, silent API failures, lifecycle bugs, timeout miscalculations, and a fundamental strategic pivot from hardcoded thresholds to data-driven discovery.

This article synthesizes that journey. It traces the arc from the first comprehensive status document through the OOM diagnosis, the two-phase warmup fix, the dynamic concurrency rewrite, the lifecycle bug correction, and the eventual deployment of hardened instances that validated the entire effort. Along the way, it examines the reasoning patterns, assumptions, and engineering discipline that turned a cascade of failures into a robust, self-tuning infrastructure.

The Meta-Status Document: Taking Stock Before the Storm

The chunk opens with an extraordinary artifact: message 1028, a self-initiated, 2,000-word status document written by the assistant during a period when the user had instructed it to operate non-interactively [1]. The assistant had been told "do not stop, do not ask questions until instances run correctly" ([msg 1009]), and it responded by producing a comprehensive project overview covering goals, technical instructions, ten discoveries about Vast.ai and cuzk behavior, a status of completed and in-progress work, and a file inventory.

This document is remarkable for what it reveals about the assistant's cognitive architecture. It wasn't following a template or responding to a prompt—it chose to pause its active work and produce a structured knowledge artifact because it understood that the human would eventually return and need to understand what had happened. The message organizes information into five sections—Goal, Instructions, Discoveries, Accomplished, Relevant Files—mirroring the way experienced engineers document complex projects [1].

The discoveries section is particularly valuable. It captures ten hard-won lessons about Vast.ai platform behavior, bash scripting pitfalls, and cuzk memory characteristics. Discovery #5 identifies the critical blocking issue: "OOM on first benchmark with low RAM—PCE extraction with all partition workers simultaneously growing organically causes OOM on machines with 125GB RAM" [1]. This single observation would drive the next several hours of debugging.

But the document also reveals the assistant's blind spots. It assumes the OOM root cause is understood (too many partition workers during PCE extraction) without considering alternative explanations like memory leaks or kernel driver issues. It doesn't discuss fallback plans if the warmup fix fails. And it doesn't question the economic viability of single-GPU instances—an assumption that would later prove costly [1].

Diagnosis Complete: The Turning Point

After the meta-status document, the assistant spends several rounds probing instance states, discovering a routing bug in the vast-manager API (the /api/instances endpoint returned 404 while the actual endpoint was /api/dashboard), and working around it to assemble a clear picture [6][7][8][9][10][11][12][13]. By message 1041, the assistant can declare: "Now I have the full picture" [14].

The picture is stark. The BC Canada instance—a dual RTX 3090 machine with 125GB RAM—has been killed during benchmark warmup, returning a bench_rate of 0. The Norway instance—a single RTX 4090 with 500GB RAM—is alive and running its benchmark with 12 proofs at concurrency 5 [14].

Message 1041 is the diagnostic pivot. The assistant transitions from investigation to remediation, reading benchmark.sh to understand the exact mechanism of the OOM failure. This is disciplined engineering: diagnose from evidence, not from assumption. The assistant doesn't speculate about what might have caused the crash—it reads the code that orchestrates the benchmark to find the root cause [14].

The assistant's reasoning at this point reveals several implicit assumptions. It labels the BC Canada instance as "OOM failure" without having seen an OOM killer message in the logs—the diagnosis is inferred from the combination of "killed" state, bench_rate 0, and the machine's 125GB RAM constraint. It assumes the fix lies in benchmark.sh rather than in the cuzk daemon itself. And it assumes that concurrency=5 is a reasonable default for the Norway instance—an assumption that would later prove incorrect when the benchmark achieved only 41.3 proofs/hour, below the 50 proofs/hour minimum [14][49].

The Warmup Gambit: A Two-Phase Daemon Restart

The root cause analysis in messages 1042-1059 reveals a specific failure mechanism. The cuzk daemon, when started for the first time, must perform PCE (Pre-Compiled Constraint Evaluator) extraction—a memory-intensive initialization that generates a ~26GB cache file. During this extraction, the daemon spawns all partition workers simultaneously, each allocating significant memory for circuit synthesis. On a 125GB machine with 10 partition workers, this simultaneous allocation exceeds available RAM and triggers the OOM killer [15][17].

The fix, implemented in benchmark.sh, is elegant in its simplicity: detect the absence of a PCE cache file, start the daemon with partition_workers=2 for the warmup proof (which generates the cache), and then restart the daemon with the full partition count for the actual benchmark [17][19][22]. This two-phase approach prevents the memory spike of simultaneous partition synthesis while still allowing full parallelism for the production benchmark.

The implementation required careful attention to bash scripting details. The assistant discovered a subtle bug where function ordering in bash caused the start_daemon_with_workers function to be called before it was defined [20][21][22]. A syntax check (bash -n) caught this issue before deployment [23][24]. The assistant also realized that the daemon restart was redundant—the warmup proof could simply run with reduced workers and then the benchmark script could proceed with the full configuration without restarting [26][30]. This optimization was applied, removing an unnecessary restart cycle.

The verification of this fix was methodical. The assistant SSH'd into the running Norway instance, checked that the PCE cache existed, confirmed the daemon was running with the correct partition workers, and monitored memory usage [27][28][29][31][32]. Each verification step built confidence that the fix was correct before proceeding to deployment.

The 41.3 Proofs/Hour Threshold: When Success Reveals Deeper Problems

While the OOM fix was being developed, the Norway instance continued its benchmark. At message 1076, the results arrived: 12 proofs completed, 0 failed, throughput of 41.3 proofs/hour [49]. This was a success in one sense—the benchmark completed without crashing—but a failure in another: 41.3 proofs/hour fell below the 50 proofs/hour minimum threshold that defined economic viability.

The assistant's reaction is telling. It doesn't celebrate the successful completion or immediately jump to optimizing the proving pipeline. Instead, it checks the meta-system: does the automated lifecycle management correctly handle a below-threshold result? The assistant queries the entrypoint log to verify that the bench-done reporting path includes a threshold check [49].

This prioritization—verify the automation before optimizing the performance—is the hallmark of mature systems engineering. The assistant understands that if the lifecycle management is broken, optimization work is moot because underperforming instances won't be automatically decommissioned.

The Norway result also triggers a lifecycle bug discovery. The dashboard shows the instance in "killed" state, but vastai show instances reveals it's still running [50][51]. The manager's internal state had transitioned to "killed" but the actual destruction command (vastai destroy) hadn't been executed. This disconnect between the manager's internal state and the actual Vast.ai state is a critical gap [52][53][54][55][56][57][58][59][60][61][62][63][64][65].

The Lifecycle Bug: When "Killed" Doesn't Mean Destroyed

The lifecycle bug in handleBenchDone is one of the most consequential discoveries in the chunk. The assistant traces through the vast-manager source code and finds that when an instance fails the benchmark, the endpoint marks it as "killed" in the database but doesn't call vastai destroy to actually terminate the instance on Vast.ai [52][53][54][55]. This means underperforming instances continue running and accruing charges indefinitely.

The fix is a single line: adding vastai destroy to the kill handler [62][64][65][66]. But the implications are broader. The assistant discovers that the handleBenchDone endpoint was never being called in some failure modes—instances that crashed during benchmark never reported their results, so the lifecycle manager never knew to destroy them [69][70]. This leads to a more robust design where the manager actively monitors instance state rather than waiting for callbacks.

The fix is deployed to the vast-manager service, and the assistant verifies that the kill handler now works correctly [67][68][69][70]. This lifecycle fix is essential for the economic viability of the system—without it, failed instances would bleed money indefinitely.

The Moment the Benchmarks All Failed

With the OOM fix deployed and the lifecycle bug corrected, the assistant creates new instances to test the pipeline. A US instance with 2x RTX 3090 and 376GB RAM begins its benchmark. The warmup succeeds—the PCE cache is generated with partition_workers=2 without OOM. The daemon restarts with full partition_workers=10. The batch benchmark begins. Everything looks promising [80][81][82][83][84][85][86][87].

Then, at message 1121, the assistant discovers that every instance has been killed [94]. The dashboard shows three instances in "killed" state:

The Pivot to Hardware-Aware Configuration

The cascade of failures forces a fundamental rethinking. At message 1124, the assistant articulates the new approach: "I need to: 1. Detect GPU count, 2. Scale benchmark concurrency to GPUs, 3. Also consider partition workers vs available RAM more carefully" [97].

This is the transition from hardcoded configuration to hardware-aware auto-tuning. The assistant recognizes that BENCH_CONCURRENCY=5 was a brittle assumption that didn't account for the diversity of hardware on Vast.ai. A machine with 125GB RAM and a machine with 2TB RAM should not use the same concurrency [97][98].

The implementation replaces the hardcoded value with a formula:

The Per-Proof Memory Correction

The first deployment of the dynamic concurrency formula reveals a critical flaw. The Czechia instance (251GB RAM, 2 GPUs) reports concurrency=5 with partition_workers=10. The assistant does the math: available=151GB, per_proof=30GB (10×3GB), 151/30=5, gpu_cap=6 → 5. But this leaves only 1GB of margin—any memory spike would trigger an OOM [113][114].

Tracing back to the US instance OOM, the assistant reconstructs the likely memory consumption: if each proof needed 50-60GB (10 workers × 5-6GB), then 5 concurrent proofs would consume 250-300GB plus 100GB overhead = 350-400GB, right at the edge of 376GB. This explains the OOM [114].

The fix is to increase the per-worker multiplier from 3GB to 6GB. This changes the Czechia concurrency from 5 to 2—a much safer configuration. The assistant recalculates for all target machines [114][115]:

Deploying Hardened Instances

With the corrected formula, the assistant rebuilds the Docker image, destroys the existing instances, and creates two new ones: a Czechia instance (2x RTX 3090, 251GB RAM, $0.282/hr) and a Belgium instance (2x A40, 1TB RAM, $0.574/hr) [118][119][120][121].

The deployment is not without hiccups. The assistant assumes the SSH port is derived from the instance ID (e.g., port 34145 for instance 32714145), but Vast.ai assigns ports independently—the actual port is 34144 [120]. The Belgium instance's SSH isn't available immediately, requiring a diagnostic pivot to query the Vast.ai API for connection details [120].

But the instances register successfully with the manager. The Czechia instance detects 2 GPUs and 251GB RAM, selects partition_workers=10 and concurrency=2. The Belgium instance detects 2 A40 GPUs and 2003GB RAM, selects partition_workers=16 and concurrency=6 [121][122][123][124].

The validation is methodical. The assistant checks the entrypoint logs via SSH, grepping for the hardware detection and configuration decisions. The log output confirms the auto-configuration is working correctly [122]. The instances proceed through parameter download, warmup, and benchmark without OOM crashes.

The Strategic Pivot: From Hardcoded Thresholds to Data-Driven Discovery

The chunk concludes with the hardened instances running their benchmarks, but the story doesn't end there. The analyzer summary for chunk 1 reveals that both instances would eventually fail in new ways—Belgium hit the 20-minute benchmark timeout, Czechia suffered a gRPC transport error [chunk 1 summary]. These failures would drive further iteration: increasing the timeout to 45 minutes, adding a post-restart warmup proof, and ultimately shifting to a data-driven experimental system for automatic hardware discovery.

This strategic pivot is the most significant outcome of the entire chunk. The assistant learned that real-world proving performance cannot be reliably predicted from hardware specs alone. A 2x A40 with 2TB RAM underperformed a single RTX 4090. The only way to know which hardware works is to test it empirically. This insight leads to the creation of a host_perf database table, an offer search API with known host performance overlay, and a deploy endpoint—the foundations of a self-tuning deployment pipeline.

Themes and Lessons

Several themes emerge from this chunk:

1. The primacy of verification. The assistant never assumes a fix works until it sees evidence. Every code change is syntax-checked, every deployment is verified via SSH or API, every configuration decision is validated against real hardware. This discipline separates robust systems from fragile ones.

2. The danger of hardcoded assumptions. The hardcoded BENCH_CONCURRENCY=5 caused OOMs across diverse hardware. The hardcoded 50 proofs/hour threshold killed instances that were performing well for their hardware class. The hardcoded 20-minute timeout killed instances that needed more time for warmup. Each hardcoded value was a bet that all hardware is the same—a bet that lost repeatedly.

3. The value of empirical calibration. The per-proof memory estimate started at 3GB per partition worker, was revised to 4GB, then 3GB again, and finally settled at 6GB based on empirical OOM data. This iterative refinement, grounded in real failure observations, produced a configuration that worked across the fleet.

4. The lifecycle management paradox. The assistant built a system that automatically destroys underperforming instances—and then discovered that its own system was destroying instances it wanted to keep. The lifecycle manager was working correctly, but its parameters were wrong. This is a recurring pattern in automation: the system does exactly what you told it to do, not what you wanted it to do.

5. The pivot from engineering to science. The chunk ends with the recognition that you cannot predict proving performance from hardware specs. The only reliable approach is to measure empirically and build a database of known performance. This is a shift from engineering (building the system) to science (discovering how the system behaves in the wild).

Conclusion

The OOM wars documented in this chunk transformed a distributed GPU proving system from a collection of brittle, crash-prone scripts into a self-adapting, hardware-aware deployment pipeline. The journey required diagnosing OOM kills across three continents, rewriting the benchmark and entrypoint scripts, fixing a lifecycle bug that let dead instances bleed money, correcting memory estimates through empirical calibration, and ultimately recognizing that the entire threshold-based approach needed to be replaced with data-driven discovery.

The assistant's methodical approach—observe, hypothesize, test, iterate—exemplifies the engineering discipline required to build robust distributed systems. Each failure was a data point. Each fix was a hypothesis to be tested. And the final insight—that you must measure rather than predict—is a lesson that applies far beyond GPU proving, to any system that must operate reliably across heterogeneous, unpredictable infrastructure.