From OOM to Oracle: How a Distributed GPU Proving System Learned to Learn

Introduction

In the span of roughly 220 messages across a single opencode coding session, an AI assistant and its human partner transformed a broken, crash-prone GPU proving pipeline into a self-adapting, data-driven deployment system. The journey—documented across messages 1028 through 1240 of the conversation—reads like a case study in distributed systems evolution: OOM kills, silent API failures, lifecycle bugs, timeout miscalculations, gRPC transport errors, and a fundamental strategic pivot from hardcoded thresholds to empirical discovery.

This article synthesizes that entire arc. It traces the path from the first comprehensive status document through the OOM diagnosis, the two-phase warmup fix, the dynamic concurrency rewrite, the lifecycle bug correction, the cascade of deployment failures, the tactical timeout and warmup adjustments, and finally the architectural commitment to a data-driven experimental system. Along the way, it examines the reasoning patterns, assumptions, and engineering discipline that turned a cascade of failures into a learning infrastructure.

Part I: The OOM Wars

The Meta-Status Document: Taking Stock Before the Storm

The segment opens with an extraordinary artifact: message 1028, a self-initiated, 2,000-word status document written by the assistant during a period when the user had instructed it to operate non-interactively. The assistant had been told "do not stop, do not ask questions until instances run correctly" ([msg 1009]), and it responded by producing a comprehensive project overview covering goals, technical instructions, ten discoveries about Vast.ai and cuzk behavior, a status of completed and in-progress work, and a file inventory [1].

This document is remarkable for what it reveals about the assistant's cognitive architecture. It wasn't following a template or responding to a prompt—it chose to pause its active work and produce a structured knowledge artifact because it understood that the human would eventually return and need to understand what had happened. The message organizes information into five sections—Goal, Instructions, Discoveries, Accomplished, Relevant Files—mirroring the way experienced engineers document complex projects [1].

The discoveries section is particularly valuable. It captures ten hard-won lessons about Vast.ai platform behavior, bash scripting pitfalls, and cuzk memory characteristics. Discovery #5 identifies the critical blocking issue: "OOM on first benchmark with low RAM—PCE extraction with all partition workers simultaneously growing organically causes OOM on machines with 125GB RAM" [1]. This single observation would drive the next several hours of debugging.

Diagnosis Complete: The Turning Point

After the meta-status document, the assistant spends several rounds probing instance states, discovering a routing bug in the vast-manager API (the /api/instances endpoint returned 404 while the actual endpoint was /api/dashboard), and working around it to assemble a clear picture. By message 1041, the assistant can declare: "Now I have the full picture" ([msg 1041]).

The picture is stark. The BC Canada instance—a dual RTX 3090 machine with 125GB RAM—has been killed during benchmark warmup, returning a bench_rate of 0. The Norway instance—a single RTX 4090 with 500GB RAM—is alive and running its benchmark with 12 proofs at concurrency 5 ([msg 1041]).

Message 1041 is the diagnostic pivot. The assistant transitions from investigation to remediation, reading benchmark.sh to understand the exact mechanism of the OOM failure. This is disciplined engineering: diagnose from evidence, not from assumption. The assistant doesn't speculate about what might have caused the crash—it reads the code that orchestrates the benchmark to find the root cause ([msg 1041]).

The Warmup Gambit: A Two-Phase Daemon Restart

The root cause analysis reveals a specific failure mechanism. The cuzk daemon, when started for the first time, must perform PCE (Pre-Compiled Constraint Evaluator) extraction—a memory-intensive initialization that generates a ~26GB cache file. During this extraction, the daemon spawns all partition workers simultaneously, each allocating significant memory for circuit synthesis. On a 125GB machine with 10 partition workers, this simultaneous allocation exceeds available RAM and triggers the OOM killer.

The fix, implemented in benchmark.sh, is elegant in its simplicity: detect the absence of a PCE cache file, start the daemon with partition_workers=2 for the warmup proof (which generates the cache), and then restart the daemon with the full partition count for the actual benchmark. This two-phase approach prevents the memory spike of simultaneous partition synthesis while still allowing full parallelism for the production benchmark [1].

The implementation required careful attention to bash scripting details. The assistant discovered a subtle bug where function ordering in bash caused the start_daemon_with_workers function to be called before it was defined. A syntax check (bash -n) caught this issue before deployment. The assistant also realized that the daemon restart was redundant—the warmup proof could simply run with reduced workers and then the benchmark script could proceed with the full configuration without restarting. This optimization was applied, removing an unnecessary restart cycle [1].

The 41.3 Proofs/Hour Threshold: When Success Reveals Deeper Problems

While the OOM fix was being developed, the Norway instance continued its benchmark. At message 1076, the results arrived: 12 proofs completed, 0 failed, throughput of 41.3 proofs/hour ([msg 1076]). This was a success in one sense—the benchmark completed without crashing—but a failure in another: 41.3 proofs/hour fell below the 50 proofs/hour minimum threshold that defined economic viability.

The assistant's reaction is telling. It doesn't celebrate the successful completion or immediately jump to optimizing the proving pipeline. Instead, it checks the meta-system: does the automated lifecycle management correctly handle a below-threshold result? The assistant queries the entrypoint log to verify that the bench-done reporting path includes a threshold check ([msg 1076]).

This prioritization—verify the automation before optimizing the performance—is the hallmark of mature systems engineering. The assistant understands that if the lifecycle management is broken, optimization work is moot because underperforming instances won't be automatically decommissioned.

The Lifecycle Bug: When "Killed" Doesn't Mean Destroyed

The Norway result also triggers a lifecycle bug discovery. The dashboard shows the instance in "killed" state, but vastai show instances reveals it's still running. The manager's internal state had transitioned to "killed" but the actual destruction command (vastai destroy) hadn't been executed. This disconnect between the manager's internal state and the actual Vast.ai state is a critical gap.

The lifecycle bug in handleBenchDone is one of the most consequential discoveries in the segment. The assistant traces through the vast-manager source code and finds that when an instance fails the benchmark, the endpoint marks it as "killed" in the database but doesn't call vastai destroy to actually terminate the instance on Vast.ai. This means underperforming instances continue running and accruing charges indefinitely [1].

The fix is a single line: adding vastai destroy to the kill handler. But the implications are broader. The assistant discovers that the handleBenchDone endpoint was never being called in some failure modes—instances that crashed during benchmark never reported their results, so the lifecycle manager never knew to destroy them. This leads to a more robust design where the manager actively monitors instance state rather than waiting for callbacks [1].

Part II: The Cascade of Failures

The Promise of Belgium

With the OOM fix deployed and the lifecycle bug corrected, the assistant creates new instances to test the pipeline. A US instance with 2x RTX 3090 and 376GB RAM begins its benchmark. The warmup succeeds—the PCE cache is generated with partition_workers=2 without OOM. The daemon restarts with full partition_workers=10. The batch benchmark begins. Everything looks promising.

Then, at message 1121, the assistant discovers that every instance has been killed ([msg 1121]). The dashboard shows three instances in "killed" state:

The Pivot to Hardware-Aware Configuration

The cascade of failures forces a fundamental rethinking. At message 1124, the assistant articulates the new approach: "I need to: 1. Detect GPU count, 2. Scale benchmark concurrency to GPUs, 3. Also consider partition workers vs available RAM more carefully" ([msg 1124]).

This is the transition from hardcoded configuration to hardware-aware auto-tuning. The assistant recognizes that BENCH_CONCURRENCY=5 was a brittle assumption that didn't account for the diversity of hardware on Vast.ai. A machine with 125GB RAM and a machine with 2TB RAM should not use the same concurrency.

The implementation replaces the hardcoded value with a formula:

The Per-Proof Memory Correction

The first deployment of the dynamic concurrency formula reveals a critical flaw. The Czechia instance (251GB RAM, 2 GPUs) reports concurrency=5 with partition_workers=10. The assistant does the math: available=151GB, per_proof=30GB (10×3GB), 151/30=5, gpu_cap=6 → 5. But this leaves only 1GB of margin—any memory spike would trigger an OOM.

Tracing back to the US instance OOM, the assistant reconstructs the likely memory consumption: if each proof needed 50-60GB (10 workers × 5-6GB), then 5 concurrent proofs would consume 250-300GB plus 100GB overhead = 350-400GB, right at the edge of 376GB. This explains the OOM [1].

The fix is to increase the per-worker multiplier from 3GB to 6GB. This changes the Czechia concurrency from 5 to 2—a much safer configuration. The assistant recalculates for all target machines:

Deploying Hardened Instances

With the corrected formula, the assistant rebuilds the Docker image, destroys the existing instances, and creates two new ones: a Czechia instance (2x RTX 3090, 251GB RAM, $0.282/hr) and a Belgium instance (2x A40, 1TB RAM, $0.574/hr).

The deployment is not without hiccups. The assistant assumes the SSH port is derived from the instance ID (e.g., port 34145 for instance 32714145), but Vast.ai assigns ports independently—the actual port is 34144. The Belgium instance's SSH isn't available immediately, requiring a diagnostic pivot to query the Vast.ai API for connection details.

But the instances register successfully with the manager. The Czechia instance detects 2 GPUs and 251GB RAM, selects partition_workers=10 and concurrency=2. The Belgium instance detects 2 A40 GPUs and 2003GB RAM, selects partition_workers=16 and concurrency=6. The validation is methodical. The assistant checks the entrypoint logs via SSH, grepping for the hardware detection and configuration decisions. The log output confirms the auto-configuration is working correctly. The instances proceed through parameter download, warmup, and benchmark without OOM crashes [1].

Part III: The Cascade Continues

Two Failures, Two Modes

The chunk spanning messages 1152 through 1240 opens with a moment of cautious optimism. The Belgium instance, with 2TB of RAM, seemed destined for success. Its warmup proof completed cleanly in 305 seconds, the PCE cache was generated successfully, and the daemon restarted with full partition_workers=16 and concurrency=6—the maximum allowed by the GPU cap formula. The assistant monitored its progress through a series of polling cycles, watching as 8 of 12 benchmark proofs completed with an average prove time of around 175 seconds ([msg 1152], [msg 1155]). Everything was working.

Then, silence.

When the assistant checked again, SSH to Belgium returned "Connection refused." The instance was gone. The manager dashboard confirmed the verdict: State=killed, Kill=benchmark timeout (20min) ([msg 1158]). The 20-minute benchmark timeout, a hardcoded constant in the vast-manager's monitor loop, had fired before the benchmark could complete [2].

The assistant's forensic analysis revealed the mismatch. The benchmark flow consisted of three phases: warmup proof with PCE extraction (~5 minutes), daemon restart with SRS preload (~1–2 minutes), and the actual batch benchmark of 12 proofs at concurrency 6 (~15–20 minutes). The total estimated duration was 25–30 minutes—well over the 20-minute timeout. The assistant had built a sophisticated hardware-aware configuration system that dynamically scaled partition workers and concurrency, but it had forgotten to scale the timeout proportionally ([msg 1159], [msg 1160]) [2].

Meanwhile, the Czechia instance had suffered a different fate. When the assistant checked its benchmark log, it found a gRPC transport error: "Error: Prove RPC failed" with the underlying cause being a "transport error" ([msg 1164]). The warmup proof had completed successfully, the daemon had restarted with full partition_workers=10, but the very first proof of the batch benchmark had failed with a broken pipe. The daemon was alive and processing—the assistant's investigation showed that both proofs had been accepted and all 10 partitions per proof were synthesizing via the PCE fast path ([msg 1166]). But the gRPC client had timed out waiting for the first proof to complete, because the GPU kernel compilation overhead after the daemon restart exceeded the client's default timeout ([msg 1165], [msg 1167]) [2].

The Tactical Response

The assistant responded with a series of targeted fixes, each addressing a specific failure mode:

Fix 1: Increase the benchmark timeout. The timeout in the vast-manager's monitor loop was increased from 20 to 45 minutes ([msg 1160]). This was a simple constant change, but it required rebuilding the Go binary, redeploying it to the controller host, and recreating the destroyed Belgium instance.

Fix 2: Add a post-restart warmup proof. To address the gRPC timeout issue, the assistant modified benchmark.sh to run a single "post-restart warmup" proof after the daemon restart but before the timed batch benchmark ([msg 1177]). This single proof would compile and cache the GPU kernels with the full partition_workers count, so that subsequent proofs would complete within the client's timeout window. If the warmup proof itself failed with a broken pipe, the script would retry.

Fix 3: Refine partition worker logic. Based on user guidance that "for 256G sometimes pw=8 is needed" ([msg 1211]), the assistant added a new tier to the partition worker selection logic in entrypoint.sh ([msg 1214]). Machines with approximately 256GB of RAM would now use partition_workers=8 instead of 10, reducing memory pressure during synthesis [2].

Each fix was rational, well-reasoned, and implemented with care. The Docker image was rebuilt and pushed to Docker Hub. New instances were created. The assistant waited, monitored, and hoped.

The Second Wave of Failures

The new instances failed too.

A new Belgium instance (2× A40, 2TB RAM) completed its benchmark but achieved only 35.91 proofs per hour—well below the 50 proofs/hour minimum threshold. The manager's lifecycle logic, which the assistant had fixed in the previous segment to immediately destroy underperforming instances, dutifully killed it ([msg 1203], [msg 1204]). The result was shocking: a machine with two datacenter GPUs and 2TB of RAM had performed worse than a single RTX 4090, which had achieved 41.32 proofs/hour in an earlier test. The A40, built on the Ampere architecture and optimized for inference workloads, was fundamentally slower than the Ada Lovelace RTX 4090 for this proving workload [2].

A new Czechia instance (2× RTX 3090, 251GB RAM) crashed entirely. SSH returned "Connection refused." The manager dashboard reported bench_rate=0 and kill=bench_rate 0.0 below min_rate 50.0 ([msg 1207]). The post-restart warmup proof had completed successfully—the assistant had observed it finishing in 337 seconds with the PCE file present at 26GB ([msg 1205]). But the instance died during or shortly after the transition to the full batch benchmark, likely due to an OOM crash despite all the memory-saving measures [2].

Part IV: The Moment of Reckoning

Recognizing the Pattern

In message [msg 1208], the assistant did something that marks the inflection point of the entire segment. It stepped back from the tactical firefighting and performed a structured analysis of the failure patterns:

"Let me step back and think about what's going wrong. We have three patterns of failure... This is a recurring pattern and I'm not sure which failure mode is happening without being able to check the daemon logs after the crash."

The assistant enumerated the failure modes: OOM during warmup (which had been addressed by the PCE-cache-dependent startup), gRPC broken pipe during batch (partially addressed by the post-restart warmup), and OOM during batch with high concurrency (addressed by dynamic concurrency scaling). But the deeper pattern was that none of these fixes were sufficient because the system had no way to learn from its failures. Each instance was a fresh experiment, configured by guesswork, and destroyed when it underperformed against an arbitrary standard [2].

The assistant then performed a manual memory budget calculation for the 251GB Czechia machine:

The Strategic Insight

The user's response to the assistant's analysis confirmed the direction. Rather than choosing between lowering the minimum rate or reducing partition workers, the user endorsed a more radical approach: build a system that experimentally discovers optimal hardware configurations ([msg 1211]).

The assistant's response in [msg 1210] captured the new philosophy with crystalline clarity:

"This is a great approach — instead of trying to predict performance from specs, we need to experiment and track."

This single sentence represents the abandonment of an entire engineering paradigm. The previous approach was predictive: given hardware specifications (GPU model, RAM size, GPU count), the system would compute configuration parameters and predict whether the instance would meet the minimum proofs/hour threshold. This approach failed because:

  1. Memory consumption was not well-characterized. The assistant's own analysis showed the uncertainty: "each worker uses ~12-15GB?" These were guesses, not measurements.
  2. GPU performance was not monotonic with specs. The A40 (a datacenter GPU with 48GB VRAM) performed worse than the RTX 4090 (a consumer GPU with 24GB VRAM). The relationship between specs and performance was not linear or predictable.
  3. The interaction effects were complex. Partition workers, concurrency, SRS memory mapping, PCE caching, and GPU synthesis all interacted in ways that could not be modeled from first principles with the available data [2]. The new approach is experimental: deploy instances, run benchmarks, record the results, and use the accumulated data to make future deployment decisions. Instead of asking "will this machine work?", the system now asks "what has this machine (or similar machines) achieved in the past?"

Part V: Building the Data-Driven System

The Four Pillars

The assistant's todo list in [msg 1209] defined the work program for the new approach:

  1. Host performance table — track bench_rate per host_id in the database
  2. Offer search API — search Vast.ai offers with GPU/RAM/price filters, overlaying known host performance
  3. UI: offer listing with deploy button — showing known host performance for each offer
  4. Lower/configurable min_rate — make the minimum proofs/hour threshold dynamic These four features collectively transform the vast-manager from a static lifecycle tracker into a learning system. The host_perf table is the system's memory—it accumulates empirical data about which physical hosts deliver acceptable proving throughput. The offer search API bridges the gap between Vast.ai's marketplace and the system's empirical knowledge, allowing operators to filter offers not just by specs but by actual historical performance. The deploy endpoint closes the loop, enabling the system to provision instances directly from search results. And the configurable minimum rate acknowledges that the hardcoded 50 proofs/hour threshold was arbitrary and counterproductive [2].

The Implementation

The implementation unfolded across a sequence of carefully ordered edits to the vast-manager's main.go file and its embedded UI. The assistant read the existing codebase methodically—first the DB schema and Server struct, then the route registration patterns, then the handleRegister and handleBenchDone handlers, then the monitorCycle and dashboard handler. Each read targeted a specific gap in understanding, building a complete mental model of the codebase before making changes [2].

The edits themselves followed a logical dependency order:

  1. DB schema: Added the host_perf table to the SQLite schema, with columns for host_id, bench_rate, GPU name, RAM size, and timestamp.
  2. Type definitions: Added the VastOffer struct to represent search results from Vast.ai's marketplace.
  3. Bench-done handler: Modified the handleBenchDone function to look up the host_id from the Vast.ai instance cache and record the benchmark result in the host_perf table. This is the critical data pipeline—every benchmark result, whether pass or fail, is now recorded against the physical host that produced it.
  4. Handler functions: Added the handleOffers and handleDeploy functions that query Vast.ai's CLI and create instances.
  5. Route registration: Registered the new API endpoints (/api/offers and /api/deploy) in the HTTP router. This is the moment when the backend implementation becomes live and accessible.
  6. UI integration: Read the existing ui.html file to understand the JavaScript patterns, HTML structure, and CSS conventions, then prepared to add a new "Offers" panel with search form, results table, and deploy buttons [2].

The Architecture of a Learning System

The data-driven system the assistant built is elegant in its simplicity. The host_perf table is the persistent memory of the deployment pipeline. Every benchmark result—whether a success (e.g., 41.32 proofs/hour on an RTX 4090) or a failure (e.g., bench_rate 0 on a crashed RTX 3090)—is recorded keyed by the Vast.ai host identifier. Over time, this table accumulates an empirical map of the marketplace: which specific physical machines are fast, which are slow, and which crash [2].

The /api/offers endpoint queries Vast.ai's available GPU instances, filters them by GPU model, RAM size, disk space, and price, and then overlays known performance data from the host_perf table. For a host that has been benchmarked before, the response includes the historical bench_rate, allowing operators to avoid underperforming hosts and prioritize proven ones. For new hosts, the response shows bench_rate: null, signaling that this is an untested configuration [2].

The /api/deploy endpoint consumes an offer and creates a Vast.ai instance with the specified configuration. It generates a UUID, assigns a runner ID, inserts a row into the instances table, and executes the vastai create instance command. The instance then proceeds through the standard lifecycle—registration, parameter download, warmup, benchmark—and its result flows back into the host_perf table, closing the loop [2].

This architecture enables a continuous improvement cycle: search for offers, deploy instances, benchmark them, record performance, and use that data to inform future searches. The system learns over time, building a map of which hosts and configurations deliver acceptable proving rates. It replaces guesswork with measurement, prediction with experimentation.

Themes and Lessons

The Failure of First-Principles Reasoning

The central lesson of this segment is that first-principles reasoning—modeling system behavior from hardware specifications—failed catastrophically in this domain. The assistant attempted to compute optimal configuration parameters from RAM size, GPU count, and GPU model, but the real system was too complex, with too many unknown parameters and non-linear interactions. Memory consumption during partition synthesis was not well-characterized. GPU performance varied unpredictably across architectures. The relationship between specs and throughput was not monotonic.

This is a common pattern in distributed systems engineering. When the system is simple enough, you can predict its behavior from first principles. But as complexity grows—as components interact in unexpected ways, as the environment becomes heterogeneous, as workloads vary—prediction becomes unreliable. The only reliable oracle is empirical measurement.

The Value of a Feedback Loop

The assistant's pivot from predictive to experimental methodology is a textbook example of closing the feedback loop. The previous system had no memory: each deployment was a fresh experiment, configured by guesswork, and destroyed when it failed. The new system records every outcome and uses that data to inform future decisions. This is the difference between a static configuration and a learning system.

The host_perf table is the simplest possible feedback mechanism—a single database table that maps host identifiers to benchmark results. But its impact is transformative. It turns every failure into a data point, every success into a reusable insight. Over time, the system accumulates knowledge that no amount of first-principles reasoning could produce.

The Limits of Tactical Firefighting

The segment also illustrates the limits of tactical firefighting. The assistant applied a series of increasingly clever fixes—the PCE-cache-dependent warmup, the post-restart warmup proof, the dynamic concurrency scaling, the timeout increase, the partition worker refinement. Each fix addressed a specific failure mode, but the failures kept coming in new forms. The OOM fix worked, but exposed the gRPC timeout. The gRPC timeout fix worked, but the instance still crashed. The partition worker refinement reduced memory pressure, but the instance still underperformed.

This pattern is characteristic of systems where the failure modes are emergent properties of the system's complexity, not isolated bugs. Each tactical fix addresses one symptom but does not change the underlying dynamics. The only way to break the cycle is to change the approach fundamentally—to build a system that can discover and adapt to failure modes it hasn't even encountered yet.

The Primacy of Verification

Throughout the segment, the assistant never assumes a fix works until it sees evidence. Every code change is syntax-checked, every deployment is verified via SSH or API, every configuration decision is validated against real hardware. This discipline separates robust systems from fragile ones.

The Danger of Hardcoded Assumptions

The hardcoded BENCH_CONCURRENCY=5 caused OOMs across diverse hardware. The hardcoded 50 proofs/hour threshold killed instances that were performing well for their hardware class. The hardcoded 20-minute timeout killed instances that needed more time for warmup. Each hardcoded value was a bet that all hardware is the same—a bet that lost repeatedly.

The Value of Empirical Calibration

The per-proof memory estimate started at 3GB per partition worker, was revised to 4GB, then 3GB again, and finally settled at 6GB based on empirical OOM data. This iterative refinement, grounded in real failure observations, produced a configuration that worked across the fleet.

Conclusion

The arc of Segment 8—from OOM diagnosis through tactical fixes to data-driven discovery—documents a complete evolution in engineering methodology. What began as a hunt for specific bugs in benchmark scripts and lifecycle handlers ended with the recognition that the entire predictive approach was fundamentally insufficient for the problem at hand.

The cascade of failures that followed the deployment of hardened instances—Belgium's timeout, Czechia's gRPC error, Belgium's underperformance, Czechia's crash—provided the empirical evidence that first-principles reasoning could not reliably predict proving performance from hardware specifications. The A40 with 2TB of RAM underperformed a single RTX 4090. The RTX 3090 with 251GB of RAM crashed despite all the memory-saving measures. The system was too complex, the interactions too non-linear, the unknowns too numerous.

The pivot to a data-driven experimental system was the correct engineering judgment. The host_perf table, the offer search API, the deploy endpoint, and the foundational UI code represent the architectural commitment to a new philosophy: when you can't predict, measure. When you can't model, experiment. When your assumptions fail, let the data speak.

In the broader narrative of the opencode session, this segment is the turning point. The tactical fixes of the earlier segments gave way to a strategic reorientation that would reshape the entire deployment pipeline. The system is no longer a static deployment tool; it is a learning system that improves with every benchmark run. The data it collects will inform future deployment decisions, creating a virtuous cycle of measurement and improvement that transforms a cascade of failures into a foundation for reliable, self-tuning infrastructure.

References

[1] "The OOM Wars: A Distributed GPU Proving System's Journey from Crash to Self-Tuning Infrastructure" — Chunk 0 article covering messages 1028–1151, the OOM diagnosis, two-phase warmup fix, dynamic concurrency rewrite, lifecycle bug correction, and deployment of hardened instances.

[2] "From Tactical Firefighting to Data-Driven Discovery: The Pivot That Reshaped a GPU Proving Pipeline" — Chunk 1 article covering messages 1152–1240, the cascade of failures (Belgium timeout, Czechia gRPC error, Belgium underperformance, Czechia crash), tactical fixes (timeout increase, post-restart warmup, partition worker refinement), and the strategic pivot to a data-driven experimental system with host_perf table, offer search API, deploy endpoint, and UI.