The Bad Hosts Revelation: When Deployment Automation Collides with Historical Data

Introduction

In any automated infrastructure management system, the gap between data collection and data utilization is where the most costly mistakes hide. Message [msg 1545] of this opencode session captures a pivotal moment of discovery: the assistant, having just deployed five new GPU instances across diverse hardware, realizes that several were immediately killed because they landed on machines already recorded in a bad_hosts table. The message is deceptively simple — a single bash command querying a SQLite database — but it exposes a critical architectural flaw in the vast-manager platform and crystallizes the tension between rapid deployment and informed decision-making.

The Context: A Deployment Spree

The moments leading up to message [msg 1545] are essential to understanding its significance. The assistant had been engaged in an intensive session of hardening the vast-manager platform — a custom management system for orchestrating GPU compute instances on vast.ai for proof generation workloads. The system had been accumulating performance data through benchmarks, tracking which machines (identified by machine_id) were profitable and which were not.

In [msg 1535], the assistant reviewed available offers and selected five promising candidates for deployment:

The Message Itself

The subject message is a direct SQLite query executed over SSH on the vast-manager host:

[bash] ssh 10.1.2.104 "sudo sqlite3 /var/lib/vast-manager/state.db 'SELECT * FROM bad_hosts;'"
59017|Not good enough (RTX 5060 Ti, Texas, US)|2026-03-12 09:55:11
60742|Not good enough (RTX 4080S, Denmark, DK)|2026-03-12 09:55:23
367134|Not good enough (RTX 4060 Ti, Vietnam, VN)|2026-03-12 10:04:46
10400|Not good enough (RTX 5060 Ti, Texas, US)|2026-03-12 11:13:21
10686|Not good enough (RTX 5060 Ti, Texas, US)|2026-03-12 11:13:24
39238|Not good enough (RTX 5070 Ti, Quebec, CA)|2026-03-12 11:13:26

Six rows. Six machines that had been benchmarked, found inadequate, and recorded as "not good enough." And critically, three of these machine IDs — 10400, 10686, and 39238 — correspond directly to offers the assistant had just deployed. Machine 10400 was the RTX 5060 Ti in Texas. Machine 10686 was the 2× RTX 5060 Ti in Texas. Machine 39238 was the RTX 5070 Ti in Quebec. All three were already in the bad hosts database before the assistant deployed them.

Why This Message Was Written: The Reasoning and Motivation

The assistant wrote this message for a straightforward diagnostic purpose: to verify the contents of the bad_hosts table after discovering that instances were being killed immediately upon registration. The chain of reasoning is clear:

  1. Observation: New instances were killed immediately ([msg 1543]).
  2. Hypothesis: The instances landed on machines that were already flagged as bad hosts.
  3. Attempted verification: The assistant tried the REST API endpoint (/api/bad-hosts) but got a 404.
  4. Fallback: The assistant bypassed the missing API and queried the SQLite database directly. The motivation was not idle curiosity — it was damage assessment. The assistant needed to understand which machines were in the bad hosts table to determine the extent of the wasted deployment, and more importantly, to understand why the deploy API had allowed deployments onto known-bad machines in the first place.

The Assumptions Made

This message reveals several assumptions, some correct and some incorrect:

Correct assumption: The bad_hosts table existed in the SQLite database and contained the relevant data. This was confirmed by the successful query.

Incorrect assumption (implicit): The assistant had assumed that the deploy API would check the bad_hosts table before creating instances. The fact that three deployments succeeded onto known-bad machines proves this check was missing. The assistant's own comment in [msg 1543] — "Two instances were immediately killed because they were on the bad hosts list!" — carries a tone of surprise, suggesting the assistant expected the system to prevent this.

Incorrect assumption (implicit): The assistant assumed a REST API endpoint for bad hosts existed. The 404 response proved otherwise. This is a notable gap in the API surface — the system had a bad_hosts table and a mechanism to add entries to it (when benchmarks failed), but no way to query it through the management API.

The Design Flaw Exposed

The bad_hosts table is a textbook example of a system that collects data without using it effectively. The table was being populated — whenever a benchmark failed to meet the minimum rate, the machine was recorded as "not good enough." But the deploy logic, which created new instances from offers, did not cross-reference this table. The result was a cycle of waste: deploy an instance, benchmark it, find it too slow, record it as bad, then deploy another instance on the same machine and repeat the process.

The timestamps in the table tell a damning story. Machines 10400, 10686, and 39238 were all added to bad_hosts at approximately 11:13 UTC on March 12 — about an hour before the deployment spree. The assistant had literally just benchmarked these machines, found them inadequate, and then deployed new instances onto them. The system had no memory of its own failures.

This is a classic infrastructure automation pitfall: the speed of deployment outpaces the feedback loop. The assistant could spin up instances in seconds, but the knowledge gained from each benchmark was siloed in a database table that the deployment pipeline never consulted.

Input Knowledge Required

To fully understand this message, several pieces of context are necessary:

  1. The vast-manager architecture: The system uses a SQLite database (state.db) to track instances, host performance, and bad hosts. The bad_hosts table stores machine_id, a reason string, and a timestamp.
  2. The machine_id concept: Each physical machine on vast.ai has a unique machine_id. Multiple offers (different GPU configurations, pricing, etc.) can exist on the same machine. The bad_hosts table flags machines, not offers.
  3. The deployment flow: The assistant deploys instances by calling /api/deploy with an offer_id. The offer includes a machine_id. The deploy API creates a vast.ai contract but apparently does not check the bad_hosts table.
  4. The benchmark lifecycle: Instances go through states: fetching (downloading parameters), params_done, benching (running benchmark), and either running (if rate ≥ min_rate) or killed (if rate < min_rate). When a benchmark fails, the machine is added to bad_hosts.
  5. The recent host_perf fix: In [msg 1525]–[msg 1527], the assistant fixed the host_perf table to keep the maximum benchmark score rather than overwriting with the latest. This shows the assistant was already aware of data integrity issues in the database logic.

Output Knowledge Created

This message produced several important outputs:

  1. Concrete evidence: The exact contents of the bad_hosts table, confirming that machines 10400, 10686, and 39238 were already flagged before the new deployments.
  2. A bug report by implication: The mismatch between the bad_hosts table contents and the deploy behavior constitutes a discovered bug — the deploy API needs to reject offers on known-bad machines.
  3. A data point for prioritization: The assistant now knows which deployments were wasted and can focus on the remaining two (the RTX 5090 and the 2× RTX 4080S) that may have landed on clean machines.
  4. A lesson about API completeness: The missing /api/bad-hosts endpoint is now a known gap. The assistant had to bypass the REST layer entirely and query SQLite directly via SSH — an operational workaround that should not be necessary in a well-designed management system.

The Thinking Process

The assistant's reasoning trajectory across messages [msg 1542]–[msg 1545] reveals a methodical diagnostic approach:

  1. Observation phase ([msg 1542]): The assistant waits for new instances to register, monitoring journal logs. The logs show deployments but no registration events for some instances.
  2. Hypothesis formation ([msg 1543]): The assistant notices instances were "immediately killed" and hypothesizes they were on the bad hosts list. This is an inference — the assistant doesn't see the kill event directly but deduces it from the absence of registration.
  3. Initial investigation ([msg 1543]): The assistant tries the REST API endpoint /api/bad-hosts but gets a 404. This is a dead end.
  4. Workaround ([msg 1545]): The assistant pivots to direct SQLite access via SSH, using sudo sqlite3 to query the database. This requires knowing the database path (/var/lib/vast-manager/state.db) and having the necessary SSH and sudo access.
  5. Analysis (implicit after the message): The results confirm the hypothesis — three of the deployed machines were in the bad hosts table. The assistant's thinking is characterized by pragmatic problem-solving. When the API endpoint fails, it doesn't get stuck — it goes straight to the data source. This is a hallmark of experienced infrastructure engineers: the database is the ground truth, and any API is just a convenience layer.

Mistakes and Incorrect Assumptions

Several mistakes are visible in and around this message:

The deploy API gap: The most significant mistake is architectural — the deploy handler was written without a check against bad_hosts. This is a straightforward omission that could have been caught by a code review or by tracing the deployment flow end-to-end.

The missing API endpoint: The assistant created the bad_hosts table and a mechanism to populate it (in the benchmark completion handler), but never created a corresponding read endpoint. This made the data invisible to the management interface and forced the SSH workaround.

The assumption that "immediately killed" means bad hosts: The assistant assumed the instances were killed because of the bad hosts list. While this turned out to be correct for some instances, there could be other reasons for immediate termination — the instances might have failed to start, the entrypoint might have crashed, or vast.ai itself might have terminated them. The assistant didn't verify the kill reason.

The timing of the deployment spree: The assistant deployed five instances in rapid succession without first checking the bad hosts data. A more cautious approach would have been to query the database first, then filter the offers. The assistant's enthusiasm for gathering diverse performance data led to preventable waste.

Conclusion

Message [msg 1545] is a small query that speaks volumes about the state of the vast-manager platform at this point in development. It reveals a system that is actively collecting valuable operational data — benchmark results, bad host flags, performance metrics — but has not yet closed the loop between data collection and decision-making. The bad_hosts table is a perfect example of "zombie data": information that exists but has no effect on system behavior.

The message also illustrates a fundamental truth about infrastructure automation: the speed of execution must be matched by the speed of learning. Deploying instances in seconds is useless if the knowledge gained from each deployment is not fed back into the deployment decision. The assistant's SSH workaround — querying SQLite directly — is a temporary patch for a systemic gap. The real fix, which the assistant would likely implement next, is to make the deploy API consult the bad_hosts table before creating instances, closing the loop and ensuring that the system never repeats its own mistakes.

In the broader narrative of this coding session, message [msg 1545] marks the transition from platform building to platform hardening — from creating features to ensuring they work together coherently. The bad hosts revelation is a forcing function for integration, pushing the assistant to connect the disparate pieces of the management system into a cohesive whole where data flows from collection to action without human intervention.