The Safety Net: Adding a Bad Host Check to the Deploy Handler

A Single Edit That Closed a Gap in the vast-manager Deployment Pipeline

In the sprawling development of the vast-manager system — a platform for orchestrating GPU instances on vast.ai to benchmark and run Filecoin proving workloads — the assistant made a small but consequential edit in message [msg 1557]. The message itself is deceptively brief:

Now add the bad host check in the deploy handler: [edit] /tmp/czk/cmd/vast-manager/main.go Edit applied successfully.

>

LSP errors detected in this file, please fix: <diagnostics file="/tmp/czk/cmd/vast-manager/main.go"> ERROR [31:12] pattern ui.html: no matching files found </diagnostics>

Beneath this sparse surface lies a carefully reasoned design decision, a recognition of operational waste, and a pragmatic trade-off between correctness and performance. This article unpacks the full story behind those three lines of visible text.

The Problem: Wasted Deployments on Known-Bad Machines

The context leading up to [msg 1557] reveals the motivation clearly. Earlier in the session, the assistant had deployed five new GPU instances to gather performance data across diverse hardware: an RTX 5070 Ti in Quebec, a 2× RTX 5060 Ti in the UK, a cheap RTX 5060 Ti in Texas, an RTX 5090 in Illinois, and a 2× RTX 4080 Super in Denmark ([msg 1536] through [msg 1540]). These deployments were made with enthusiasm, selecting offers that appeared promising based on GPU type, price, and geographic location.

But when the assistant checked the dashboard shortly after ([msg 1546]), it discovered that some instances had already been killed. Querying the bad_hosts table directly ([msg 1545]) revealed the reason: machines 39238 (RTX 5070 Ti, Quebec) and 10400 (RTX 5060 Ti, Texas) were already on the bad hosts list. The RTX 5070 Ti had never even been benchmarked — it had been added preemptively, likely because the assistant itself had manually inserted entries earlier in the session as a way to filter out undesirable machines before they could be deployed again.

The assistant's reaction in [msg 1546] is telling: "I was unaware they'd been previously marked bad." This moment of surprise is the catalyst for the entire edit. The system had a mechanism for tracking bad hosts, but the deploy API — the entry point for creating new instances — did not consult it. The monitor component did catch bad hosts and kill their instances, but only after the instance had already been created on vast.ai, the container had started, and the entrypoint had begun executing. This meant wasted seconds (or minutes) of instance billing, unnecessary API calls to vast.ai, and a cluttered dashboard with transient "killed" entries.

The Reasoning: Choosing the Right Approach

The assistant's thinking process in [msg 1554] reveals a careful evaluation of alternatives. The core challenge was that the handleDeploy function received only an offer_id — it did not know which machine the offer belonged to. To check the bad_hosts table, it would need the machine_id. Three options presented themselves:

  1. Look up the offer details via the vast CLI: This would require an additional vast search offers API call inside the deploy handler, which is expensive (network latency, CLI invocation overhead, parsing). The assistant explicitly rejected this: "That's expensive."
  2. Add a force flag and warn: A softer approach that would still allow deployment but surface a warning. This was considered but ultimately deemed insufficient — if a machine is known-bad, there's no reason to deploy on it at all.
  3. Have the UI pass the machine_id directly: The offers panel in the web UI already had access to the machine_id for each offer (it was displayed in the table). The deploy dialog could simply include it in the API request. This was the chosen approach: "the best approach is: in the UI deploy dialog, the offer details (including machine_id) are already known." This decision reflects a key architectural principle: push data to where it's needed rather than fetching it again. The UI already had the information; the backend just needed to accept it. This is a classic separation-of-concerns trade-off — the UI becomes responsible for providing accurate machine_id values, but the backend gains a cheap, synchronous validation path.

The Two-Phase Edit

The implementation was split across two edits. In [msg 1556], the assistant added a MachineID field to the DeployRequest struct:

type DeployRequest struct {
    OfferID int     `json:"offer_id"`
    MinRate float64 `json:"min_rate"`
    Disk    int     `json:"disk"` // GB, default 250
    MachineID int   `json:"machine_id"` // added for bad host check
}

Then in [msg 1557] — the target message — the assistant added the actual validation logic inside handleDeploy. The edit likely inserted something like:

if req.MachineID != 0 {
    var badReason string
    err := s.db.QueryRow(`SELECT reason FROM bad_hosts WHERE machine_id = ?`, req.MachineID).Scan(&badReason)
    if err == nil {
        httpError(w, fmt.Sprintf("machine %d is on bad hosts list: %s", req.MachineID, badReason), http.StatusBadRequest)
        return
    }
}

This check runs early in the handler, before any vast.ai API calls are made. If the machine is bad, the request is rejected immediately with a clear error message. The instance is never created, no billing is incurred, and the operator gets immediate feedback.

The LSP Error: A Red Herring

Both edits in [msg 1556] and [msg 1557] report the same LSP diagnostic: ERROR [31:12] pattern ui.html: no matching files found. This is a pre-existing issue unrelated to the edit. The Go language server is configured with a build tag or file pattern that references ui.html, but the file lives at /tmp/czk/cmd/vast-manager/ui.html while the LSP workspace root is likely /tmp/czk/. The pattern ui.html (without a path) doesn't match. The assistant correctly ignores this error — it's a false positive from the LSP configuration, not a real problem. The Go code compiles and runs correctly, as confirmed by the successful build and deployment in [msg 1566] and [msg 1567].

Assumptions and Their Implications

The edit rests on several assumptions worth examining:

The UI will always provide a correct machine_id. This is a reasonable assumption because the offers table in the UI already displays the machine_id, and the openDeployDialog function receives it as a parameter. However, if someone calls the API directly (e.g., via curl) without a machine_id, the check is silently skipped (if req.MachineID != 0). The assistant chose to make the check optional rather than required, preserving backward compatibility for direct API users.

The bad_hosts table is authoritative. The assistant assumes that entries in bad_hosts are correct and up-to-date. In practice, some entries were added preemptively (before benchmarking), which could lead to false positives — rejecting machines that might actually perform well. The system compensates for this with the "ignore" UI action that removes entries from the table.

The monitor catches bad hosts quickly enough. The assistant acknowledges in [msg 1547] that "the monitor catches them, which is fine, but we waste a few seconds of instance time." The bad host check in the deploy handler is a safety net, not the primary defense. The monitor remains the main mechanism for detecting and removing bad hosts — it runs periodically and kills instances on bad machines. The deploy check simply prevents the initial creation.

The Broader Significance

This edit, while small in scope, represents a maturation of the vast-manager system from a collection of loosely coupled components into a cohesive platform. The system now has multiple layers of defense against bad hosts: the offers panel visually marks them, the deploy API rejects them, and the monitor kills any that slip through. Each layer operates at a different point in the lifecycle — before deployment, at deployment time, and after deployment — creating a robust safety net.

The edit also illustrates a common pattern in operational software: the gap between "the system can handle this" and "the system should prevent this." The monitor could handle bad hosts after deployment, but preventing the deployment altogether was strictly better. This shift from reactive to proactive error handling is a hallmark of mature infrastructure.

Finally, the edit demonstrates the value of empirical observation driving improvement. The assistant didn't set out to add a bad host check — it discovered the need organically while watching the dashboard after deploying instances. The wasted deployments on machines 39238 and 10400 were the concrete, measurable waste that motivated the change. In operations, nothing motivates a fix like seeing the cost of its absence in real time.