The False-Positive Kill: When a Label Mismatch Nearly Destroyed a Healthy Worker

In the middle of a complex deployment session spanning Docker builds, Vast.ai instance management, and a custom manager service, the user sent a brief but critical message:

".471 marked as killed even tho it still runs and is fetching"

This message — message index 938 in the conversation — is a masterclass in concise bug reporting. In eleven words, the user communicates that the vast-manager service has made a false-positive determination, marking instance C.32710471 as killed when it is in fact alive and actively downloading Filecoin proving parameters. The message triggered a deep debugging session that uncovered a fundamental mismatch between two different "label" systems in the Vast.ai platform, leading to a fix that reshaped the core matching logic of the entire management service.

The Immediate Context

To understand why this message was written, we must understand what happened in the minutes before. The assistant had just completed a complex sequence of operations: rebuilding and redeploying the vast-manager service with a fix to handle stale database entries. The old instance 32709851 had been destroyed but remained visible in the manager's UI because the monitor's disappearance check only examined instances in the running state — not registered or other states. The assistant fixed this by expanding the check to cover all non-killed states, rebuilt the Go binary, deployed it to the controller host, and restarted the service.

This fix was correct in intent but incomplete in execution. It addressed the symptom (stale entries not being cleaned up) without accounting for a subtle dependency: the monitor's ability to determine whether an instance still exists on Vast.ai depends on matching the instance's label against the Vast API's label field. If that matching fails, the monitor will incorrectly conclude the instance has disappeared — and kill it.

The user observed exactly this outcome. Instance 32710471, which had been created just minutes earlier, was running perfectly: it had established its portavaild tunnel, registered with the manager, and was actively downloading the ~100GB of proving parameters needed for Filecoin proof generation. Yet the monitor marked it as killed. The user, monitoring the system, saw this contradiction and reported it immediately.

Input Knowledge Required

Understanding this message requires significant domain knowledge spanning several systems. The reader must know that .471 refers to Vast.ai instance ID 32710471 (the convention of using the last three digits as shorthand). They must understand the architecture: instances are Docker containers running on remote GPUs that register with a central manager service, which monitors their lifecycle through states like registered, fetching, benchmarking, and running. The "killed" state is a terminal state in the manager's SQLite database that prevents the instance from being used.

The reader must also understand the dual-label system on Vast.ai. Internally, Vast.ai injects a VAST_CONTAINERLABEL environment variable into running containers with the format C.<instance_id> (e.g., C.32710471). This is how instances know their own identity. However, the Vast API exposes a separate label field that is only populated if the user explicitly runs vastai label instance <id> <label>. These two labels are entirely independent — one is an automatic injection, the other is a user-managed annotation.

The message also implies knowledge of the manager's monitor cycle: it runs every 60 seconds, caches the list of active Vast instances from the API, builds a map keyed by the API's label field, and then iterates over database instances to see if they have a matching Vast instance. If no match is found, the instance is assumed to have disappeared and is marked as killed.

The Root Cause: A Map Built on the Wrong Key

The assistant's investigation, which began immediately after receiving this message, traced the bug to a single line of Go code in the monitor function:

labelMap := make(map[string]VastInstance)
for _, vi := range vastInstances {
    if vi.Label != "" {
        labelMap[vi.Label] = vi
    }
}

The labelMap was keyed by vi.Label — the Vast API's label field. For instance 32710471, this field was null because nobody had ever run vastai label instance 32710471 C.32710471. The instance was simply not in the map. When the monitor later iterated over database instances and tried to find a matching Vast instance by looking up C.32710471 in the label map, it got nothing. The disappearance check concluded the instance was gone and marked it as killed.

This is a classic software bug: a map built on an optional field that is often empty. The fix the assistant implemented was to build a secondary map keyed by the Vast instance ID (an integer that is always present), and to create a lookupVast helper that tries the label first, then falls back to extracting the numeric ID from the C.<id> pattern in the database label. This two-tier lookup ensures that even when the Vast API label is empty, instances can still be matched by their ID.

Assumptions and Mistakes

Several assumptions collided to create this bug. The original author of the monitor assumed that the Vast API's label field would always be populated for instances under management — perhaps because they planned to set labels manually, or because they assumed the API would mirror the container label. This assumption was incorrect: Vast.ai treats these as entirely separate systems, and the API label remains null until explicitly set.

The assistant's earlier fix — expanding the disappearance check to cover all non-killed states — was itself correct but exposed this pre-existing vulnerability. Before the fix, the stale entry 32709851 (in registered state) was never checked for disappearance, so the label matching bug was latent. The fix made the check comprehensive, which immediately triggered the bug on the new instance.

There was also an assumption about the relationship between VAST_CONTAINERLABEL and the API label. Earlier in the session, the assistant had confirmed that VAST_CONTAINERLABEL=C.32710471 was being set inside the container, and the instance used this to register itself with the manager. The assistant assumed this label would also be visible through the Vast API — but it was not. The container environment variable and the API field are separate channels with no synchronization between them.

The Fix and Its Implications

The assistant's fix added an ID-based map (idMap) keyed by the Vast instance ID (an integer), and a lookupVast function that extracts the numeric ID from the C.<id> pattern when the label lookup fails. This required updating every call site that previously used labelMap alone: the disappearance check, the killTimedOut function, the bad hosts check, and the dashboard merge that enriches database instances with Vast API data.

The fix is architecturally significant because it decouples the matching logic from a user-managed field (the API label) and anchors it to an immutable identifier (the Vast instance ID). This makes the system more robust: even if labels are never set, instances can still be correctly matched. The C.<id> convention, which was originally just a convenient naming scheme, becomes the backbone of the matching system.

Output Knowledge Created

This message and the debugging session it triggered produced several lasting artifacts. The lookupVast helper function and idMap became part of the vast-manager's core matching logic. The session also produced a clear understanding of Vast.ai's dual-label system — a piece of platform-specific knowledge that is not well documented. Future developers working on this system now know that VAST_CONTAINERLABEL and the API label field are independent, and that matching must be done by instance ID, not by label.

The debugging session also revealed that the killTimedOut function, the bad hosts check, and the dashboard merge all suffered from the same vulnerability. Each was fixed in turn, making the entire system more resilient to label mismatches.

The Broader Lesson

The user's eleven-word message is a perfect example of why concise bug reports are valuable. It identified a contradiction between observed state (instance is running and fetching) and system state (instance is marked as killed) without speculating about the cause. This allowed the assistant to investigate with an open mind, tracing through the code to find the root cause rather than chasing a misdiagnosis.

The bug itself illustrates a recurring pattern in distributed systems: the gap between what a system knows internally and what it exposes externally. Vast.ai knows the instance ID internally (it injects it as VAST_CONTAINERLABEL), but the API exposes it as an optional label field. The manager assumed these were the same thing, and the assumption broke when the optional field was absent. The fix — matching by immutable ID with a fallback — is a pattern that applies broadly to any system that must reconcile internal and external identifiers.