"Should be kept around": The Data Persistence Wake-Up Call That Reshaped a Deployment Manager

"Logs and all instance metadata disappears when an instance is killed, should be kept around"

This seven-word user message ([msg 1377]) lands like a stone in still water. It is brief, almost terse—a complaint about something that used to work but now doesn't. Yet beneath its surface simplicity lies a profound observation about the architectural assumptions underpinning the vast-manager system, a custom deployment and monitoring platform for GPU proving instances on Vast.ai. The message identifies a critical data integrity failure: the entire operational history of a compute instance—its GPU type, geographic location, cost, SSH command, benchmark results—vanishes the moment that instance is destroyed. To understand why this message was written, and why it triggered such a substantial engineering response, we must trace the system's evolution and the implicit contract between operator and tooling.

The Context: A System Built for Live Monitoring

The vast-manager, as it existed before this message, was designed primarily as a real-time monitoring dashboard. Its architecture reflected this focus: the dashboard fetched instance data from two sources—a SQLite database containing operational state (UUID, label, state, benchmark rates, timestamps) and a live cache of the Vast.ai API's show instances output. The dashboard merged these two streams: DB rows provided the state machine progression (registered → params_done → bench_done → running → killed), while the Vast cache provided rich hardware metadata (GPU name, location, cost per hour, SSH command, public IP).

This two-source architecture worked well for live instances. The monitor cycle, running every 60 seconds, would match DB records to Vast API entries by label or instance ID, and the dashboard would render a rich, informative view. But it had a fatal blind spot: killed instances fell through the crack between the two data sources. When an instance was destroyed on Vast.ai—either manually or automatically by the manager's own kill logic (triggered when benchmark rates fell below min_rate)—it disappeared from the Vast API cache. The dashboard, now unable to find a matching Vast entry, could only render the sparse DB columns: a UUID, a label, a state, and a kill reason. The GPU name, location, cost, and SSH command were simply gone.

What the User Actually Saw

The user had just witnessed this data loss firsthand. In the messages immediately preceding this one ([msg 1352]), the assistant had verified the dashboard and found:

Summary: running=0 benching=0 fetching=1 loading=0 killed=9 total=10
  state=registered   vast_id=         0 label=C.32728623           gpu=
  state=killed       vast_id=         0 label=C.32715618           gpu=

Nine killed instances, all with vast_id=0 and empty GPU fields. The metadata had evaporated. The user, who had been actively deploying and managing instances through the UI—setting min-rates, ignoring bad hosts, reviewing benchmark performance—could no longer see what hardware had been killed, where it was located, or what it cost. The dashboard had become a graveyard of anonymous rows.

This was particularly jarring because the previous round of UI improvements had just made the dashboard more informative for live instances. The assistant had added a "Loading" state for pre-registration instances (<msg id=1337-1353>), showing GPU, location, cost, and SSH command from the Vast cache before the instance even contacted the manager. The user could see instances before they registered, but could not see them after they died. The asymmetry was glaring.

The Reasoning: Why This Matters Operationally

The user's message reflects a deep operational need. In a system that automatically provisions, benchmarks, and kills GPU instances based on performance thresholds, the history of killed instances is not mere logging—it is the primary feedback mechanism for tuning the system. When an instance is killed for failing to meet its min_rate, the operator needs to know: What GPU was it? Where was it located? How much did it cost? Was it a pattern—are all instances of this GPU type failing? Are instances in a particular region underperforming?

Without persistent metadata, each killed instance becomes a black box. The operator sees a failure but cannot diagnose it. The bad_hosts and host_perf tables track some of this information at the host level, but the per-instance view—the specific deployment's history—is lost. The user's "should be kept around" is not a casual suggestion; it is an operational requirement for a system that learns from its failures.

The Assistant's Diagnosis: Tracing the Root Cause

The assistant's response ([msg 1378]) immediately identifies the core mechanism of data loss:

"The issue is that killed instances have vast_id=0 because they're no longer in the vast cache (destroyed), so all the vast-sourced metadata (GPU, location, SSH, etc.) is lost. We need to persist the vast metadata into the DB when we first see it."

This diagnosis reveals several assumptions baked into the original design:

  1. The Vast cache was treated as authoritative and permanent. The system assumed that if an instance existed in the DB, it would also exist in the Vast API cache. This was true for live instances but catastrophically false for killed ones.
  2. The DB schema was minimal by design. The instances table stored only what the state machine needed: uuid, label, runner_id, state, min_rate, timestamps, bench_rate, kill_reason. Hardware metadata was considered transient display data, not historical record.
  3. The dashboard merge logic had no fallback. When lookupVast returned no match (as it would for a destroyed instance), the dashboard simply left the metadata fields empty. There was no "if not in cache, check DB" path.
  4. Logs were purely in-memory. The LogBuffer ring buffer stored log entries in process memory, meaning logs survived only as long as the vast-manager process ran. A restart would wipe all historical logs, even for live instances.

The Solution: A Three-Part Data Persistence Strategy

The assistant's implementation (<msg id=1378-1407>) addresses the problem through three coordinated changes:

1. Schema Expansion (12 New Columns)

The instances table grew from a lean operational schema to a rich historical record. The new columns—vast_id, host_id, machine_id, gpu_name, num_gpus, dph_total, geolocation, cpu_name, cpu_ram_mb, gpu_ram_mb, ssh_cmd, public_ip—capture every piece of metadata that the Vast API provides about an instance. The migration was designed to be idempotent: ALTER TABLE ADD COLUMN statements wrapped in error-ignoring logic, so existing databases would upgrade gracefully without data loss.

2. Proactive Metadata Persistence in the Monitor Cycle

Rather than persisting metadata at kill time (which would be too late—the Vast cache entry is already gone), the assistant inserted a new step into the monitor cycle, labeled "Step 0.5," that runs every 60 seconds. For every active (non-killed) DB instance that matches a Vast cache entry, it writes the full metadata to the database. This means metadata is captured continuously throughout the instance's lifecycle, ensuring it survives any subsequent destruction.

3. Dashboard Fallback Logic

The dashboard handler was updated with a two-tier lookup: first try the Vast cache for live metadata, then fall back to the DB-stored metadata for instances that no longer appear in the cache. This ensures that killed instances render with their full hardware profile—GPU name, location, cost, SSH command—exactly as they did when alive.

4. Extended Retention

The cleanup query that purges old killed instances was changed from 24 hours to 7 days, giving operators a full week to review historical data.

Assumptions and Tradeoffs

The solution makes several implicit assumptions worth examining:

The monitor cycle is reliable enough. The approach depends on the monitor running at least once while the instance is alive and the Vast cache contains its metadata. If an instance is created and killed within the same 60-second monitor window, its metadata would never be persisted. This is an edge case—instances typically take minutes to hours to progress through registration, param download, and benchmarking—but it is a real gap.

Logs remain in-memory. The assistant explicitly acknowledges this tradeoff: "Logs are still in-memory (LogBuffer) — they survive as long as the vast-manager process runs, but are lost on restart. That's the current tradeoff; persisting logs to DB could be added later if needed." This is a reasonable scoping decision—persisting logs to SQLite would introduce write amplification and complexity far beyond the metadata problem—but it means the "Logs" part of the user's request is only partially addressed.

The Vast API schema is stable. The 12 columns map directly to Vast.ai's instance JSON structure. If Vast.ai changes their API, the metadata persistence could silently fail (the UPDATE would set columns to NULL or empty values). There is no schema validation or type coercion in the persistence logic.

The Thinking Process Visible in the Reasoning

The assistant's reasoning traces a clear arc from symptom to cause to solution. The initial read of the dashboard code reveals the structural problem: the DashboardInstance struct has fields like gpu_name, geolocation, dph_total that are populated only from the Vast cache merge block. When there's no Vast match, those fields remain at their Go zero values (empty strings, zero floats).

The assistant then examines the monitor cycle to find the right insertion point. The existing monitor has discrete steps: "Step 0: Kill bad hosts," "Step 1: Kill unregistered," etc. The new "Step 0.5" is inserted between them, reflecting its priority—metadata persistence should happen before any kill logic, ensuring that even instances about to be killed have their metadata captured.

The migration strategy shows careful consideration of SQLite's limitations. Since CREATE TABLE IF NOT EXISTS doesn't modify existing tables, the assistant uses ALTER TABLE ADD COLUMN in the NewServer function, with the understanding that SQLite will error if a column already exists. The error is intentionally ignored, making the migration idempotent across restarts and version upgrades.

The Output Knowledge Created

This message and its resolution produced several lasting artifacts:

  1. A persistent instance history that survives instance destruction, enabling post-mortem analysis of failed deployments.
  2. A reusable migration pattern for the vast-manager's SQLite schema, documented in the codebase.
  3. A design principle: metadata should be captured proactively, not reactively, in distributed systems where the source of truth (the Vast API) can disappear without notice.
  4. An acknowledged gap: the log persistence problem is deferred but documented, preventing future confusion about why logs vanish on restart.

Conclusion

The user's seven-word message—"Logs and all instance metadata disappears when an instance is killed, should be kept around"—is a masterclass in effective bug reporting. It identifies a concrete symptom (data disappears), a trigger condition (when an instance is killed), and an expected behavior (should be kept around). It does not prescribe a solution, leaving the implementor to discover the architectural root cause. The assistant's response—tracing the data flow from Vast cache to dashboard, identifying the missing persistence step, and implementing a three-part fix—demonstrates how a clear problem statement enables focused engineering. The result is a system that not only manages instances but remembers them, turning each killed instance from a data loss event into a learning opportunity.