The Schema That Remembers: Preserving Instance Metadata After Death in vast-manager

In the lifecycle of any distributed system that manages ephemeral cloud instances, the moment an instance is destroyed is also the moment its history is most valuable. When a GPU worker on Vast.ai gets killed—whether because it failed its benchmark, underperformed, or was deliberately terminated—the operator needs to know what was lost: which GPU model, in which datacenter, at what cost, and with what performance record. Without this information, capacity planning becomes guesswork, and diagnosing failures becomes archaeology.

Message [msg 1387] is a single, deceptively simple edit confirmation in the vast-manager coding session. Its content is brief:

[assistant] [edit] /tmp/czk/cmd/vast-manager/main.go
Edit applied successfully.

LSP errors detected in this file, please fix:
<diagnostics file="/tmp/czk/cmd/vast-manager/main.go">
ERROR [31:12] pattern ui.html: no matching files found
</diagnostics>

Yet this message represents the successful application of a critical data persistence fix—one that transforms the vast-manager from a system that forgets its dead instances into one that remembers them. The edit adds SQLite ALTER TABLE migration statements to the NewServer function, extending the instances table with a dozen new columns that capture Vast.ai metadata. This is the infrastructure equivalent of giving a graveyard headstones instead of unmarked graves.

The Problem: Ephemeral Cache, Ephemeral Knowledge

The context leading to this message begins with a user complaint at [msg 1377]: "Logs and all instance metadata disappears when an instance is killed, should be kept around." This was not a cosmetic issue—it was a fundamental data integrity problem in the vast-manager's architecture.

The assistant investigated and traced the root cause. The vast-manager's dashboard sourced its instance metadata (GPU name, number of GPUs, hourly cost, geographic location, SSH command, public IP, status message) exclusively from the live Vast.ai API cache. When an instance was destroyed on Vast.ai, it disappeared from the vastai show instances output. The next time the dashboard rendered, that instance showed vast_id=0 with empty GPU and location fields—all the identifying information was gone. The database instances table had only the bare minimum columns: uuid, label, runner_id, state, min_rate, and timestamps. Everything else was ephemeral, fetched fresh each monitor cycle and never persisted.

This design had a subtle but serious consequence: the system could not distinguish between a destroyed RTX 4090 that had benchmarked at 45 proofs/hour and a destroyed RTX A4000 that had benchmarked at 12 proofs/hour. Both became identical rows with vast_id=0, gpu_name="", and no cost data. Historical analysis, operator trust, and informed decision-making all suffered.

The Architecture of the Fix

The assistant's response, visible in the reasoning at [msg 1383], laid out a five-point plan:

Add vast metadata columns to the instances table — a comprehensive set of fields including vast_id, host_id, machine_id, gpu_name, num_gpus, dph_total, geolocation, cpu_name, cpu_ram_mb, gpu_ram_mb, ssh_cmd, public_ip, and status_msg.
Persist vast metadata during the monitor cycle — when a database instance is matched to a Vast.ai instance, UPDATE the row with the current metadata if it hasn't been set yet (or if it has changed).
Use DB metadata as fallback in the dashboard — when rendering instances that are no longer in the Vast cache (killed or destroyed), fall back to the persisted metadata instead of showing empty fields.
Extend killed instance retention from 24 hours to 7 days — giving operators a full week to review historical data before cleanup.
Accept in-memory log limitations — the LogBuffer is in-memory, so logs survive until the process restarts. This was deemed acceptable for the current architecture. This plan reflects a mature understanding of the system's data flow. The assistant recognized that the Vast.ai API cache was the source of truth during an instance's lifetime, but the SQLite database needed to become the persistent record after death.

The SQLite Migration Challenge

Message [msg 1385] reveals a key technical decision. The assistant noted: "Since this is SQLite with CREATE TABLE IF NOT EXISTS, I can't just add columns to the schema—existing tables won't be altered. I'll use a migration approach: add the columns via ALTER TABLE in the NewServer function after schema creation."

This is a critical insight about SQLite's behavior. The schema constant in the Go source uses CREATE TABLE IF NOT EXISTS, which only creates the table if it doesn't already exist. For a fresh deployment, adding columns to the schema definition would work fine—the table would be created with all the new columns. But for existing databases (which already have the old schema), the CREATE TABLE IF NOT EXISTS is a no-op. The new columns would never be added.

The correct approach, which the assistant implemented, is to add ALTER TABLE statements that run after the schema creation, in the NewServer function. These statements add the new columns to the existing table. The assistant used a pattern of ALTER TABLE instances ADD COLUMN IF NOT EXISTS (though SQLite doesn't natively support IF NOT EXISTS for ALTER TABLE—the assistant likely wrapped each statement in a way that catches the "duplicate column" error, or used a different approach to check existence).

Message [msg 1386] shows the assistant reading the NewServer function to find the right insertion point. Then message [msg 1387] confirms the edit was applied successfully.

The LSP Error: A Distraction, Not a Bug

The LSP diagnostic in message [msg 1387] reports ERROR [31:12] pattern ui.html: no matching files found. This is a pre-existing issue unrelated to the schema migration. It appears in multiple earlier messages ([msg 1346], [msg 1385]) and refers to a Go source file that references an embedded ui.html template file at line 31. The LSP (likely gopls) cannot find this file because the project is being edited in a temporary directory (/tmp/czk/) where the ui.html file may not exist at the expected relative path, or because the file is embedded via a build step that hasn't run yet.

The assistant's response to this error is notable: it does not attempt to fix it in this message. This is a deliberate prioritization. The LSP error is a development-environment issue (missing file in the workspace), not a runtime bug. The schema migration is the critical task at hand. The assistant correctly distinguishes between a blocking error and a cosmetic one, and proceeds with the deployment.

Assumptions and Their Validity

The assistant made several assumptions in this work:

That the ALTER TABLE migration would be safe for existing databases. This is reasonable for SQLite, where ALTER TABLE ADD COLUMN is a well-supported operation. However, SQLite does have restrictions: the new column cannot have a NOT NULL constraint (unless a default is provided), cannot have a PRIMARY KEY or UNIQUE constraint, and cannot be REFERENCES another table. The assistant's new columns (text fields, integer fields, real fields) all comply with these restrictions.

That the metadata from the Vast.ai cache is trustworthy and complete. This is generally true, but the assistant implicitly assumes that the cache will always have the fields it expects. If Vast.ai's API changes its response format, the metadata persistence could fail silently. A more robust approach might include validation or default values.

That the ui.html LSP error is harmless. This assumption proved correct—the error appears repeatedly throughout the session without causing runtime issues. The ui.html file is referenced in the Go source via //go:embed or a similar mechanism, and the actual file exists at build time even if the LSP can't find it during editing.

Input Knowledge Required

To understand this message, one needs:

SQLite schema migration patterns: Knowing that CREATE TABLE IF NOT EXISTS doesn't alter existing tables, and that ALTER TABLE is needed for schema evolution.
The vast-manager architecture: Understanding that instance metadata was previously sourced only from the live Vast.ai cache, not persisted in the database.
The monitor cycle: Knowing that the vast-manager periodically fetches vastai show instances and matches instances by label or ID.
The dashboard rendering pipeline: Understanding that the dashboard handler constructs instance objects by merging DB data with cache data, and that a cache miss leaves fields empty.
Go LSP behavior: Recognizing that the pattern ui.html: no matching files found error is a workspace configuration issue, not a code defect.

Output Knowledge Created

This message produces:

A migration path for existing databases: The ALTER TABLE statements in NewServer ensure that when the new binary starts, existing databases get the new columns automatically.
A foundation for historical visibility: Once the full pipeline is implemented (metadata persistence in the monitor cycle, fallback in the dashboard), operators will be able to see GPU model, location, cost, and performance for killed instances for up to 7 days.
A pattern for future schema evolution: The migration approach (schema const for new tables + ALTER TABLE for existing ones) becomes the standard pattern for any future schema changes.

The Broader Significance

This message, for all its brevity, represents a shift in the vast-manager's design philosophy. The system was originally built as a live monitor—it showed what was happening right now, and the past was irrelevant. The user's complaint at [msg 1377] exposed the limitations of this worldview. Operators need to learn from history, not just observe the present.

The schema migration is the enabling infrastructure for that learning. Without it, every killed instance is a black box. With it, the vast-manager becomes a system that accumulates knowledge over time—which GPU models perform well in which regions, which hosts are unreliable, what price-to-performance ratio is achievable. This is the difference between a monitoring tool and a management platform.

The assistant's decision to persist 12 separate metadata fields, rather than just a few, reflects an understanding that you never know in advance which data will be important. The GPU name matters for capacity planning. The location matters for latency analysis. The cost matters for budgeting. The SSH command matters for debugging. By storing all of it, the system future-proofs itself against questions that haven't been asked yet.

Conclusion

Message [msg 1387] is a single edit confirmation—one line of output indicating that a code change was applied. But in the context of the vast-manager's evolution, it is the moment the system stopped treating instance death as an information erasure event. The schema migration it represents is the bedrock upon which historical analysis, operator trust, and data-driven decision-making are built. It is a small edit with outsized consequences, and it exemplifies the kind of infrastructure work that makes a system not just functional, but knowledgeable.