Persisting Instance Metadata Beyond Death: A Verification Moment in the Vast-Manager Evolution

In the sprawling, multi-session effort to build a comprehensive GPU instance management platform for Vast.ai, few problems are as frustrating as data disappearing when you need it most. The user's complaint was succinct and devastating: "Logs and all instance metadata disappears when an instance is killed, should be kept around" ([msg 1377]). This single sentence launched a significant engineering effort to redesign how the vast-manager system handles the lifecycle of its managed instances. Message [msg 1403] captures the culmination of that effort—a quiet verification moment where the assistant confirms the schema migration succeeded and acknowledges the limits of what can be retroactively recovered.

The Problem: Ephemeral Data in a Distributed System

The vast-manager system operates as a control plane for GPU compute instances rented through Vast.ai. Each instance goes through a lifecycle: registration, parameter fetching, benchmarking, and finally either productive proving work or termination. The system maintains a SQLite database with an instances table tracking state, but critically, the rich metadata about each instance—GPU model, number of GPUs, geographic location, SSH connection command, public IP address, cost per hour, CPU and RAM specifications—was never stored in the database. Instead, it lived only in the in-memory vast cache, populated by periodic API calls to Vast.ai.

When an instance was killed (either because it failed its benchmark, was manually terminated, or was destroyed by the system for underperformance), its vast_id was set to zero because the instance no longer appeared in the Vast.ai API responses. This meant the dashboard could no longer display any of the rich metadata. The instance became a ghost: visible in the killed instances list but stripped of all identifying information. Users could see that an instance had died, but not what had died—no GPU type, no location, no cost data. The logs, stored in a ring buffer in memory, survived only until the vast-manager process restarted.

This was not merely an aesthetic problem. Without persisted metadata, operators could not analyze which types of instances tended to fail, whether certain geographies had higher failure rates, or whether particular GPU models were underperforming. The system was blind to its own history.

The Solution: Schema Migration and Metadata Persistence

The assistant's approach, executed across messages [msg 1378] through [msg 1402], was methodical and multi-layered. Rather than creating a separate metadata table, the assistant chose to extend the existing instances table with twelve new columns: vast_id, host_id, machine_id, gpu_name, num_gpus, dph_total, geolocation, cpu_name, cpu_ram_mb, gpu_ram_mb, ssh_cmd, public_ip, and status_msg. This design decision kept the data model simple—every instance row would carry its own metadata, and the dashboard query could fetch everything in a single SELECT.

The schema change required a migration strategy because SQLite's CREATE TABLE IF NOT EXISTS does not alter existing tables. The assistant added ALTER TABLE statements in the NewServer function, executed after schema creation. These statements use ALTER TABLE ... ADD COLUMN with IF NOT EXISTS-style guards (or rather, they simply attempt the ALTER and silently ignore errors if the column already exists, a common SQLite migration pattern).

The metadata persistence itself was woven into the monitor cycle—the periodic process that checks Vast.ai for instance status. A new step was inserted early in the monitor's pipeline (between killing bad hosts and killing unregistered instances) that iterates all non-killed database instances, matches them against the vast cache, and writes the metadata columns. This ensures that metadata is persisted proactively, not lazily on dashboard access.

The dashboard handler was updated to use the persisted metadata as a fallback. When an instance is still alive in the vast cache, the live data takes precedence. But when the instance is killed and the vast cache has no entry, the dashboard falls back to the database columns. This dual-source approach provides the best of both worlds: fresh data for live instances, persistent data for dead ones.

Finally, the killed instance cleanup retention was extended from 24 hours to 7 days, giving operators a full week to review terminated instances before they are pruned.## The Verification Message: Message 1403

Message [msg 1403] is the moment of truth—the assistant has implemented the schema changes, deployed the new binary, and is now verifying that the migration actually worked. The message contains two distinct actions: a bash command that queries the SQLite schema directly, and the assistant's reflection on what the output means.

The assistant begins with an important acknowledgment: "The existing killed instances don't have metadata yet because they were killed before the metadata persistence was added. That's expected — going forward, any instance that gets matched by the monitor will have its metadata persisted." This statement reveals a critical assumption and a boundary condition. The assistant assumes that the metadata persistence will work for all future instances, but it explicitly recognizes that the past is unrecoverable. The existing killed instances in the database—those with vast_id=0 and empty GPU/location fields—cannot be retroactively populated because the metadata was never captured. The Vast.ai API no longer returns data for destroyed instances, so there is no source of truth to backfill.

This is a pragmatic decision, but it carries implications. If a user needed historical analysis of which GPU types had been killed, they would find the data missing for instances that died before this deployment. The assistant implicitly accepts this tradeoff: the cost of implementing a separate backfill mechanism (perhaps from archived API responses or from the ring-buffer logs) outweighs the benefit, given that the system is still in active development and most instances going forward will have metadata captured.

The Schema Verification

The bash command sqlite3 /var/lib/vast-manager/state.db ".schema instances" is a direct, low-level inspection of the database. The assistant could have relied on the Go code's migration logic to report success, but instead chose to verify at the SQLite level—a more trustworthy approach that bypasses any application-layer bugs. The output confirms that the columns were added:

CREATE TABLE instances (
	uuid           TEXT PRIMARY KEY,
	label          TEXT NOT NULL,
	runner_id      INTEGER UNIQUE NOT NULL,
	state          TEXT NOT NULL DEFAULT 'registered',
	min_rate       REAL NOT NULL DEFAULT 50,
	registered_at  TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
	param_done_at  TIMESTAMP,
	bench_done_at  TIMESTAMP,
	bench_rate     REAL,
	killed_at      TIMESTAMP,
	kill_reason    TEXT
, vast_id INTEGER, host_id INTEGER, machine_id INTEGER, gpu_name TEXT, num_gpus INTEGER, ...

The output is truncated in the message (the ... at the end), but the critical detail is visible: the new columns appear after the original schema, appended by the ALTER TABLE migration. The comma before vast_id is notable—it's a SQLite artifact of how ALTER TABLE appends columns, but it doesn't affect functionality.

Assumptions and Their Validity

Several assumptions underpin this message and the work it verifies:

Assumption 1: The monitor cycle will run before any instance is killed. The metadata persistence happens in the monitor, which runs periodically (every 60 seconds by default). If an instance is killed between registration and the next monitor cycle, its metadata will never be captured. The assistant implicitly assumes this race condition is unlikely or acceptable. In practice, instances spend minutes to hours in the registered/params_done/bench_done states before being killed, so the monitor almost certainly runs at least once.

Assumption 2: The vast cache is always populated when the monitor runs. The monitor fetches the vast cache at the start of each cycle. If the Vast.ai API is unreachable or returns an error, the cache will be empty, and no metadata will be persisted. The assistant's code handles this gracefully—it simply skips the metadata step if the cache is empty—but the instance will remain metadata-less until the next successful cycle.

Assumption 3: The ALTER TABLE migration is idempotent. The migration code attempts to add columns that may already exist (if the schema was already migrated). The assistant relies on SQLite's behavior of silently ignoring ALTER TABLE errors when the column already exists. This is correct for SQLite, but it's worth noting that the migration code doesn't check for errors—it simply executes the ALTER statements and moves on.

Assumption 4: The dashboard fallback is sufficient. When a killed instance has no vast cache entry, the dashboard uses the DB metadata. But the DB metadata might be stale if the instance was alive when last seen but its metadata changed (e.g., GPU memory usage fluctuated). The assistant correctly prioritizes live cache data over DB data, so this is only a concern for killed instances.

What This Message Reveals About the Engineering Process

This message is notable for what it doesn't contain: there is no triumphant announcement, no detailed explanation of the code changes, no user-facing description of the new feature. Instead, it is a quiet, technical verification—the kind of message that an engineer writes to themselves as much as to anyone else. The assistant is confirming that the migration worked, acknowledging the limitation (old instances won't have metadata), and implicitly closing the loop on the user's original complaint.

The message also reveals the assistant's disciplined approach to deployment. Rather than assuming the code works because it compiled, the assistant deploys the binary, restarts the service, and then independently verifies the schema using a different tool (sqlite3 CLI) than the one that created it (Go's database/sql). This double-checking is characteristic of production engineering: trust the code, but verify the data.

Output Knowledge Created

This message produces concrete, verifiable knowledge:

  1. The schema migration succeeded. The instances table now has the new metadata columns. This is confirmed by direct SQLite inspection.
  2. The existing killed instances are metadata-less. The earlier dashboard query showed vast_id=0 and empty GPU/location fields for all instances, confirming that the metadata columns exist but are NULL for pre-existing rows.
  3. The system is now in a state where future instances will have metadata persisted. The monitor cycle will capture metadata for any instance it sees, and the dashboard will fall back to DB data for killed instances.
  4. The deployment pipeline works. The build, scp, stop, move, start, and verify sequence completed successfully, demonstrating that the continuous deployment process is functional.

Conclusion

Message [msg 1403] is a small but crucial verification step in a larger engineering effort. It answers the question "Did it work?" with a concrete yes, while honestly acknowledging the boundary conditions. The assistant's decision to verify at the SQLite level, rather than trusting the application code, reflects a production-oriented mindset. The acknowledgment that old instances are unrecoverable sets appropriate expectations. And the implicit assumptions about monitor timing, cache availability, and migration idempotency reveal the engineering tradeoffs that underpin the solution.

In the broader context of the vast-manager system, this message marks the transition from implementation to verification. The feature is deployed, the schema is confirmed, and the system is ready to accumulate metadata for all future instances. The ghosts of the past remain ghosts, but the future is recorded.