The Metadata Persistence Breakthrough: How One Edit Preserved Instance History in a Vast.ai Management System

The Message

[assistant] Now update the dashboard handler to read the persisted metadata and use it as fallback when the vast cache doesn't have the instance. I need to update the DB query to include the new columns, and update the Instance struct: [edit] /tmp/czk/cmd/vast-manager/main.go Edit applied successfully.

LSP errors detected in this file, please fix: <diagnostics file="/tmp/czk/cmd/vast-manager/main.go"> ERROR [31:12] pattern ui.html: no matching files found </diagnostics>

The Context: A System Built on Ephemeral Data

The vast-manager is a custom-built control plane for managing GPU proving workers rented from Vast.ai, a decentralized cloud marketplace. It handles the full lifecycle: discovering available GPU instances via Vast's API, deploying workloads onto them, monitoring their benchmark performance, and automatically destroying underperforming instances. The system's dashboard provides real-time visibility into every instance's state — registered, loading, benchmarking, bench_done, or killed.

But there was a critical flaw hiding in plain sight. When a Vast.ai instance was destroyed (either manually or automatically by the manager for failing its benchmark), all of its rich metadata vanished from the dashboard. The GPU model, the geographic location, the SSH command, the hourly cost, the public IP address — everything disappeared, replaced by a sparse row showing only the UUID, label, and state. The "killed" view became a graveyard of anonymous entries, offering no insight into what had been lost or why.

The user identified this problem in message 1377 with a simple but devastating observation: "Logs and all instance metadata disappears when an instance is killed, should be kept around."

The Root Cause: A Cache-Only Architecture

The root cause was architectural. The dashboard handler built its response by merging two data sources: rows from the local SQLite instances table, and a live cache of Vast.ai API responses (vastCache). The SQLite table stored only operational fields — UUID, label, runner_id, state, timestamps, benchmark rate. Everything visual — GPU name, location, cost, SSH command — came exclusively from the Vast API cache. When an instance was destroyed on Vast.ai, it disappeared from vastai show instances, the cache was updated, and the dashboard lost all ability to display that information.

The merge logic in the dashboard handler worked like this: for each database instance row, it looked up the corresponding entry in the vast cache by label or ID. If found, it copied over the GPU info, location, cost, and other display fields. If not found — which was the case for all killed instances — those fields remained empty. The instance appeared in the dashboard but with no identifying characteristics beyond its UUID.

The Three-Phase Solution

The assistant devised a three-phase solution, articulated in message 1383:

Schema expansion: Add twelve new columns to the instances SQLite table to store vast-sourced metadata permanently: vast_id, host_id, machine_id, gpu_name, num_gpus, dph_total, geolocation, cpu_name, cpu_ram_mb, gpu_ram_mb, ssh_cmd, public_ip, and status_msg.
Monitor-side persistence: During the periodic monitor cycle (which runs every few seconds and reconciles database state with the vast cache), persist the vast metadata into these new columns whenever a database instance matches a live Vast.ai instance.
Dashboard fallback: In the dashboard handler, when a database instance has no matching entry in the vast cache (i.e., the instance is destroyed), fall back to the persisted metadata columns to populate the display fields. Phases 1 and 2 were already implemented in messages 1385–1390. The schema migration used SQLite's ALTER TABLE (since the CREATE TABLE IF NOT EXISTS statement couldn't modify existing tables), and the monitor cycle gained a new step — inserted between "kill bad hosts" and "kill unregistered" — that called a persistVastMeta function for every non-killed instance.

Message 1392: The Payoff

Message 1392 is the third phase: making the dashboard handler actually read the persisted metadata and use it as a fallback. This is the moment when all the preceding infrastructure work becomes visible to the user.

The edit targeted two things:

First, the database query. The dashboard handler's SQL query against the instances table needed to be updated to SELECT the twelve new columns. Previously, the query selected only the core operational fields. Without this change, the Go sql.Rows scan would never populate the metadata fields even though they existed in the database.

Second, the Instance struct. This Go struct defines the shape of instance data that the dashboard serializes to JSON for the frontend. It needed new fields to receive the persisted metadata. Without this change, the scanned data would have nowhere to go — the struct simply didn't have fields like GpuName, GeoLocation, DphTotal, or SshCmd.

The edit itself was applied with a single [edit] command targeting /tmp/czk/cmd/vast-manager/main.go, and it succeeded. But immediately after, the LSP (Language Server Protocol) integration reported an error: pattern ui.html: no matching files found at line 31. This is a false positive — it refers to a //go:embed ui.html directive that works correctly at build time (the file exists relative to the package directory, not the workspace root where the LSP runs). The assistant would encounter this same spurious error throughout the session and correctly ignore it each time.

The Fallback Logic

The fallback logic itself, implemented in the subsequent messages (1393–1396), worked as follows. The dashboard handler already iterated over database instances and attempted to match each one against the vast cache. If a match was found, the vast cache data was used (as before). But if no match was found — the killed-instance case — the handler now checked the database instance's persisted metadata fields. If those fields were non-empty, they were used to populate the display fields instead of leaving them blank.

This meant that a killed instance would still show its GPU model ("RTX 4090"), its location ("Finland"), its cost ($0.42/hr), and its SSH command — exactly as it did when it was alive. The only difference was that the vast cache fields (like current disk usage or reliability score) would be absent, since those are genuinely ephemeral.

Why This Matters

This fix transformed the killed-instances view from a debugging liability into a historical record. Operators could now see what was killed, where it was, and how much it cost — information essential for making deployment decisions. If a particular GPU model or geographic region was producing poor results, the historical data made that pattern visible. Without it, each killed instance was just a UUID fading into the noise.

The fix also had a subtle but important operational benefit: it eliminated the race condition between instance destruction and data visibility. Previously, if an instance was destroyed between monitor cycles, its metadata was lost forever. Now, the metadata was persisted during the first monitor cycle after registration — long before any destruction could occur. The data was safe.

Assumptions and Subtleties

The solution relied on several assumptions. First, that the vast cache data available during the instance's lifetime was accurate and complete — an assumption validated by the fact that this same data was already being displayed successfully for live instances. Second, that the twelve columns were sufficient to capture all useful metadata — a reasonable bet given that they covered GPU, location, cost, connectivity, and system specs. Third, that the ALTER TABLE migration would work on existing databases — which it did, since SQLite supports additive schema changes.

One subtlety worth noting: the assistant chose to persist metadata only for non-killed instances during the monitor cycle. This means an instance that was killed before its first monitor cycle (a very short-lived instance) would still have empty metadata. In practice, this edge case is unlikely because the monitor runs every few seconds and instances typically live for minutes to hours. But it's a gap worth acknowledging.

Another subtlety: the LogBuffer (in-memory ring buffer for instance logs) was left unchanged. The assistant noted in message 1383 that "logs for killed instances are already there until the process restarts" — acceptable because the vast-manager process is long-lived. But logs are still lost on restart, unlike the metadata which is now persisted permanently.

The Thinking Process

The assistant's reasoning in this message is concise but reveals a clear mental model. The phrase "read the persisted metadata and use it as fallback when the vast cache doesn't have the instance" shows that the assistant understood the architecture's data flow: the dashboard handler was the consumer, the vast cache was the primary source, and the database was now the secondary source. The two changes — updating the query and updating the struct — are the minimal set required to make the database data flow through to the frontend.

The assistant also correctly identified that this was the final piece of a three-phase plan. The todo list from message 1391 shows "Use DB metadata as fallback in dashboard handler for killed/missing instances" as "in_progress" — message 1392 advances it toward completion.

Conclusion

Message 1392 is a small edit with outsized impact. It completes a data persistence pipeline that transforms the vast-manager from a system that forgets its history into one that remembers. The twelve new columns in the database, populated by the monitor and consumed by the dashboard, ensure that every instance — alive or dead — tells its full story. For an operator managing dozens of GPU instances across global regions, this is the difference between flying blind and having a complete operational record.