Persisting Instance Metadata Beyond Death: A Critical Data Architecture Fix in vast-manager

In message [msg 1394] of this opencode session, the assistant performs a seemingly mundane operation—reading a section of Go source code—that reveals the core architectural challenge of building a reliable deployment management system for ephemeral cloud GPU instances. The message captures the exact moment when the assistant transitions from theory to implementation, reading the "merge section" of the vast-manager dashboard handler to understand where to insert fallback logic for killed instances. This single read operation is the keystone of a multi-step data persistence fix that transforms the vast-manager from a system that forgets its history into one that retains critical operational metadata even after instances are destroyed.

The Problem: Ephemeral Instances and Lost Knowledge

The context for this message begins at [msg 1377], where the user states plainly: "Logs and all instance metadata disappears when an instance is killed, should be kept around." This complaint strikes at a fundamental tension in the vast-manager's design. The system was originally built around the Vast.ai API as the authoritative source of instance information—GPU type, location, cost, SSH commands, and other metadata were all fetched live from the Vast.ai cache during each dashboard render. When an instance was running, this worked perfectly. But the moment an instance was killed (destroyed on Vast.ai), the cache no longer contained it, and all that metadata vanished from the dashboard. Killed instances became ghost entries with empty fields, their logs still in memory but disconnected from any identifying context.

The assistant's response reveals a sophisticated understanding of the data architecture. Rather than patching the symptom, the assistant traces the root cause through the entire data flow: the dashboard handler loads DB instances, then merges them with vast cache data. If the cache doesn't have the instance (because it was killed), the merge produces empty metadata fields. The fix requires a multi-layered approach spanning schema changes, monitor-cycle persistence, dashboard fallback logic, and retention policy adjustments.

Reading the Merge Section: The Architectural Pivot

Message [msg 1394] is the fourth step in a carefully sequenced plan. The assistant has already:

Added 12 new metadata columns to the SQLite instances table schema (<msg id=1385-1387>)
Written a migration to add these columns to existing databases ([msg 1387])
Implemented a persistVastMeta function in the monitor cycle that saves vast cache data to the DB when instances are first matched ([msg 1390])
Updated the Instance struct and dashboard query to include the new columns (<msg id=1392-1393>) Now, in message [msg 1394], the assistant reads the merge section of the dashboard handler to understand exactly where the fallback logic must be inserted. The code shown reveals the current architecture:

// Load vast cache — build both label and ID maps for matching
s.vastCacheMu.RLock()
vastMap := make(map[string]VastInstance)
vastIDMap := make(map[int]VastInstance)
for _, vi := range s.vastCache {
    if vi.Label != "" {
        vastMap[vi.Label] = vi
    }
    vastIDMap[vi.ID] = vi
}

This section builds two lookup maps from the vast cache—one keyed by label (the human-readable name like "C.12345") and one keyed by numeric Vast.ai instance ID. After this, the code iterates over DB instances and tries to find a matching vast instance in these maps. If found, it overlays the live vast data (GPU name, location, cost, etc.) onto the DB record. If not found (killed instance), the vast metadata fields remain whatever was stored in the DB—which, before the fix, was nothing.

The assistant's reasoning is clear: the merge section is the single point where DB and vast data are combined, so it's the natural place to add the fallback. After reading this section, the assistant will modify the merge logic to check: "If no vast cache match is found, use the persisted DB metadata columns as fallback instead of leaving them empty."

Assumptions and Design Decisions

The assistant makes several key assumptions in this approach. First, it assumes that the vast cache is always the authoritative source for live instance data—GPU name, location, cost, SSH command, etc.—and that DB-persisted metadata is a suitable fallback only when the instance is no longer in the cache. This is reasonable for the use case (killed instances), but it means that if an instance's GPU configuration somehow changes while running (unlikely but possible with some cloud providers), the DB metadata could become stale.

Second, the assistant assumes that the monitor cycle runs frequently enough to capture metadata before instances are killed. The persistVastMeta function is called during each monitor cycle, which runs every few minutes. If an instance is deployed and killed within seconds (unlikely given benchmark times of 20-45 minutes), the metadata might never be persisted. This is a pragmatic trade-off—the monitor cycle is the natural synchronization point, and adding per-instance persistence on deploy would add complexity.

Third, the assistant assumes that the SQLite database is the right place for this metadata. SQLite is embedded in the vast-manager binary and requires no external infrastructure, making it ideal for a single-controller deployment. However, it means the metadata is only as durable as the controller host's disk.

The Thinking Process Visible in the Message

What makes message [msg 1394] particularly interesting is what it reveals about the assistant's methodical approach to complex changes. The assistant doesn't jump directly into editing the merge section. Instead, it first reads the exact code it needs to modify, understanding the existing structure before making changes. This is visible in the [read] tool call—the assistant is deliberately gathering context.

The message also shows the assistant working through a prioritized todo list. Earlier messages in this sequence show [todowrite] blocks with items like "Add vast metadata columns to instances table schema" (status: completed), "Persist vast metadata in monitor cycle when matched" (status: completed), "Use DB metadata as fallback in dashboard handler for killed/missing instances" (status: in_progress). The assistant is systematically working through these items, and message [msg 1394] represents the transition from "in_progress" to the actual implementation of the third item.

Input and Output Knowledge

To understand this message, the reader needs knowledge of: the Go programming language (specifically SQLite usage with database/sql), the Vast.ai API structure (instances have labels, IDs, GPU metadata, location, cost), the vast-manager's architecture (monitor cycle, dashboard handler, merge logic), and the concept of ephemeral cloud instances that can be destroyed at any time.

The message creates output knowledge about the merge section's current implementation—specifically how vastMap and vastIDMap are built from the cache, and where the fallback logic needs to be inserted. This knowledge directly informs the subsequent edit that the assistant will make.

Broader Significance

This message, though brief, represents a critical architectural insight: in systems that manage ephemeral resources, data persistence must be designed proactively, not reactively. The vast-manager was originally built with the assumption that instance metadata would always be available from the live API. The user's complaint revealed this assumption was wrong, and the assistant's response—spanning schema design, migration strategy, monitor-cycle persistence, dashboard fallback, and retention policy—demonstrates a comprehensive understanding of what it takes to make ephemeral instance data durable.

The fix ensures that when an operator looks at the dashboard and sees a killed instance, they still see the GPU type, location, cost, and SSH command that instance had when it was alive. This transforms the dashboard from a live-monitoring tool into a historical record, enabling post-mortem analysis of failed instances and better decision-making about which hardware to deploy on next.