The Provenance Problem: Why an Autonomous GPU Fleet Agent Needed a Performance Memory

In the middle of building an autonomous LLM-driven fleet management agent for a distributed GPU proving cluster, the user delivered a message that would fundamentally reshape the agent's decision-making architecture. The message was concise but carried profound implications:

"Also agent should prefer instances which we saw working, especially ones that we've seen perform in curio. Agent should, on cron, check on instances - proof/error counts in curio, write to a ~100 line .md file (rewrite/edit tools) with instance summary that is used to weigh whether the instance is good"

This single message, sent at a critical juncture in the conversation, introduced the concept of provenance into an agent that had until that point been making decisions based entirely on surface-level metrics: marketplace prices, benchmark rates, and instantaneous demand snapshots. The user was asking the agent to develop institutional memory — to remember which machines had actually delivered in production and to use that knowledge to make better decisions.

The Context: An Agent Learning to See

To understand why this message was written, we must understand the conversation's trajectory. The user and assistant had been building an autonomous agent to manage a fleet of GPU instances on vast.ai, running Curio SNARK proving workloads. The agent's initial design was naive: it monitored pending task counts and launched instances reactively. But the user quickly identified a fundamental flaw — pending tasks are volatile and meaningless for a workload where each machine churns out proofs every 30 seconds and instance startup takes 1–2 hours. The agent had already launched a wasteful instance based on a momentary queue spike of 8 tasks that would be cleared in 4 minutes ([msg 4430], [msg 4431]).

The conversation had just pivoted to fixing this. The assistant was in the middle of rewriting the Go API endpoints to expose sustained demand metrics (15-minute and 1-hour completion windows) instead of instantaneous pending counts, and updating the Python agent's system prompt to focus on capacity targets rather than queue depths ([msg 4433], [msg 4435]). The user had given clear, simple rules: scale up to 500 proofs/hour capacity when there's active demand, scale down after an hour of inactivity ([msg 4432]). No complex control systems.

But this message revealed that the user was thinking one step ahead. Even with correct scaling logic, the agent could still make bad decisions — it could launch an instance on a machine that looks good on paper (low price, high benchmark) but crashes in production, wastes the startup cost, and contributes nothing. The user recognized that the agent needed not just what to scale, but which machines to trust.

The Core Insight: Benchmarks Lie, Production Doesn't

The user's message reflects a deep understanding of the cloud GPU rental market. On vast.ai, marketplace listings advertise GPU models, RAM, and benchmark scores, but these are unreliable proxies for actual production performance. A machine might have stellar benchmarks but unstable drivers, flaky networking, or a GPU that throttles under sustained load. The only reliable signal is whether the machine has actually completed proofs in Curio without errors.

This is the provenance problem: how does an autonomous agent distinguish between a machine that looks good and a machine that is good? The user's answer was elegantly simple — have the agent check Curio's production data on every cron cycle, extract per-instance proof and error counts, and maintain a running performance summary in a markdown file that the LLM can read before making launch decisions.

The mechanism is noteworthy. The user specified "write to a ~100 line .md file (rewrite/edit tools)" — this reveals an understanding of the LLM's tool-calling interface and its limitations. A markdown file is a natural format for an LLM to consume: structured enough for tables and summaries, unstructured enough for narrative context. The ~100 line limit suggests a desire for conciseness — enough data to be useful, not so much that it overwhelms the context window. The mention of "rewrite/edit tools" shows the user knows the agent can use file operations to maintain this document incrementally.

Assumptions Embedded in the Message

The message makes several assumptions worth examining. First, it assumes that Curio's database contains per-machine completion and error data that can be reliably queried. This is plausible — Curio tracks task history with completed_by_host_and_port fields — but the mapping between vast.ai instance identifiers and Curio's internal machine names is non-trivial. Vast.ai instances are ephemeral containers with dynamically assigned hostnames; correlating them with Curio's persistent machine registry requires careful engineering.

Second, the message assumes that a ~100 line markdown file is the right persistence mechanism. This is a pragmatic choice for an LLM-driven system — it keeps the data in plain text, versionable, and directly consumable by the model. But it also means the agent must handle file I/O, concurrent access, and corruption risks, adding operational complexity.

Third, the user assumes the LLM can effectively weigh this performance data when making launch decisions. This is a non-trivial prompt engineering challenge: the model must understand that a machine with 10,000 successful proofs and 0 errors is preferable to one with 100 proofs and 5 errors, even if the latter is cheaper. The user's phrasing — "used to weigh whether the instance is good" — suggests they expect the LLM to apply this judgment autonomously, not through hard-coded rules.

The Thinking Process: From Reactive to Proactive to Principled

The progression visible in this conversation is instructive. The user started with a reactive approach (scale on pending tasks), quickly learned it was wrong, pivoted to a proactive approach (scale on sustained demand targets), and now was pushing toward a principled approach (prefer proven machines). Each iteration added sophistication without adding complexity — the rules remained simple, but the information available to the agent became richer.

This message also reveals the user's mental model of the agent. They saw it not as a script with if-else logic but as a reasoning entity that could use tools (check Curio, write a file, read a file) and make contextual judgments. The instruction to "prefer instances which we saw working" is a high-level goal, not a detailed algorithm — the user trusted the LLM to figure out the weighting, given the right data.

The Output Knowledge Created

This message generated a concrete specification for a new subsystem: the agent performance tracker. The assistant's response ([msg 4443]) immediately recognized the scope of work: querying Curio's harmony_task_history for per-machine completion data, mapping hostnames to vast instance IDs, building a /api/agent/perf endpoint in Go, updating the Python agent to write and read the markdown summary, and integrating the performance data into the LLM's launch decision context. The todo list expanded with items for "per-machine Curio completion/error counts" and "prefer proven machines" — direct descendants of this message.

The message also implicitly created a design principle that would echo through subsequent chunks: the agent should be evidence-driven, not speculation-driven. This principle would later manifest in the diagnostic grounding sub-agent system (chunk 4), where the agent was required to gather factual evidence before making destructive decisions. The performance markdown file was the first instance of this pattern — a persistent, verifiable record of reality that the agent could reference instead of guessing.

Why This Message Matters

In the broader arc of the conversation, this message marks the transition from a naive scaling agent to a sophisticated fleet manager with institutional memory. It introduced the concept that an autonomous system must not only observe the present but remember the past — and that memory must be structured, persistent, and actionable. The markdown file became the agent's hippocampus, encoding which machines had earned trust and which had not.

For anyone building autonomous infrastructure management systems, this message encapsulates a critical lesson: the most valuable data is often not the data you can buy (marketplace listings, benchmark scores) but the data you generate yourself (production performance, error rates). An agent that cannot distinguish between a proven worker and an unknown gamble is not truly autonomous — it is just guessing with someone else's money.