The System Prompt That Saved the Fleet: Teaching an LLM Agent to See Future Capacity

The Message

Now fix the system prompt — add projected capacity awareness and rate-limit handling: [edit] /tmp/czk/cmd/vast-manager/agent/vast_agent.py Edit applied successfully.

This three-line message, dispatched by the assistant in message [msg 4498], is deceptively brief. On its surface, it records a routine file edit—the third in a sequence of changes to a Python agent script. But this message represents the culmination of a critical diagnostic arc that transformed an over-provisioning, iteration-wasting autonomous fleet manager into a cost-aware, capacity-aware agent capable of making sound scaling decisions. To understand why this message matters, one must trace the chain of reasoning that led to it, the assumptions it corrected, and the architectural insight it encoded.

The Crisis: An Agent That Could Not Do the Math

The story begins with the user's simple directive in [msg 4486]: "Look at how the agent is doing, if what it's doing makes sense, maybe review prompt optimality." The assistant responded by gathering forensic data from the production fleet. The results were alarming.

The agent had launched eight instances in rapid succession. Yet only one was actually running and producing proofs (at 40 proofs/hour). Six were still in the loading/bootstrap phase, and one was registered but not yet provisioned. The fleet was spending $3.32/hour for a meager 40 proofs/hour of actual output—a terrible return on investment. And the agent was still trying to launch more, burning through all five of its LLM iterations attempting to work around the rate limit instead of recognizing when to stop.

The root cause was stark: the agent saw capacity_proofs_hr = 40 and target_proofs_hr = 500, computed a gap of 460, and dutifully launched instances to fill it. It had no way to account for the six loading instances that would, within 1–2 hours, add roughly 300 proofs/hour of capacity. The LLM was making decisions based on incomplete information, and the system prompt—the very text that governs the agent's reasoning—contained no instruction to consider pending capacity.

The Diagnostic Chain

Message [msg 4498] is the third link in a three-part fix chain. The assistant's reasoning, visible in [msg 4489], shows a clear diagnostic process:

  1. Observe the symptom: The agent keeps launching despite six loading instances.
  2. Identify the root cause: The LLM sees only capacity_proofs_hr (40) and doesn't mentally add projected capacity from loading instances.
  3. Design the fix in three layers: - Layer 1 (Go API): Add a projected_proofs_hr field to the fleet endpoint that combines running capacity with estimated capacity from loading instances, using average benchmark rates. This was done in [msg 4489]. - Layer 2 (Python fast-path): Rewrite the fast-path logic in vast_agent.py to use projected_proofs_hr instead of capacity_proofs_hr for the "no action needed" decision. This was done in [msg 4497]. - Layer 3 (System prompt): Update the LLM's system prompt to explicitly describe projected capacity and instruct the agent not to launch when projected capacity already meets the target. This is the subject of [msg 4498]. The layered approach is significant. The fast-path logic handles the common case without invoking the LLM at all—if projected capacity already meets the target, the agent skips the LLM entirely and reports "no action needed." But the LLM path still needed to be fixed for cases where the fast-path doesn't apply (e.g., when projected capacity is close to the target but the agent needs to decide whether to launch one more instance, or when instances need to be stopped). The system prompt fix ensures that even when the LLM is invoked, it has the right information and rules.

Assumptions and Their Corrections

The original agent design made several assumptions that this message corrected:

Assumption 1: The LLM can infer projected capacity from the observation string. The observation string listed running and loading instance counts, but the LLM proved unable to multiply "6 loading instances × ~50 proofs/hour each" and add it to the running capacity. The fix made this calculation explicit in the fleet API response and the prompt.

Assumption 2: The LLM will stop launching when rate-limited. In practice, the agent received a 429 error and simply tried a different offer on the next iteration, wasting all five iterations on futile retries. The system prompt now includes a directive to accept rate limits and stop.

Assumption 3: Raw capacity numbers are sufficient for decision-making. The agent needed not just numbers but projections—estimates of future state. This is a fundamental insight about autonomous agents: they need the same forward-looking information that a human operator would use.

Input Knowledge Required

To understand this message, one needs to know:

Output Knowledge Created

This message produced a corrected system prompt that:

  1. Explicitly states the projected capacity (running + loading) alongside current capacity
  2. Instructs the agent not to launch new instances if projected capacity already meets the target
  3. Directs the agent to accept rate limits gracefully rather than retrying
  4. Provides the agent with the forward-looking information needed for sound scaling decisions

The Thinking Process

The assistant's reasoning, visible in [msg 4489], shows a methodical diagnostic approach. It didn't jump to conclusions or apply a generic fix. It examined journal logs, queried the fleet API, inspected the SQLite database, and counted instances on the vast.ai platform. It identified not just the surface symptom (over-provisioning) but the underlying mechanism (the LLM couldn't do the mental math). And it designed a fix that addressed the problem at three levels: data availability (Go API), automated logic (Python fast-path), and LLM reasoning (system prompt).

The system prompt fix in [msg 4498] is the keystone of this architecture. Without it, the fast-path would handle the common case, but any situation requiring LLM reasoning would still suffer from the original blindness to future capacity. With it, the agent has a complete picture of the fleet—not just what it is now, but what it will be in 1–2 hours—and can make decisions that a human operator would recognize as sensible.

Conclusion

Message [msg 4498] is a testament to the principle that autonomous agents are only as good as the information they receive. The assistant recognized that the LLM wasn't failing due to poor reasoning but due to incomplete data. By adding projected capacity to the system prompt, it gave the agent the one piece of information it needed to stop over-provisioning. The fix was small in code but large in impact: it transformed an agent that was burning money on unnecessary instances into one that could see the future and act accordingly.