The Projected Capacity Fix: Teaching an Autonomous Agent to See the Future

Introduction

In the course of building an autonomous LLM-driven fleet management agent for a distributed GPU proving infrastructure, a critical flaw emerged that threatened to bankrupt the operation through over-provisioning. The agent, tasked with scaling a fleet of GPU instances to meet a target of 500 proofs per hour, had launched eight instances when it only needed two or three. The problem was not malice or incompetence—it was a failure of perception. The agent could only see the present, not the future.

Message 4506 in this conversation is the assistant's summary of the diagnosis and fix for this over-provisioning crisis. It is a compact, three-paragraph post-mortem that distills hours of debugging, architectural reasoning, and cross-language implementation into a clean narrative. But beneath its brevity lies a deep lesson about the design of autonomous systems: an agent that cannot project its own future state will inevitably overcorrect, waste resources, and frustrate its human operators.

The Context: An Agent Built to Scale

The autonomous fleet agent, written in Python and running as a systemd timer on a management host, was designed to observe the state of a distributed GPU proving cluster and make scaling decisions using a large language model (Qwen 3.5-122b). The agent had access to a suite of tools: it could launch instances on vast.ai, stop them, check demand, and inspect fleet state. Its primary objective was to maintain enough capacity to handle the Curio SNARK demand, targeting 500 proofs per hour.

The system had been through several iterations of hardening. A persistent conversation log in SQLite maintained context across runs. A diagnostic sub-agent could SSH into instances to collect raw logs. A verdict system structured the agent's output. But despite all this sophistication, the agent was making a rookie mistake: it was scaling based on current capacity alone, ignoring the half-dozen instances already booting up.

The Diagnosis: What the Agent Couldn't See

The user's prompt that triggered this investigation was simple and open-ended: "Look at how the agent is doing, if what it's doing makes sense, maybe review prompt optimality" ([msg 4486]). The assistant responded by pulling agent logs and fleet state. What it found was alarming.

The fleet endpoint reported: "1 running, 6 loading (will add capacity in 1-2h), 40 proofs/hr capacity." The agent saw that 40 p/h gap to the 500 target and kept launching. It had already launched eight instances. Seven were still loading—booting, benchmarking, registering—and would each contribute roughly 50 p/h once ready. That meant ~390 p/h was already on its way. The agent just couldn't see it.

The assistant's reasoning in [msg 4489] is worth quoting directly, as it reveals the thinking process:

"The core issue is that the agent doesn't account for loading instances as future capacity — it sees the 40 p/h gap and keeps launching more, when those 6 loading instances will eventually provide ~300 p/h. It also burns through all its LLM iterations trying to work around the rate limit instead of recognizing when to stop, and it ignores the 1-2 hour startup window mentioned in the prompt."

This is a classic systems design problem: the agent's observation of the world was accurate but incomplete. It knew about running capacity but had no concept of pending capacity—instances that were already paid for and booting but not yet producing proofs. The agent was operating with a snapshot of the present and using it to make decisions about the future, which in a system with 1-2 hour startup latencies is a recipe for disaster.

Three Problems, Three Fixes

Message 4506 enumerates three problems and three corresponding fixes. The problems are:

1. The agent launched eight instances because it only saw 40 p/h capacity (running), ignoring seven loading instances.
2. It wasted all five LLM iterations retrying launches with different offers after hitting the rate limit.
3. The system prompt did not instruct the LLM to account for loading instances' projected capacity. The fixes spanned two codebases—Go and Python—reflecting the architecture's split between the backend API server and the agent logic.

The Go Fix: Adding projected_proofs_hr

The first fix was in agent_api.go, the Go backend that serves the fleet endpoint. The assistant added a projected_proofs_hr field to the FleetTotals struct. This field computes projected capacity as the sum of current running capacity plus an estimate for each loading instance (defaulting to 50 p/h, the approximate benchmark for an RTX 5090). The fleet summary was updated to show both current and projected capacity side by side: "1 running, 7 loading (~350 p/h when ready). Capacity: 40 p/h now, ~390 p/h projected."

This is a textbook example of making implicit knowledge explicit. The human operator looking at the dashboard could mentally add up the loading instances and estimate future capacity. But the LLM, which has no intuitive sense of "these instances will be ready in an hour," needed that calculation served to it as a concrete number. The Go change was small—a few lines of arithmetic—but it fundamentally changed what the agent could perceive.

The Python Fix: Fast-Path and Prompt Rewrite

The Python agent received two changes. First, the fast-path logic—a set of deterministic rules that can short-circuit the expensive LLM call—was updated to use projected_proofs_hr instead of capacity_proofs_hr. If projected capacity already meets or exceeds the target, the agent skips the LLM entirely and reports "No action needed." This is a critical efficiency gain: it prevents the agent from even consulting the LLM when the math is clear.

Second, the system prompt was rewritten to include explicit instructions about projected capacity. The prompt now states: "projected_proofs_hr = current + loading×50. If this meets target, do NOT launch more." This is a direct, unambiguous rule that the LLM can follow without needing to reason about instance startup times or do mental arithmetic.

The Rate-Limit Fix: Stopping Instead of Spinning

The third fix addressed a different failure mode: the agent was wasting all five of its LLM iterations retrying launches with different offers after hitting the vast.ai rate limit (429 Too Many Requests). Each retry consumed a full LLM call, burning tokens and time without any chance of success.

The fix was simple but effective: when a launch tool call returns a 429, the iteration loop breaks immediately with a log message saying "will retry next cycle." A new prompt rule reinforces this: "If rate-limited, STOP trying. Do not try different offers." This reduced the agent's run time from 57 seconds (five wasted iterations) to 43 seconds (one attempt plus immediate stop).

The Result: An Agent That Can See the Future

The deployed fix produced immediate results. When the assistant tested the updated agent ([msg 4505]), the behavior was dramatically different:

1. Sees projected capacity: "390 p/h projected / 500 target — may need more instances"
2. Does the math: "Gap: 500 - 390 = 110 p/h shortfall. Need ~2 more RTX 5090s"
3. Tries to launch just 1 (not 10) to close the gap
4. Rate limited → immediately stops: "Rate-limited, will retry on next scheduled run" The agent went from launching eight instances to launching one. It went from burning through all five iterations to stopping after one attempt. The cost savings are substantial: at roughly $0.50/hr per instance, avoiding five unnecessary instances saves $2.50/hr, or $60/day, or over $1,800/month.

Assumptions and Limitations

The fix relies on an important assumption: that each loading instance will contribute approximately 50 p/h once ready. This is a reasonable default for RTX 5090 GPUs running the standard proving workload, but it could be wrong for slower GPUs, instances with configuration issues, or workloads that are more computationally intensive. The assistant acknowledged this implicitly by using the term "estimated" in the fleet summary.

There is also an assumption that the rate limit is the only reason a launch might fail. The fix breaks the loop on 429, but other errors (insufficient funds, invalid offer, instance creation failure) could still cause the agent to waste iterations. A more robust approach might break on any error, not just rate limits.

The Deeper Lesson: Autonomous Agents Need Projection

The over-provisioning bug is a specific instance of a general problem in autonomous systems: agents that operate on delayed feedback loops need to model the future state of the world, not just the present. When instance startup takes 1-2 hours, decisions made now affect capacity hours from now. An agent that only looks at current metrics will oscillate wildly—launching too many when demand rises, then stopping too many when the delayed capacity finally comes online.

The fix in message 4506 is elegant because it doesn't try to teach the LLM about startup latencies or exponential smoothing. Instead, it pre-computes the projection server-side and serves it as a simple number. The LLM doesn't need to understand why projected capacity is 390 p/h—it just needs to know that it is. This is a separation of concerns: the backend handles the arithmetic, and the LLM handles the decision-making.

Conclusion

Message 4506 is a compact summary of a significant engineering achievement: teaching an autonomous agent to see the future. By adding a single computed field to the fleet API, rewriting a few lines of fast-path logic, and tightening the system prompt, the assistant transformed an over-provisioning, rate-limit-spamming agent into a measured, cost-conscious operator. The fix reduced instance launches from eight to one, cut run time by 25%, and saved thousands of dollars per month in GPU rental costs.

But more importantly, it demonstrates a principle that applies broadly to LLM-driven autonomous systems: don't ask the model to do arithmetic or reason about delayed feedback loops. Compute the projections server-side, serve them as concrete numbers, and let the LLM focus on what it does best—making contextual decisions based on clear data. The agent doesn't need to be smarter. It just needs better inputs.