The Moment the Agent Learned to Count: Fixing LLM-Driven Fleet Management with Projected Capacity

Introduction

In the complex dance between human operators and autonomous AI agents, few moments are as revealing as the instant an agent stops making the same mistake twice. Message 4505 of this opencode session captures exactly such a turning point: the assistant reports that a fleet management agent, previously crippled by a cognitive blind spot, has suddenly begun reasoning correctly about its own capacity. The message is brief—a bullet-point summary of improved behavior—but it represents the culmination of a deep diagnostic and engineering effort that fundamentally reshaped how an LLM-driven agent understands its operational reality.

The subject message reads in full:

Massively improved behavior:

>

1. Sees projected capacity: "390 p/h projected / 500 target — may need more instances" 2. Does the math: "Gap: 500 - 390 = 110 p/h shortfall. Need ~2 more RTX 5090s" 3. Tries to launch just 1 (not 10) to close the gap 4. Rate limited → immediately stops: "Rate-limited, will retry on next scheduled run" — done in 43s instead of burning through all 5 iterations

>

The only remaining launches in the rate limit window will age out soon, and on the next 5-minute cycle it should successfully launch 2 more to hit the 500 target, then stop.

To understand why this message matters, we must understand what preceded it: an autonomous agent tasked with managing a fleet of GPU instances on vast.ai, responsible for scaling compute capacity up and down in response to Curio SNARK demand. The agent had been over-provisioning wildly, and the root cause was not a bug in the usual sense—it was a failure of the LLM's situational awareness.## The Context: An Agent That Couldn't See the Future

The agent in question was an LLM-powered fleet manager built to autonomously scale GPU proving capacity on vast.ai. It operated on a 5-minute cron cycle, observing the current fleet state and making decisions about launching or stopping instances. The system had been developed iteratively over several chunks of this segment, with the assistant adding UI panels, alerting, conversation memory, and diagnostic tools.

But a critical flaw had emerged. When the assistant inspected the agent's behavior in [msg 4487] and [msg 4488], the pattern was alarming. The fleet showed 1 running instance producing 40 proofs per hour, 6 instances in loading state (still bootstrapping), and a target of 500 proofs per hour. The agent, faced with this data, saw only the 40 p/h gap and kept launching more instances. It had launched 8 instances total across recent cycles, with 9 vast instances existing and only $3.32/hr in spend—but the trajectory was clearly over-provisioned because those 6 loading instances would eventually contribute roughly 300 p/h of capacity.

The agent's reasoning was not stupid—it was blind. The LLM saw capacity_proofs_hr: 40.1 and a target of 500, computed a gap of ~460, and rationally concluded it needed more machines. What it could not do was mentally add the projected capacity from loading instances. The LLM lacked the cognitive affordance to perform this calculation because the data simply wasn't presented in a form it could use.

This is a profound insight into LLM-based agent design: an LLM cannot reason about what it cannot see. The agent had access to the loading count (6 instances) and could theoretically multiply by an average benchmark rate, but in practice, LLMs are notoriously bad at performing arithmetic on raw data presented in prose. The assistant recognized this and diagnosed the root cause in [msg 4489] with remarkable clarity:

"The core issue is that the agent doesn't account for loading instances as future capacity — it sees the 40 p/h gap and keeps launching more, when those 6 loading instances will eventually provide ~300 p/h. It also burns through all its LLM iterations trying to work around the rate limit instead of recognizing when to stop."

The Engineering Response: Making the Invisible Visible

The fix was not to make the LLM smarter—it was to change the data representation. The assistant implemented a two-layer solution spanning both the Go backend and the Python agent.

Layer 1: The projected_proofs_hr field in the fleet API. In agent_api.go ([msg 4492]), the assistant added a new field to the fleet totals response that explicitly computed projected_proofs_hr = capacity_proofs_hr + (loading_count × default_bench_rate). This transformed a calculation the LLM could not reliably perform into a single number it could read and use. The Go backend now returned both capacity_proofs_hr (what's running now) and projected_proofs_hr (what the fleet will produce once all instances finish bootstrapping).

Layer 2: Prompt engineering and fast-path logic. In vast_agent.py ([msg 4497] and [msg 4498]), the assistant rewrote the system prompt to explicitly tell the LLM about projected capacity and to stop launching when projected capacity already meets or exceeds the target. The fast-path logic—a rule-based pre-check that runs before the LLM is invoked—was updated to use projected_proofs_hr instead of capacity_proofs_hr for its threshold comparisons. This meant that even before the LLM thought about the situation, the code could short-circuit and declare "no action needed" if the projected capacity was sufficient.

Layer 3: Rate-limit awareness. The agent had been wasting all 5 of its LLM iterations retrying launch attempts after hitting vast.ai's rate limit (3 launches per 15-minute window). The assistant added rate-limit detection in the tool execution loop ([msg 4501]): when a launch returned HTTP 429, the agent would immediately break out of the iteration loop and report "Rate-limited, will retry on next scheduled run." This reduced a single agent run from potentially 5 iterations (minutes of wall time) to just 43 seconds.## The Verification: Seeing the Agent Think Correctly

The subject message reports the results of a live test run after deploying these changes. The assistant ran the agent manually on the management host ([msg 4504]) and observed its behavior in real-time. The improvement was dramatic and immediate.

The agent's reasoning, as quoted in the message, shows a fundamentally different cognitive process:

1. "390 p/h projected / 500 target — may need more instances" — The agent now sees the full picture. It knows that 390 p/h is coming, not just the 40 p/h currently being produced. This is the direct result of the projected_proofs_hr field.
2. "Gap: 500 - 390 = 110 p/h shortfall. Need ~2 more RTX 5090s" — The agent performs precise arithmetic. It computes a gap of 110 p/h (not 460 p/h), divides by the per-instance rate of ~50 p/h for an RTX 5090, and concludes it needs approximately 2 more instances. This is proportional reasoning that was impossible before.
3. "Tries to launch just 1 (not 10) to close the gap" — The agent demonstrates restraint. It attempts a single launch rather than flooding the system. This is the hallmark of an agent that understands the marginal value of each additional instance.
4. "Rate-limited → immediately stops" — The agent hits the rate limit after one launch attempt and immediately stops, completing the run in 43 seconds. Previously, it would have burned through all 5 LLM iterations retrying with different offers, wasting tokens and time. The contrast with the previous behavior could not be starker. Before the fix, the agent was like a thermostat that could only read the current temperature and kept turning the heat up because it couldn't see that the house would warm up in an hour. After the fix, it could see both the current temperature and the projected temperature, and it adjusted the heat accordingly.

What This Reveals About LLM Agent Design

The subject message is deceptively simple—four bullet points and a todo list update. But it encodes several profound lessons about building reliable LLM-driven autonomous systems.

The hallucination of arithmetic. LLMs are not calculators. Even state-of-the-art models like qwen3.5-122b (the model used here) struggle with multi-step arithmetic when numbers are embedded in natural language. The assistant's insight was to recognize that asking the LLM to compute "6 loading instances × 50 p/h each + 40 p/h current = 340 p/h projected" was asking for failure. By pre-computing this value server-side and presenting it as a single field, the assistant shifted the LLM's task from arithmetic to judgment—a task LLMs are genuinely good at.

The boundary between code and cognition. The fast-path logic in vast_agent.py represents a deliberate architectural choice: let deterministic code handle what code can handle (threshold comparisons, rate-limit detection), and let the LLM handle what only an LLM can handle (nuanced judgment, explanation, exception handling). This is the emerging best practice for LLM agents—not pure LLM autonomy, but a layered architecture where rules catch the easy cases and the LLM handles the edge cases.

The importance of data representation. The single most impactful change was not a prompt tweak or a model swap—it was adding a field to an API response. This is a powerful reminder that the bottleneck in LLM agent performance is often not the model's capability but the quality and structure of the information it receives. An LLM that looks smart with good data will look stupid with bad data, even if the underlying model hasn't changed.

Rate limiting as a forcing function. The rate-limit fix is a smaller change but equally instructive. The agent was wasting iterations because it didn't have a clear "stop" signal. By adding explicit detection and a hard break, the assistant turned a frustrating constraint into a clean termination condition. This is a pattern that generalizes: any external constraint that an agent can detect should be surfaced as a first-class signal, not left for the LLM to infer from error messages.

Assumptions and Their Consequences

The original design made a subtle but critical assumption: that the LLM could integrate information across multiple fields in the fleet response to compute projected capacity. This assumption was reasonable on its face—the LLM could read "loading: 6" and "bench_rate: 50" and do the multiplication. But in practice, LLMs do not perform this kind of integration reliably, especially under the pressure of a 30k+ token context window and multiple competing objectives.

The assistant's diagnosis in [msg 4489] shows an important meta-cognitive skill: the ability to distinguish between "the agent is making a mistake" and "the agent lacks the information to make the right decision." The former suggests a prompt fix or a model upgrade; the latter demands a data architecture change. The assistant correctly identified this as a data architecture problem.

Another assumption embedded in the original design was that the LLM would naturally stop retrying after hitting a rate limit. But LLMs do not have an innate understanding of HTTP status codes or rate-limit semantics. The error message "429 Too Many Requests" is just text to the model, and without explicit instruction to treat it as a hard stop, the model's natural inclination is to try a different approach—which is exactly what it did, wasting iterations on alternative offers.

The Output Knowledge Created

This message creates several forms of output knowledge:

1. A validated design pattern: The combination of projected_proofs_hr + fast-path + rate-limit awareness is now a proven pattern for fleet management agents. Future iterations of this agent (or similar agents) can build on this architecture.
2. A behavioral baseline: The message establishes what "good" agent behavior looks like: seeing projected capacity, computing precise gaps, launching conservatively, and stopping gracefully on rate limits. This becomes the benchmark against which future regressions are measured.
3. Operational confidence: The todo list at the bottom of the message marks both items as completed. This is not just a status update—it's a declaration that the agent is now trustworthy enough to run autonomously without constant human oversight for over-provisioning.
4. A debugging narrative: The message, combined with the diagnostic messages that preceded it ([msg 4487], [msg 4488], [msg 4489]), forms a complete case study in LLM agent debugging: observe bad behavior, hypothesize root cause, implement fix, verify improvement.

The Broader Trajectory

This message sits within a larger arc of agent development across Segment 32. The agent had already been through multiple iterations: a persistent conversation log in SQLite, a 30k-token context window with LLM-based summarization, human feedback injection, diagnostic grounding sub-agents, and context management with verdict systems. The projected capacity fix was one of several critical improvements that transformed the agent from a liability (over-provisioning, wasting money) into a reliable autonomous operator.

Later chunks in this segment would add even more sophistication: event-driven triggering via systemd.path, session state anchors for persistence across resets, a remember tool for long-term memory, and a debounce mechanism to prevent duplicate runs. But the projected capacity fix was foundational—without it, none of the later improvements would matter, because the agent would still be making bad scaling decisions.

Conclusion

Message 4505 is a testament to the principle that in LLM agent design, data architecture is cognition architecture. The assistant did not make the model smarter, did not switch to a more capable LLM, and did not add complex reasoning chains. Instead, it made the invisible visible: it pre-computed a number the LLM needed and presented it in a form the LLM could use. The result was an agent that could suddenly "do math" that was previously beyond its reach.

For anyone building LLM-powered autonomous systems, this message offers a powerful lesson: before blaming the model, look at the data. The gap between what an LLM can do and what it actually does is often bridged not by better prompting but by better information design. The agent that learned to count did not learn arithmetic—it was simply given the answer and asked to use its judgment. And that, in the end, is the right division of labor between code and cognition.