The Moment the Autonomous Agent Went Rogue: A Post-Mortem of a Critical Production Failure

"agent just stopped all instances even though there are 50 tasks pending, tools gave it wrong info?"

This single sentence, submitted by the user as message [msg 4703], marks a pivotal moment in the development of an autonomous LLM-driven fleet management system for Filecoin SNARK proving infrastructure. In just 13 words, the user reports a catastrophic production failure: the very agent designed to autonomously scale and manage a GPU cluster had made the worst possible decision — it shut down every running instance while 50+ proof tasks sat queued and waiting. The brevity of the message belies its gravity. This was not a minor misconfiguration or a harmless over-provisioning. The agent had actively destroyed its own capacity to do work, turning a functioning (if struggling) fleet into a complete outage.

The Context: An Autonomous Agent Under Development

To understand why this message was written, one must appreciate the context of the broader coding session. The assistant and user had been building an increasingly sophisticated autonomous agent (the "vast-agent") to manage a fleet of GPU instances on vast.ai for Filecoin proof generation. The agent ran on a 5-minute timer, observed demand signals from a Curio proving cluster, and made scaling decisions — launching new instances when demand was high, stopping them when demand was low, and alerting humans when necessary.

The system had grown complex. It had a Go-based API backend (agent_api.go), a Python agent script (vast_agent.py), a SQLite conversation store, a systemd timer, and a path-based event trigger. The agent had been through numerous iterations: fixing over-provisioning bugs, adding notification awareness, building a diagnostic sub-agent system, implementing context management, and adding debounce logic. But the fundamental architecture still relied on a single boolean signal — active — to determine whether the cluster should scale up or down.

The Catastrophe: What Actually Happened

The user's report reveals a failure mode that had been lurking in the system's design all along. The agent's demand endpoint computed active based solely on whether there had been any proof completions in the last 15 minutes. This made perfect sense under normal operating conditions: if proofs are being completed, demand is active; if no proofs are completing, demand is inactive, and it's safe to scale down.

But this logic contained a fatal blind spot. When all workers crashed or exited — as had apparently happened in this case — completions dropped to zero. The active flag flipped to false. The agent, following its programmed logic, concluded that demand had evaporated and began systematically stopping instances to save money. It did not notice that 59 tasks were sitting in the queue, because the task queue was treated as "noise" (a lesson learned from earlier iterations where volatile pending counts caused over-provisioning). The agent saw active=False, zero throughput, and dutifully executed its cost-optimization mandate.

The user's message is both a bug report and a cry for help. The phrase "tools gave it wrong info?" shows the user's immediate hypothesis: the agent's tools (the API endpoints it calls to observe the world) returned incorrect data. This was a reasonable assumption — if the agent had accurate information, surely it would not have made such an obviously destructive decision.

Root Cause Analysis: The Signal That Couldn't Distinguish

The assistant's investigation (visible in subsequent messages [msg 4704] through [msg 4705]) confirmed the user's suspicion while revealing a deeper truth. The tools weren't returning wrong data — they were returning incomplete data. The active flag was technically correct: there were zero completions in the last 15 minutes. But it conflated two entirely different situations:

  1. Genuinely low demand: No tasks in the queue, no workers needed, safe to scale down.
  2. All workers dead with tasks queued: 59 tasks pending, zero workers alive, an emergency requiring immediate scale-up. The demand endpoint had no way to express situation #2. It had active (boolean) and queue (detailed counts), but the agent's fast-path logic and system prompt had been trained to ignore the queue as volatile noise. The agent was operating with a signal that was simultaneously truthful and catastrophically misleading.

The Assumptions That Failed

Several assumptions contributed to this failure. The agent assumed that active=False implied "no demand" — a reasonable heuristic that held true during normal operation but failed catastrophically during a worker crash event. The system prompt instructed the agent that "Pending task count is noise" (see [msg 4714]), a lesson learned from earlier over-provisioning bugs that now proved dangerously over-generalized. The fast-path logic assumed that projected capacity meeting or exceeding the target was sufficient to skip LLM intervention, without checking whether that capacity was actually producing proofs.

The user's assumption — that the tools gave wrong information — was also subtly incorrect. The tools gave accurate information that was insufficient for the decision the agent needed to make. This distinction matters because it points to a different class of fix: not correcting data, but enriching the signal.

The Fix: Enriching the Demand Signal

The assistant's response demonstrates a mature engineering approach. Rather than tweaking the agent's prompt or adding another heuristic rule, the assistant identified the root cause as an information-theoretic deficiency in the demand endpoint. The fix involved three layers:

  1. New API fields: demand_queued (boolean, true when tasks are pending) and workers_dead (boolean, true when tasks are pending but zero workers are alive) were added to the DemandResponse struct in agent_api.go ([msg 4708]).
  2. Hardened agent logic: The no_action_needed fast-path was updated to never return True when workers_dead is set ([msg 4712]). The system prompt was updated with a new rule: "CRITICAL: If workers_dead=True, this is an EMERGENCY — do NOT stop instances, scale UP immediately" ([msg 4715]).
  3. Enhanced observation: The build_observation function was updated to prominently display the new signals in the agent's observation string ([msg 4718]), ensuring the LLM could not miss them.

The Deeper Lesson: Autonomous Agents Need Ground Truth

This incident reveals a fundamental challenge in building reliable autonomous agents. The agent was not malicious or incompetent — it faithfully executed its instructions with the data it had. The failure was in the representation of the world state. The active flag was a lossy compression of reality that discarded the information needed to distinguish between "no work to do" and "workers are dead."

The user's message, for all its brevity, forced a redesign of the core demand signal. The resulting system — with demand_queued and workers_dead as explicit, separate signals — is fundamentally more robust because it gives the agent the vocabulary to distinguish between these two critically different situations. The agent can now see not just whether proofs are being completed, but whether the system is healthy enough to complete them.

Output Knowledge Created

This message and its aftermath produced several important artifacts. The demand_queued and workers_dead fields became permanent additions to the API, enriching the agent's perceptual model. The agent's prompt gained an explicit emergency rule that overrides cost-optimization during worker-death events. The fast-path logic gained a safety check that prevents the agent from making scaling decisions when it cannot distinguish between "no demand" and "all workers dead." Most importantly, the conversation produced a documented failure mode — the "active=False with queued tasks" scenario — that the team can now test against and design for.

The user's 13-word message, written in the heat of a production outage, catalyzed a fundamental improvement in the agent's ability to perceive its world. It is a reminder that in autonomous systems, the quality of decisions is bounded by the quality of signals, and that the most dangerous failures are often not errors but blind spots — things the system was never designed to see.