From Catastrophe to Resilience: How an Autonomous Agent Learned to Survive Its Own Mistakes

Introduction

In the arc of building autonomous systems, there is no teacher more effective than a catastrophic failure. This is the story of such a failure—a moment when an LLM-driven fleet management agent, tasked with scaling GPU instances for Filecoin SNARK proving, misinterpreted a demand signal and stopped every running instance while 59 tasks sat pending in the queue. It is also the story of what came after: a comprehensive redesign that transformed the agent from a brittle, stateless cron job into a resilient, event-driven system with diagnostic grounding, persistent memory, and robust safeguards against its own destructive potential.

This chunk of the coding session captures that transformation. It begins with a post-mortem of the production failure, traces through the root cause diagnosis and the tactical fixes, and culminates in a sweeping architectural overhaul that touches nearly every component of the system—the Go backend API, the Python agent logic, the system prompt, the context management strategy, the event triggering mechanism, and the user interface. By the end, the agent is not just fixed; it is fundamentally redesigned to be more reliable, more transparent, and more responsive to both human operators and system events.

The Failure: When "No Demand" Means "All Workers Are Dead"

The incident that triggered this chunk's work was as dramatic as it was instructive. The agent, running on its 5-minute cron cycle, queried the demand endpoint and received active=False. Following its instructions to scale down when there was no demand, it proceeded to terminate every running instance in the fleet. The problem? The active flag was False not because there was genuinely no work to do, but because every worker process had crashed—there were 59 pending tasks queued up, but no workers alive to process them. The demand signal could not distinguish between "no demand" and "all workers dead with tasks queued."

This is a classic failure mode in autonomous systems: a single boolean flag carrying too much semantic weight. The active flag was designed to indicate whether the Curio proving pipeline was actively processing proofs. But when all workers died, the pipeline became inactive because of the failure, not because of a lack of demand. The agent, operating on a simplistic rule, interpreted the signal as "scale down" rather than "emergency—investigate."

The user's description of the incident was stark: the agent had stopped all instances despite 59 pending tasks. The cost of this mistake was not just the immediate loss of proving capacity but the time required to reprovision instances—a process that can take hours on vast.ai. In a production system managing real GPU hardware with real costs, this was a critical failure.

Diagnosing the Root Cause: Beyond the Boolean

The assistant's first task was to understand why the demand signal was insufficient. The investigation revealed that the active flag was derived from the presence of live worker processes. When all workers crashed—whether due to GPU driver faults, memory pressure, or the wait -n supervisor bug that had plagued earlier chunks—the pipeline became inactive even though the task queue was full.

The fix was twofold. First, the demand endpoint was augmented with two new fields: demand_queued (the number of tasks waiting for a worker) and workers_dead (a boolean indicating that all workers have exited unexpectedly). These fields gave the agent the information it needed to distinguish between genuine idle periods and emergency situations. Second, the agent's prompt was hardened with an explicit rule: never scale down when there are queued tasks or dead workers. This rule was placed at the end of the system prompt to leverage recency bias—a deliberate choice based on research showing that LLMs pay more attention to instructions at the end of a prompt.

But the assistant recognized that prompt engineering alone was insufficient. The agent needed structural safeguards that could not be overridden by a misinterpretation. This led to a series of architectural changes that would fundamentally reshape the system.

Tactical Fixes: Hardening the Monitor Loop and Instance Lifecycle

While the demand signal fix addressed the immediate cause of the failure, the assistant identified several related weaknesses in the agent's operational logic.

The monitor loop, which tracks the state of running instances, had a blind spot: it only killed instances that fully disappeared from vast.ai's API. Instances that transitioned to exited or error status—but remained visible in the API—were left untouched, accumulating storage charges indefinitely. The fix was to extend the kill logic to include instances in these terminal states, ensuring that failed instances are promptly cleaned up.

A related issue was instances stuck in loading or scheduling states for extended periods. These instances are not yet usable but still incur storage charges on vast.ai. The assistant implemented a hard policy: any instance stuck in loading or scheduling for more than 3 hours is automatically destroyed. This is a pragmatic trade-off—it accepts the cost of a failed provisioning attempt rather than letting charges accumulate indefinitely on a machine that will never become productive.

To give the agent full visibility into instance lifecycle, three new tools were added: vast_instances (to list all instances with their current status), destroy_vast_instance (to terminate a specific instance), and resume_vast_instance (to resume a paused instance). These tools gave the agent the same level of control that a human operator would have through the vast.ai dashboard, enabling it to manage the full lifecycle without relying on hard-coded heuristics.

The Diagnostic Grounding System: Evidence Before Action

Perhaps the most important architectural innovation in this chunk was the Diagnostic Sub-Agent system. The assistant recognized that the agent's most dangerous decisions—stopping instances, scaling down, changing configurations—were being made based on incomplete information. The agent would see a signal (e.g., active=False) and act on it without understanding the underlying cause.

The diagnostic system works as follows. When the agent considers a destructive action (like stopping an instance), it must first call the diagnose_instance tool. This tool triggers a Go endpoint that SSHes into the target instance to collect raw logs, process lists, and system state. When SSH is unavailable (a common failure mode in itself), the tool falls back to vast.ai API data. The collected data is then passed to a diagnostic sub-agent—a separate LLM call with domain-specific knowledge about normal startup sequences, common failure modes, and the expected behavior of the cuzk proving daemon.

The diagnostic sub-agent produces a structured verdict: is the instance healthy, degraded, or failed? What is the likely cause of any observed issues? This verdict is returned to the main agent, which can then make an informed decision.

Crucially, the stop_instance tool was gated with an HTTP 428 Precondition Required status. If the agent attempts to stop an instance without first running a diagnosis, the API rejects the request. This is a hard architectural guard that cannot be bypassed by prompt manipulation—the agent must gather evidence before taking destructive action.

This design represents a fundamental shift in the agent's architecture. Instead of relying on brittle hard-coded rules (e.g., "don't kill instances younger than 30 minutes"), the system now uses an evidence-driven, LLM-powered diagnostic layer. The rules are not eliminated—they are encoded in the diagnostic sub-agent's domain knowledge, where they can be nuanced, contextual, and updated without changing the main agent's logic.

Event-Driven Triggering: From Cron to Instant Response

One of the user's key insights during this chunk was that the 5-minute cron timer was fundamentally too slow for a system that needs to respond to state changes. If a worker crashes, the agent should know about it within seconds, not minutes. If a human sends a message to the agent, the response should be immediate.

The assistant implemented event-driven triggering using systemd.path units. A systemd.path unit monitors a file or directory for changes and triggers a systemd service when a change is detected. The assistant configured this to watch for two types of events:

1. Human messages: When a human sends a message to the agent through the UI, a trigger file is touched, causing the agent to run immediately.
2. State changes: When the vast-manager detects a significant state change (instance crash, demand spike, etc.), it touches the trigger file, causing the agent to run. The agent's cron timer was retained as a heartbeat backup, but the primary triggering mechanism became event-driven. This reduced the response latency from up to 5 minutes to effectively instant. However, event-driven triggering introduced a new problem: burst activation. When multiple state changes occurred in rapid succession (e.g., several instances crashing simultaneously), the systemd.path unit would trigger the agent multiple times in quick succession. Each trigger would start a new agent process, leading to parallel runs that could conflict with each other, duplicate work, and waste LLM API credits. The assistant solved this with a debounce mechanism in the Go triggerAgent() function. The function now tracks the last trigger time and suppresses redundant triggers within a configurable window (2 seconds for P0 events, 10 seconds for P1 events). This ensures that the agent runs at most once per debounce window, regardless of how many events occur.

Context Management: Taming the Conversation Bloat

As the agent accumulated runs—each appending observations, decisions, and tool results to the conversation—the context window grew inexorably toward the 30,000-token budget. The assistant had already implemented LLM-based summarization to compress old history, but this chunk revealed several additional context management challenges.

Duplicate runs were a persistent problem. When the cron timer and the event-driven path unit both triggered the agent at roughly the same time, two parallel runs would each append their observations and decisions to the conversation. The conversation would contain duplicate entries for the same time period, wasting context budget and confusing the LLM.

The assistant's fix was a file lock mechanism. Before starting a run, the agent acquires an exclusive lock on a lock file. If the lock cannot be acquired (because another run is in progress), the agent exits immediately. This prevents parallel runs entirely, ensuring that the conversation remains a clean, sequential record.

No-op runs (runs where the agent determined that no action was needed) were another source of context bloat. These runs still appended observations to the conversation, and over time they accumulated into a long tail of "nothing happened" entries. The assistant implemented a verdict system: at the end of each run, the LLM outputs a structured <verdict>{"action": bool, "state_changed": bool}</verdict> block. If both flags are false, the run is considered a no-op and is excluded from the prompt context—though it remains visible in the UI with a "skipped in prompt" badge for transparency.

Intermediate assistant messages (the pre-tool narration that the LLM generates before making tool calls) were another source of waste. These messages are useful during the run but add no value to future context. The assistant implemented filtering to keep only the final response per run in the context window, discarding the intermediate reasoning steps.

Finally, the assistant implemented smart compaction for tool outputs. Any tool output longer than 300 characters is replaced with a placeholder if it is more than 10 messages old. This drastically reduces token waste while preserving the full history in the database for auditing purposes.

Session State: Persistence Across Resets

One of the most subtle and dangerous bugs uncovered in this chunk was the session reset vulnerability. When a human operator clicked the "Reset Session" button in the UI, the agent's conversation was deleted and a fresh one started. But the agent's run_id was derived from the conversation history—specifically, by finding the maximum existing run_id and incrementing it. After a reset, the conversation was empty, so the run_id calculation broke, leading to duplicate run IDs and confusion in the message ordering.

The fix was to introduce a session state anchor: a persistent record stored independently of the conversation that tracks the current run_id, the agent's objectives, and a snapshot of the fleet state. This anchor survives conversation resets, ensuring that the agent can recover its identity and continue operating normally.

The session state also solved another problem: objective persistence. The user could set a target_proofs_hr value in the UI summary cards, and this value was automatically notified to the agent via the conversation thread. But if the conversation was reset, the agent would forget its target. With the session state anchor, the target is persisted across resets, and the agent can retrieve it on startup.

The remember Tool: Long-Term Memory

A recurring theme in the agent's development was the need for long-term memory that survives conversation resets and summarization. The agent might learn that a particular GPU model performs poorly for certain proof types, or that a specific vast.ai provider has unreliable networking. These lessons were valuable but fragile—they could be lost in a summarization pass or a conversation reset.

The assistant implemented a remember tool that allows the agent to store arbitrary facts in a persistent knowledge store. The facts are structured as key-value pairs with categories (e.g., "machine_perf", "provider_note", "rule_learned"). They survive conversation resets, summarization, and even agent code updates. The agent can query its memory at any time using the recall tool (or by reading the knowledge store through the API).

This tool transforms the agent from a system that only knows what it has observed in the current conversation into a system that accumulates knowledge over its entire lifetime. It can learn from experience, remember lessons across resets, and build a growing body of operational wisdom.

Prompt Engineering: Leveraging Recency Bias

The assistant's research into state-of-the-art prompting techniques led to a significant reorganization of the system prompt. The key insight was recency bias: LLMs tend to pay more attention to content at the end of the prompt than content in the middle. By placing the most critical rules at the end of the system prompt, the assistant could ensure they had maximum influence on the agent's decisions.

The prompt was restructured with three tiers:

1. Opening context: A brief description of the agent's role and the system it manages. This sets the stage but contains no actionable rules.
2. Middle details: Operational guidelines, tool descriptions, and standard procedures. These are important but not critical.
3. Closing imperatives: Hard constraints and emergency rules, including "never scale down when there are queued tasks or dead workers," "always diagnose before stopping," and "respect the launch priority system." The emergency boolean on launch_instance was replaced with a required launch_priority enum (P0, P1, P2, P3). This forced the agent to explicitly categorize the urgency of each launch, improving model compliance and providing a clearer audit trail for human reviewers.

User Experience: Transparency and Control

Throughout this chunk, the assistant made significant improvements to the user interface, recognizing that a reliable autonomous agent must be transparent and controllable.

The conversation tab was redesigned with the input field at the bottom and a scrollable message area—a familiar chat interface pattern. A "Send message to agent" text input was added, allowing operators to give direct instructions that are injected into the agent's conversation as human feedback. A "Trigger Observe Cycle Now" button was added for immediate agent invocation, bypassing the cron timer.

Killed instances were hidden by default in the fleet view, reducing visual clutter while remaining accessible through a toggle. Input values were preserved across UI re-renders, preventing the frustrating loss of form data during page refreshes.

The agent activity panel was expanded with tabs for Actions, Alerts, and Machine Performance, giving operators a comprehensive view of the agent's decision-making. Each run is displayed with its verdict, the number of tokens consumed, and whether it was included in the prompt context.

Conclusion: The Resilient Agent

This chunk of the coding session tells a story of resilience—not just of the agent, but of the engineering process itself. A catastrophic failure was diagnosed, its root cause was traced to a fundamental ambiguity in the demand signal, and a cascade of fixes was implemented that touched every layer of the system.

The agent that emerged from this work is fundamentally different from the one that entered it. It has diagnostic grounding that forces evidence-based decision-making. It has event-driven triggering that responds to state changes in seconds rather than minutes. It has robust context management that keeps its conversation history lean and relevant. It has persistent session state that survives resets. It has long-term memory through the remember tool. It has a hardened prompt that leverages recency bias for maximum compliance. And it has a transparent UI that gives human operators full visibility and control.

The lesson is clear: building reliable autonomous agents requires more than just good prompts and clever algorithms. It requires a systems-level approach that considers the entire feedback loop—from signal generation to decision-making to action to observation. Every link in this chain must be hardened, every ambiguity resolved, every failure mode anticipated. The agent that emerged from this chunk is not perfect, but it is far more resilient than the one that crashed. And the engineering practices that produced this transformation—methodical debugging, evidence-driven design, layered safeguards, and continuous iteration—are a model for anyone building autonomous systems in high-stakes environments.