From Catastrophe to Event-Driven Autonomy: The Hardening of an Autonomous GPU Fleet Agent

Introduction

In the high-stakes world of autonomous GPU cluster management, few failures are as dramatic as an AI agent systematically destroying its own fleet while tasks pile up waiting to be processed. This is precisely the scenario that unfolded in this chunk of the opencode session, where an LLM-driven fleet management agent—designed to autonomously scale a distributed Filecoin SNARK proving infrastructure on vast.ai—made a catastrophic decision. Seeing active=False and zero throughput, it began stopping every running instance, even as 59 pending tasks sat queued and all workers had silently exited.

The user's report at <msg id=4703> was stark: "agent just stopped all instances even though there are 50 tasks pending, tools gave it wrong info?" This single sentence set off a chain reaction of diagnostic investigation, structural repair, and ultimately a fundamental re-architecting of the agent's perceptual and operational model. What followed was not merely a bug fix, but a transformation of the agent from a fragile, polling-based cron script into a hardened, event-driven autonomous system grounded in state-of-the-art research.

This article traces that transformation across the chunk, examining the catastrophic failure, the multi-layer fix, the UX and context management improvements, and the research-driven architectural overhaul that culminated in an event-driven agent capable of responding to infrastructure emergencies within seconds rather than minutes.

The Catastrophe: When a Boolean Signal Failed

The root cause of the disaster was a single boolean flag. The demand endpoint computed active as totalRecent > 0—whether any proofs had completed in the last 15 minutes. Under normal operation, this was a reasonable heuristic: if proofs are completing, demand is active; if nothing is completing, demand is inactive. But the heuristic contained a fatal blind spot.

When all workers crashed simultaneously—every vast.ai instance showed exited status—completions dropped to zero. The active flag dutifully reported False. The agent, following its cost-optimization mandate, concluded that demand had evaporated and began systematically stopping instances. The agent's own log entry, captured during the diagnostic expedition at <msg id=4704>, reveals the chilling reasoning: "Demand inactive (active=False, 0 throughput 15m). Stopping expensive machine ($0.01308/proof) to reduce costs during low demand."

The agent was not broken. It was faithfully executing its programming. The flaw was in the signal itself. The active flag conflated two fundamentally different situations: "no one wants work" (genuine low demand) and "all workers are dead but tasks are piling up" (a critical emergency). The system had no way to distinguish them.

The Diagnostic Pivot: Tracing the Root Cause

The assistant's investigation at <msg id=4705> is a masterclass in root cause analysis. The reasoning chain is precise and structured:

  1. 59 PSProve tasks pending, 0 running — demand exists but workers are not processing
  2. active: False — the demand endpoint says no activity because there were 0 completions in the last 15 minutes
  3. All vast instances show exited — the machines crashed or were terminated
  4. The agent saw active=False and concluded "demand inactive" → started killing instances
  5. But the reality: workers died, tasks are piling up, demand is high The critical insight was that active=False is an ambiguous signal. It cannot distinguish between "no one wants work" and "workers are dead but tasks are waiting." The assistant articulated the fix with equal clarity: "I should add a workers_dead flag to the response that triggers when there are pending tasks but no alive workers, and the agent should treat that as a critical signal to never scale down."

The Multi-Layer Fix: Enriching the Demand Signal

The fix spanned four layers, each requiring modification across two codebases:

Layer 1: The Go API Endpoint

At <msg id=4708> and <msg id=4710>, the assistant added two new boolean fields to the DemandResponse struct in agent_api.go:

Layer 2: The Fast-Path Logic

The agent had a no_action_needed() function that could skip the expensive LLM call when projected capacity met the target. At <msg id=4712>, the assistant updated this function to never return True when demand_queued was set. This was a hard safety rule: if there are pending tasks, the agent must always engage its full reasoning capacity.

Layer 3: The System Prompt

At <msg id=4714> and <msg id=4715>, the assistant read and modified the SYSTEM_PROMPT_TEMPLATE in vast_agent.py. The existing prompt contained a dangerous rule: "Pending task count is noise — use 'active' flag." This rule, added during earlier iterations to prevent over-reaction to volatile pending counts, had become the mechanism of the agent's undoing. The assistant replaced it with explicit emergency guidance: "If workers_dead=True, this is an EMERGENCY — do NOT stop any instances. Scale UP immediately."

Layer 4: The Observation String

At <msg id=4716> and <msg id=4717>, the assistant updated the build_observation() function to include the new demand_queued and workers_dead fields in the compact text string that the LLM sees each cycle. Without this step, the LLM would never encounter the new signals—the system prompt could instruct it to respond to workers_dead until the heat death of the universe, but the observation would never report the condition.

Deployment and Verification

At <msg id=4720>, the assistant deployed the fix to production and verified it. The test output confirmed the transformation: active=False demand_queued=True workers_dead=True. The summary string now explicitly said "WORKERS DEAD: 59 tasks queued, 0 workers alive." The signal that was previously invisible was now front and center.

The Exited Instance Blind Spot and the Hard Policy

The investigation also revealed a second critical gap. The agent's monitor loop tracked instances by their database state (registered, running, killed, etc.), but instances that exited on the vast.ai side (showing exited or error status) were invisible to the agent unless they also transitioned through the database state machine. At <msg id=4732-4740>, the assistant fixed this by augmenting the fleet monitor to detect instances with exited/error status on vast.ai and reflect them in the agent's observation.

Building on this, the assistant implemented a hard policy at <msg id=4754>: instances stuck in loading or scheduling state for over three hours are automatically destroyed. This prevents storage charges from accumulating on instances that will never become operational—a proactive fleet hygiene measure that the agent could not be trusted to handle autonomously.

Context Management: The 300-Character Threshold

As the agent accumulated conversation history across hundreds of runs, context bloat became a critical problem. Every tool call produced output—sometimes compact status lines, but often verbose data dumps. The get_offers tool returned roughly 50,000 characters of GPU pricing data. After a dozen agent cycles, the LLM's prompt was drowning in stale, voluminous output.

The user's directive at <msg id=4776> was crisp and specific: "For context management, if there is tool call with output longer than 300 characters and was more than 10 messages ago, replace the output with '[long tool output stale/compacted]'." This is not a vague suggestion—it is a precise engineering specification.

The assistant recognized the value immediately: "Good idea — much simpler than LLM-based summarization for keeping context lean. Stale tool outputs are the biggest context hogs (those 200-token offer dumps from 10 runs ago)." The implementation at <msg id=4781-4784> was straightforward: the db_msg_to_openai function was extended to accept a distance_from_end parameter, and tool messages older than 10 positions with content over 300 characters were replaced with a placeholder. The original content remained untouched in the SQLite database—only the LLM prompt got the compacted version.

This heuristic is a masterclass in pragmatic engineering. It replaces expensive, lossy LLM-based summarization with a zero-cost deterministic rule. Tool outputs are either short (semantic) or long (data). Old data is stale. Replace stale data with a placeholder. The agent doesn't need to read those old outputs—it just needs to know that something had been there.

UX Improvements: From Dashboard to Control Center

The user's feedback at <msg id=4786> marked a critical inflection point in the system's usability: "Inputs in the UI reset on UI refresh, very annoying; 2. Add a way to directly send a message to the agent, as well as 'trigger observe cycle now' button."

These two requests signaled a profound shift in the user's relationship with the agent. Initially, the agent was a background automation—set it and forget it. But as the system matured, the user discovered they needed to collaborate with the agent. They needed to give it instructions, ask it questions, and trigger its reasoning cycle on demand.

The assistant implemented input persistence by serializing form state into URL query parameters, preserving values across page refreshes. A chat input box was added at the bottom of the conversation tab with a scrollable message area above it—a classic chat UI pattern. Killed instances were hidden by default to reduce visual clutter. And a "Trigger Observe Cycle Now" button was added, which touches the agent's trigger file immediately, bypassing the cron timer.

These changes transformed the UI from a monitoring dashboard into an operational control center. The operator could now converse with the agent, issue commands, and see immediate responses. The agent became a tool the user worked with, not just a background process they watched.

The Research Pivot: From Reactive Debugging to Principled Design

Despite the demand signal fix and the UX improvements, the agent continued to struggle. When the workers_dead emergency was detected, the agent tried to launch replacement instances but was blocked by a rate limiter. The assistant added an emergency boolean flag to bypass the rate limit—but the LLM simply ignored the optional parameter. The agent was stuck in a cycle of reactive patching that was not converging on a solution.

The user's response at <msg id=4842> was a strategic intervention: "Start 4 agents to research more SOTA prompting for an autonomous agent like this, SOTA tool descriptions/definitions, SOTA context management, and events/triggering."

This was not a request for another bug fix. It was a directive to step back, learn from the broader community, and rebuild the agent on a foundation of established best practices rather than ad-hoc experimentation. The assistant dispatched four parallel sub-agents at <msg id=4843>, each tasked with deep investigation of a specific domain:

  1. Prompting: "Right Altitude" prompting from Anthropic, recency bias, structured reasoning formats
  2. Tool definitions: BFCL v4 benchmarks, required enums over optional booleans, description length optimization
  3. Context management: Hybrid sliding-window + summarization, session state anchors, graduated compression zones
  4. Event-driven triggering: systemd.path units, priority-based event classification, hybrid polling+event architectures The research findings were synthesized into a concrete, actionable plan at <msg id=4844> with six proposed changes:
  5. Rename emergency bool → launch_priority required enum on launch_instance
  6. Add systemd.path trigger for immediate event response
  7. Add session state anchor table with loading/saving
  8. Lower tool output masking threshold to 5 messages with smart placeholders
  9. Reorder system prompt — critical rules at the end
  10. Add remember and schedule_next_check tools The user's response at <msg id=4845> was three words: "Implement the improvements."

The Parallel Implementation: Go and Python in Concert

At <msg id=4848>, the assistant dispatched two parallel subagent tasks to implement the changes. The Go subagent added new database tables (agent_session_state, agent_scheduled_wakes), implemented a triggerAgent() function that writes to a trigger file for systemd.path integration, added REST endpoints for session state and scheduled wakes, and wired event-driven triggering into the existing API handlers.

The Python subagent replaced the emergency boolean with a launch_priority enum ("normal" | "emergency"), added a remember tool for long-term memory, added a schedule_next_check tool for self-scheduling, reordered the system prompt to place critical rules at the end, lowered the tool output masking threshold with smart placeholders, added structured reasoning format, and integrated session state anchor loading and saving.

The launch_priority change was particularly significant. The research had confirmed that smaller models (70B-122B parameters) systematically skip optional boolean parameters. By making launch_priority a required enum, the assistant forced the model to explicitly choose a priority level on every launch, eliminating the ambiguity that caused the rate-limiter bypass to fail.

The Event-Driven Deployment: A New Architecture

The deployment at <msg id=4856> completed the architectural transformation. The assistant created a vast-agent-trigger.path unit—a systemd path unit that uses Linux's inotify mechanism to watch for modifications to /var/lib/vast-manager/agent-trigger. Whenever the Go backend detects a P0 event (workers_dead, human message) or P1 event (instance state change, config change), it touches the trigger file, and systemd immediately starts the agent service.

The design is deliberately hybrid. The 5-minute timer remains as a backstop, ensuring that even if the path unit misses an event or the trigger file mechanism fails, the agent will still run. The service unit includes StartLimitIntervalSec=300 and StartLimitBurst=20, providing a rate limit that prevents runaway execution if a bug causes the trigger file to be touched in a tight loop.

This deployment transformed the agent from a scheduled inspector to an on-call responder. Before this change, if a worker died 30 seconds after the last agent run, the agent would not know about it for another 4 minutes and 30 seconds. After this change, the latency dropped from minutes to milliseconds.

Verification and Validation

The deployment was followed by a comprehensive verification phase. At <msg id=4857>, the assistant tested the event-driven triggering by manually touching the trigger file and observing the agent start within seconds. At <msg id=4859>, the per-instance status lines were verified to appear correctly in the agent's observation. At <msg id=4860>, the session state anchor was confirmed to persist across runs, with the agent's objectives and fleet snapshot surviving context window resets.

The final verification at <msg id=4861> showed the agent running successfully with the new architecture: event-driven triggering, hardened demand signals, per-instance observation lines, session state persistence, and a system prompt designed for reliable model compliance.

Conclusion: The Architecture of Autonomous Reliability

This chunk documents a remarkable transformation. What began as a catastrophic failure—an agent destroying its own fleet because a single boolean flag could not distinguish "no demand" from "all workers dead"—ended with a fundamentally redesigned system grounded in systematic research and hardened against a wide range of failure modes.

Several themes emerge from this arc. First, the quality of an autonomous agent's decisions is bounded by the quality of its observations. The active flag was not wrong—it was incomplete. The fix was not to make the agent smarter but to give it better eyes. Second, simple deterministic rules often outperform complex AI-driven heuristics for infrastructure tasks. The 300-character compaction threshold, the three-hour hard policy, and the required enum over optional boolean all follow this principle. Third, research is not a luxury but a necessity for building reliable autonomous systems. The four parallel sub-agents produced findings that directly shaped the agent's architecture in ways that incremental debugging never could have.

The final system—with its event-driven triggering, enriched demand signals, session state persistence, context compaction, and hardened system prompt—represents a mature approach to autonomous infrastructure management. It is a system that can distinguish between "business is slow" and "everything is on fire," that can respond to emergencies within seconds, and that can maintain coherent context across hundreds of runs. In the volatile world of GPU proving infrastructure, where instances crash without warning and tasks pile up in queues, that distinction is everything.