Grounding, Verdict, and Recovery: Engineering Reliability into an Autonomous LLM Fleet Agent
Introduction
Building a reliable autonomous agent is fundamentally different from building a reliable traditional system. In traditional software, failures follow predictable patterns: a null pointer dereference, a race condition, a timeout. These can be caught, logged, and fixed with deterministic patches. But when the core decision-maker is a large language model—a stochastic system that can hallucinate, speculate, and misinterpret—the failure modes become qualitatively different. The agent does not crash; it makes a bad decision with perfect confidence. It does not deadlock; it slowly accumulates context until it forgets what it was doing. It does not corrupt a database; it destroys a production fleet of GPU instances because it mistook normal startup behavior for failure.
This article examines a concentrated period of development—spanning roughly 150 messages in an opencode coding session—where an autonomous LLM-driven fleet management agent underwent a fundamental transformation. The agent, built to manage a cluster of GPU instances on vast.ai running Filecoin SNARK proving workloads (cuzk/curio), had already survived several production incidents. It had once misinterpreted active=False as "no demand" and destroyed all running instances despite 59 pending tasks. It had suffered from context overflow, duplicate parallel runs, and a session reset that left it completely paralyzed.
The work in this chunk addressed three interconnected challenges: building a diagnostic grounding system to replace speculation with evidence, implementing a verdict system and context management overhaul to keep the agent's memory clean and structured, and debugging a context overflow crisis that exposed fundamental fragility in event handling and state management. Each of these challenges reveals something profound about the nature of autonomous agent reliability—and the engineering discipline required to achieve it.
Part I: Grounding the Agent—From Speculation to Evidence
The Catastrophe That Changed Everything
The diagnostic grounding system was born from a specific, painful failure. The agent observed twelve instances in various stages of startup—some as young as 0.2 hours old—and saw cuzk_alive=false with 0% GPU utilization across the board. Without any deeper investigation, it concluded the instances were "stuck" and proceeded to destroy all of them, including three that had already progressed to the params_done state, a clear signal that startup was proceeding normally [29].
The agent's reasoning, visible in the conversation logs, showed a pattern of speculation: "All 3 'params_done' instances show cuzk_alive=false and 0% GPU utilization, indicating the worker process is not running. They've been stuck for 0.8-1.1 hours, which is abnormally long for benchmarking/SRS loading." This was not a malicious decision—it was an ignorant one. The agent lacked the diagnostic tools to distinguish between a genuinely stuck instance and one proceeding through a normal but lengthy startup sequence [23][24].
The False Start: Hard Guards
The assistant's first instinct was defensive and entirely natural: add a hard-coded startup guard to the stop_instance and destroy_vast_instance API endpoints that would reject any operation on instances less than two hours old [25][26][27]. This is the classic engineering reflex—when an autonomous system makes a bad decision, constrain its decision space with rules.
But the user immediately intervened with a correction that would reshape the entire architecture [32]:
"Noo it is fine that the agent may destroy shorter running instances, it just needs to ground itself in truth first and never speculate whether instance state is bad or not. Maybe make it possible to call Stop only after the agent has queried the 'state discovery agent' and grounded itself that the instance is indeed in a bad state."
This was the pivotal insight. The problem was not that the agent was too aggressive—it was that the agent was making decisions without evidence. A hard age-based guard would prevent the agent from destroying genuinely broken young instances (e.g., an instance that crashes immediately due to a GPU driver issue). It would also prevent the agent from learning and adapting. The correct solution was not to restrict the agent's options but to give it the tools to make informed decisions [33][34][35].
The Diagnostic Sub-Agent Architecture
The assistant pivoted immediately, reverting the startup guard and committing fully to an evidence-driven approach [36][37][38][39]. The resulting architecture was a two-layer diagnostic pipeline:
Layer 1: The Go Endpoint (GET /api/agent/diagnose/{vast_id}). This endpoint SSHes into the target instance and collects raw telemetry: entrypoint logs, daemon logs, process listings (ps aux), memory statistics, GPU state, and any error messages from systemd journal or application logs. Critically, it includes a fallback for instances where SSH is broken (a common problem on vast.ai), returning vast API data and manager database state instead [40][41][42][43][44]. The endpoint logs every diagnose action to an agent_actions table in SQLite, creating an audit trail [47].
Layer 2: The Sub-Agent LLM. A smaller, focused LLM call receives the raw diagnostic data along with context from the codebase about what normal behavior looks like. The sub-agent returns a structured verdict: "healthy/progressing," "warning (e.g., OOM but recovering)," or "error (crashed, needs intervention)." This is what the main agent sees—not raw log lines, but a distilled, evidence-based assessment [29][30][31].
The stop_instance tool was then gated with an HTTP 428 "Precondition Required" response: before allowing a stop, the Go handler checks whether a recent diagnose action exists in the agent_actions table for that specific vast_id. If not, it returns 428 with a clear error message telling the agent to call diagnose_instance first [66][67][68].
This design pattern is profound. The system does not second-guess the agent's conclusions; it only ensures those conclusions are based on data. If the diagnostic sub-agent reports "instance is healthy, progressing normally," the main agent can still choose to destroy it (perhaps for cost reasons), but it will do so with full awareness of the consequences. If the diagnostic sub-agent reports "instance has crashed, OOM detected, no recovery," the main agent can act decisively with confidence [34].
The 428 That Proved the Architecture
The first successful test of this system was almost anticlimactic. The assistant curled the stop endpoint for an undiagnosed instance and received HTTP 428 [66]. This single response code validated the entire architectural approach. The precondition check correctly distinguished between instances that had been diagnosed (which returned 200) and those that had not (which returned 428). In that refusal, the system proved it was working exactly as intended [67][68].
The diagnostic grounding system represents a fundamental shift from constraint-based safety (hard rules about what the agent cannot do) to evidence-based safety (requirements about what the agent must know before acting). It is a pattern that applies far beyond instance lifecycle management—it is a general architecture for trustworthy autonomy.
Part II: The Verdict System—Structured Decisions and Clean Context
The User's Precision Bug Report
Shortly after the diagnostic grounding system was deployed, the user delivered another critical message [69]:
"There seem to be duplicate calls to cron observe, so seems a lot of the time we call the agent in two parallel streams(?) and responses also duplicate? Should have some semaphore. Also on the wait/idle no-action checking we don't remove the last message correctly, presumably because there is no sentinel that we can match on easily. The agent in cron calls should be prompted to additionally return a json return like {\"action\": bool, \"meaningful_state_change\": bool}, then we parse for this json within the larger json response."
Three problems, three proposed solutions, delivered in the terse, high-signal style of an engineer who has been watching logs scroll by [69]. The user identified:
- Duplicate parallel agent runs: The systemd timer (every 5 minutes) and the systemd path unit (triggered on state-change events) could fire simultaneously, causing two agent instances to run in parallel.
- No-op observations polluting context: When the agent ran but determined no action was needed, its observation and response still got appended to the conversation history, consuming precious context window space.
- No structured verdict: Without a machine-readable signal, the system could not reliably determine whether a run had produced meaningful action.
The File Lock
The assistant implemented a file-level lock using fcntl.flock at the start of the Python script [70][71][72]. If another instance was already running, the script exited immediately. This prevented both the timer and path unit from triggering simultaneously because whichever process acquired the lock first won, and the other exited harmlessly.
File locks were chosen over alternatives like PID files or database-level semaphores because they are automatically released when the process dies (no stale lock cleanup needed), they work across any processes on the same filesystem, and they are simple and reliable with no external dependencies [70]. However, the implementation hit real-world complications: the lock file's directory had restrictive permissions (owned by root), and the agent sometimes ran as a non-root user, causing a PermissionError that required fixing directory permissions [92][93][94][95][96][97].
The Structured Verdict JSON
The verdict system was the architectural keystone of the context management overhaul. The system prompt was modified to instruct the LLM to end every response with a structured <verdict>{"action": bool, "state_changed": bool}</verdict> block [70][82][83]. After the LLM responded, the Python wrapper parsed the response for this JSON block. If both action and state_changed were false (a no-op run), the wrapper removed all messages from that run from the conversation history [118][119][120].
This solved two problems at once: the structured JSON provided a reliable sentinel for detecting no-op runs, and the pruning kept the conversation history clean. Runs that took no action were excluded from the LLM prompt while remaining visible in the UI with a "skipped in prompt" badge, preserving the audit trail without wasting tokens [122][123][124][125].
Duplicate Suppression
A related problem was that the agent produced two messages per run: an intermediate "pre-tool narration" message and a final answer. Both were being included in the prompt context, effectively doubling the token cost of each run. The assistant implemented suppression logic that filtered out intermediate assistant messages, keeping only the final response per run in the context window [106][107][108][109][110][111][112].
The schedule_next_check tool was also made a no-op for its default 240–360s range since the 5-minute heartbeat timer already covered that window, eliminating a source of redundant tool calls that had been contributing to context bloat [128][129][130][131][132].
Part III: The Context Overflow Crisis—When the Reset Button Breaks the Agent
The Distress Signal
The most dramatic incident in this chunk arrived as a terse user message [137]:
"So: Agent crossed 31k context (actually on backend was into 38k already), and the compaction (is that even implemented correctly?) didn't run. Then I've clicked the 'Reset' reset-settion button, which did clear the session, However now the model does not run on cron nor does it respond to the 'Trigger run now' button."
In three sentences, the user reported a cascading failure: context overflow, a broken compaction mechanism, a destructive reset, and complete agent paralysis. The skepticism in the parenthetical—"(is that even implemented correctly?)"—was earned. The compaction mechanism existed in theory but had never been tested under the load of a real production conversation [137][138][139][140][141].
The Diagnosis: Stale Wakes and Re-Trigger Churn
The assistant's investigation revealed that the root cause was not a single bug but a convergence of three interacting failures [142][143][144].
First, stale scheduled wakes. The agent could schedule future check-ins via the schedule_next_check tool, which wrote wake events to a database table. But the wake-processing loop never marked wakes as "processed." Twenty stale wakes had accumulated, and every agent run would log "Processing 20 scheduled wakes..." and re-process them all, generating redundant tool calls and observation entries that inflated the conversation history [142].
Second, bursty event-driven triggers. The systemd.path unit watched a trigger file for modifications. When multiple state-change events occurred in rapid succession (e.g., several instances transitioning from "loading" to "running" within seconds), each event touched the trigger file, causing the path unit to spawn multiple agent runs in quick succession. This created a re-trigger churn where the agent ran repeatedly, each time processing the same stale wakes and generating redundant observations [142][145].
Third, fragile run ID derivation. The agent derived its run_id from the conversation history—specifically, by counting distinct run_id values in the stored messages. When the session reset cleared all messages, the run counter reset to zero. This broke deduplication logic and left the agent in a state where it could not properly identify new runs [142][152].
The Fixes: Debounce, Mark Wakes, Monotonic IDs
The assistant implemented three fixes, each targeting a distinct failure mode [145][146][147][148][149][150][151][152][153].
Mark due wakes as processed. The wake-processing loop was modified to update the status of processed wakes to "processed" in the database, so they would be excluded from future queries. This immediately eliminated the 20 stale wakes that were being reprocessed every cycle [142][149].
Debounce triggerAgent() to suppress burst triggers. The Go backend's triggerAgent() function was modified to check the trigger file's modification time before writing. If the file was modified within the last 2 seconds for priority 0 events (human messages) or 10 seconds for priority 1 events (state changes), the trigger was skipped. This prevented bursty events from spawning redundant runs while still ensuring that genuine events were processed promptly [145][146].
Make run ID monotonic via session state. The run counter was moved from conversation-history derivation to persistent session state, ensuring that even after a session reset, the run counter continued from where it left off. This provided a stable identity for each run and fixed the deduplication logic that had broken after resets [152][153].
A POST /api/agent/mark-wakes endpoint was also added to allow manual clearing of processed wakes, giving operators a recovery mechanism when the wake queue became corrupted [149][150].
The Deeper Lesson
The context overflow crisis revealed something fundamental about autonomous agent systems: the conversation history was being used for too many purposes simultaneously. It served as the LLM prompt, the session state store, the run counter, and the audit log. When one function was disrupted (clearing history), all other functions broke simultaneously [142].
The fixes addressed this fragility by separating concerns. Run identity moved to persistent state. Wake processing became idempotent. Event triggering gained rate limiting. Each subsystem was given its own stable foundation, independent of the conversation history that could be reset at any time.
Synthesis: The Architecture of Autonomous Agent Reliability
The three threads of this chunk—diagnostic grounding, verdict-based context management, and event-driven reliability—are not independent. They form a coherent architecture for autonomous agent reliability that addresses the unique failure modes of LLM-driven systems.
Grounding replaces speculation. The diagnostic sub-agent system ensures that destructive actions are based on evidence, not inference. By gating stop_instance behind a diagnosis precondition, the architecture forces the agent to ground its decisions in facts before acting.
Structure replaces ambiguity. The verdict JSON transforms the agent's unstructured natural language output into machine-readable data. This enables reliable context pruning, operational auditing, and automated decision tracking.
Discipline replaces chaos. The file lock, debounce mechanism, and monotonic run IDs impose operational discipline on a system that would otherwise be vulnerable to race conditions, burst triggers, and state corruption.
These three patterns—grounding, structure, and discipline—are the foundation of trustworthy autonomy. They do not eliminate the LLM's capacity for error, but they contain it within a framework that prevents catastrophic outcomes and enables graceful recovery.
Conclusion
The work in this chunk transformed an autonomous agent from a fragile prototype into a hardened production system. The diagnostic grounding system replaced speculation with evidence, preventing the agent from destroying healthy instances based on misinterpreted signals. The verdict system and context management overhaul kept the agent's memory clean and structured, eliminating duplicate runs and no-op pollution. The debounce mechanism, wake processing fix, and monotonic run IDs stabilized the agent loop against the combinatorial chaos of event-driven triggers and session resets.
Each of these fixes was born from a production failure—a moment when the agent did something catastrophically wrong, and the system's designers had to ask not just "how do we prevent this specific mistake?" but "what architectural pattern would make this class of mistake impossible?" The answers—evidence gating, structured verdicts, operational discipline—are not specific to GPU fleet management. They are general principles for building autonomous systems that can be trusted with real-world consequences.