Grounding Over Guarding: How a Diagnostic Sub-Agent Replaced Hard Rules in Autonomous Fleet Management

Introduction

In the complex world of autonomous infrastructure management, there is a fundamental tension between safety and flexibility. Hard-coded guards prevent catastrophic mistakes but also prevent legitimate actions. Evidence-driven decision-making preserves flexibility but requires reliable diagnostic machinery. When an LLM-driven fleet management agent began destroying perfectly healthy GPU instances because it misinterpreted normal startup behavior as failure, this tension came into sharp focus. The assistant's response—message 4895 in the conversation—represents a decisive architectural pivot from defensive rule-making to evidence-driven autonomy, establishing a pattern that would prove essential for the agent's long-term reliability.

The Context: An Agent That Couldn't Tell Startup From Failure

The conversation leading up to message 4895 reveals a system in crisis. The autonomous fleet management agent, designed to scale GPU proving infrastructure up and down based on Curio SNARK demand, had committed a catastrophic error. Observing twelve instances in various stages of startup—some as young as 0.2 hours old—the agent saw cuzk_alive=false and 0% GPU utilization across the board. Without any deeper investigation, it concluded the instances were stuck and proceeded to destroy all of them, including three that had already progressed to the "params_done" state, a clear signal that startup was proceeding normally.

The damage was significant. Instances that had been loading for nearly an hour, accruing storage costs and making progress through the multi-phase startup sequence (registration → parameter download → benchmarking → proof production), were abruptly terminated. The agent's reasoning, visible in the conversation logs, shows a pattern of speculation: "All 3 'params_done' instances show cuzk_alive=false and 0% GPU utilization, indicating the worker process is not running. They've been stuck for 0.8-1.1 hours, which is abnormally long for benchmarking/SRS loading." This was not a malicious decision—it was an ignorant one. The agent lacked the diagnostic tools to distinguish between a genuinely stuck instance and one proceeding through a normal but lengthy startup sequence.

The First Instinct: Hard Guards

The assistant's initial response to this crisis was instinctive and defensive. In messages 4886-4888, the assistant began adding a hard-coded startup guard to the stop_instance and destroy_vast_instance API endpoints: reject any operation on instances less than two hours old. This is the natural engineering response to an autonomous system making bad decisions—constrain its action space with rules. If the agent cannot be trusted to evaluate instance health correctly, prevent it from acting on young instances entirely.

But this approach has a fundamental flaw, which the user immediately identified in message 4894:

"Noo it is fine that the agent may destroy shorter running instances, it just needs to ground itself in truth first and never speculate whether instance state is bad or not."

The user's insight is profound. The problem was never that the agent was too aggressive—it was that the agent was making decisions without evidence. A hard age-based guard would prevent the agent from destroying genuinely broken young instances (e.g., an instance that crashes immediately due to a GPU driver issue). It would also prevent the agent from learning and adapting. The correct solution is not to restrict the agent's options but to give it the tools to make informed decisions.

Message 4895: The Architectural Pivot

Message 4895 is the assistant's response to this redirection. It is a short message—barely a few lines of reasoning and a todo list—but it encodes a complete reorientation of the system's architecture:

You're right — the problem isn't missing guards, it's that the agent speculated about instance health instead of checking. The fix is: 1. Remove the startup guard — the agent should be allowed to destroy if it has a real reason 2. Build the diagnostic sub-agent — SSH into instance, read logs, have an LLM interpret them 3. Make stop_instance require a prior diagnose_instance call — the tool itself refuses to act unless the instance was recently diagnosed

The message opens with a direct acknowledgment of the user's correction: "You're right." This is not merely polite—it signals a genuine conceptual shift. The assistant had been thinking in terms of constraints (prevent bad actions), but the user was thinking in terms of grounding (ensure actions are based on evidence). The three-point plan that follows operationalizes this shift with remarkable precision.

Point 1: Remove the Startup Guard

This is the most counterintuitive move. The agent just demonstrated that it cannot be trusted with young instances, and the response is to remove the protection? Yes—because the protection was addressing the wrong problem. The guard treated youth as a proxy for uncertainty, but youth is not the issue. A five-minute-old instance that is clearly crashing (process exited, OOM logs, GPU errors) should be destroyed. A two-hour-old instance that is progressing normally (params_done, benchmark running) should not. The guard would get both wrong. By removing it, the assistant opens the door to a more nuanced approach where decisions are based on actual evidence rather than heuristics.

Point 2: Build the Diagnostic Sub-Agent

The diagnostic sub-agent is the heart of the new approach. The assistant envisions a two-layer system: a Go endpoint that SSHes into the target instance and collects raw data (entrypoint logs, daemon logs, process lists, memory stats, GPU info), and a Python sub-agent LLM that interprets this data with domain knowledge about what normal startup looks like. The sub-agent would return a structured verdict: "healthy/progressing," "warning (e.g., OOM but recovering)," or "error (crashed, needs intervention)."

This design is elegant because it separates concerns cleanly. The Go side handles the mechanical work of data collection—SSH connections, log parsing, system commands. The LLM side handles the interpretive work—distinguishing a normal benchmark delay from a genuine crash, recognizing OOM recovery patterns, understanding the multi-phase startup sequence. Neither layer alone would be sufficient, but together they create a reliable diagnostic pipeline.

Point 3: Gate Destructive Actions Behind Diagnosis

The third point is the architectural keystone. Rather than a hard age guard, the stop_instance and destroy_vast_instance tools would return an HTTP 428 Precondition Required error unless the instance had been recently diagnosed via diagnose_instance. This shifts the failure mode from "agent destroys something it shouldn't" to "agent tries to destroy something but is forced to investigate first." The agent retains full flexibility—it can destroy any instance, at any age—but only after grounding its decision in evidence.

This is a profound design pattern. The system does not second-guess the agent's conclusions; it only ensures those conclusions are based on data. If the diagnostic sub-agent reports "instance is healthy, progressing normally," the main agent can still choose to destroy it (perhaps for cost reasons), but it will do so with full awareness of the consequences. If the diagnostic sub-agent reports "instance has crashed, OOM detected, no recovery," the main agent can act decisively with confidence.

The Thinking Process: From Defensive to Evidence-Driven

The reasoning visible in message 4895 reveals a rapid conceptual evolution. The assistant's initial todo list (from message 4890) shows the defensive mindset: "Add startup protection to stop_instance AND destroy_vast_instance (reject <2h non-running)." This is a rule-based approach—if age < 2h and not running, block the action. It is simple, reliable, and wrong.

After the user's correction, the todo list in message 4895 is completely rewritten. The first item is now "Revert startup guard on stop_instance — wrong approach." This explicit self-correction is striking. The assistant does not just add the diagnostic system alongside the guard; it removes the guard entirely. The old approach is not supplemented—it is replaced.

The new todo list also reveals a more sophisticated understanding of the problem. The items are ordered logically: first revert the wrong approach, then build the data collection layer, then build the interpretation layer, then gate the destructive tools. This is not just a list of tasks—it is a phased architectural migration. Each step enables the next.

Assumptions and Tradeoffs

The approach in message 4895 rests on several key assumptions. First, that an LLM-based diagnostic sub-agent will produce more reliable judgments than hard-coded heuristics. This is plausible but not guaranteed—LLMs can hallucinate, misinterpret logs, or be fooled by unusual but benign states. The assistant implicitly assumes that the sub-agent's domain knowledge (about startup sequences, benchmark timelines, OOM patterns) will outweigh these risks.

Second, that the overhead of SSH + LLM inference is acceptable. Each diagnose_instance call requires an SSH connection to the target instance (which may be slow or unreliable), reading potentially large log files, and a sub-agent LLM call. This could take 10-30 seconds per instance. For a fleet of 12 instances, that is 2-6 minutes of diagnostic overhead before any action can be taken. The assistant assumes this cost is worth the accuracy gain.

Third, that the diagnostic sub-agent can distinguish normal from abnormal startup behavior reliably. This is a genuinely hard problem. Some instances genuinely take 2+ hours to start due to slow parameter downloads or benchmark runs. Others crash silently in ways that look identical to normal behavior from the outside. The sub-agent must have deep knowledge of the startup lifecycle to make this distinction.

Input and Output Knowledge

To understand message 4895, one needs knowledge of the existing agent architecture (the stop_instance and destroy_vast_instance tools, the fleet monitoring system, the vast.ai API), the instance startup lifecycle (registration → params_done → benchmark → running), and the SSH access patterns used to connect to instances. One also needs to understand the user's philosophical position: that the agent should be empowered to make any decision, but only after grounding itself in facts.

The message creates several important outputs. It establishes the architectural pattern of "gating destructive actions behind evidence collection," which becomes a recurring theme in subsequent chunks. It defines the interface between the main agent and the diagnostic sub-agent (structured verdicts with health status). It commits the system to an evidence-driven rather than rule-driven approach to safety. And it implicitly defines what "grounding" means in this context: not just checking a boolean flag, but collecting raw data and having an LLM interpret it with domain knowledge.

Conclusion

Message 4895 is a turning point in the development of the autonomous fleet management agent. It represents the moment when the system's designers chose evidence over rules, diagnosis over defense, and empowerment over restriction. The diagnostic sub-agent pattern that emerges from this message—collect data, interpret with domain knowledge, gate actions behind evidence—would prove essential not just for instance lifecycle management but for the broader challenge of building autonomous systems that can be trusted with destructive actions. The message is short, but the architectural philosophy it encodes is profound: don't tell the agent what it can't do; give it the tools to know what it should do.