Grounding the Autonomous Agent: Building a Diagnostic Sub-Agent to Replace Speculation with Evidence

Introduction

In the lifecycle of any autonomous system, there comes a moment when the gap between what the agent thinks is happening and what is actually happening becomes a destructive force. Message [msg 4890] captures exactly such a moment in the development of a fleet management agent for a distributed GPU proving infrastructure. An LLM-driven agent, tasked with scaling a cluster of GPU instances running the CuZK proof generation engine, had just committed a catastrophic error: it systematically destroyed every instance in the fleet—instances that were, in fact, progressing through their startup sequence perfectly normally. The user's report was blunt: "Model seems to have gotten impatient and killed instances that were progressing just fine."

This message is the assistant's response to that failure. It is a turning point—a deliberate pivot from building hard-coded protective guards toward a fundamentally different architectural philosophy. Instead of constraining the agent's actions with brittle rules, the assistant proposes giving the agent the ability to ground itself in evidence before acting. The centerpiece of this shift is a diagnostic sub-agent system: a layered architecture where the main agent delegates deep investigation to a specialized sub-agent that SSHes into instances, reads raw logs, interprets them with codebase context, and returns a structured verdict. This message, though brief, contains the seed of a design that would reshape the entire agent architecture over the following chunks.

The Context: A Fleet-Wide Destruction Event

To understand why this message matters, one must understand what preceded it. The autonomous agent had been designed to manage a fleet of GPU instances on vast.ai, scaling them up and down based on Curio SNARK demand. In [msg 4889], the user presented a detailed log of the agent's catastrophic reasoning. The agent observed 12 instances in various startup states—registered, params_done—with cuzk_alive=false and gpu_util=0. It interpreted these signals as evidence that "the worker process is not running" and that instances were "stuck for 0.8-1.1 hours, which is abnormally long for benchmarking/SRS loading."

This interpretation was catastrophically wrong. The instances were progressing normally through a multi-hour startup sequence that includes downloading parameters, running benchmarks, and loading SRS data. The cuzk_alive=false flag during this phase is expected behavior—the proving daemon only starts after the benchmark completes. The agent, lacking any mechanism to verify its assumptions, called stop_instance on three of the most advanced instances and, in a subsequent cycle, destroyed the entire fleet.

The root cause was not malice or incompetence. It was a fundamental architectural gap: the agent had no way to distinguish between "instance is broken" and "instance is in a normal startup phase that happens to look like a broken state to an external observer." Its only signals were high-level status fields (registered, params_done, cuzk_alive) that were designed for fleet-level monitoring, not for diagnostic depth. The agent was forced to speculate, and speculation led to destruction.

The Assistant's Reasoning: From Guards to Grounding

The assistant's initial response, visible in the reasoning section of [msg 4890], reveals an immediate recognition of the problem and a rapid evolution of the solution strategy. The assistant begins by listing four tasks: "Fix the immediate problem," "Add a 'sub-agent' tool," ensure the sub-agent has codebase access, and "add the startup protection guard I was working on."

The startup protection guard was the assistant's first instinct—a hard rule preventing stop_instance and destroy_vast_instance from acting on instances under two hours old. This is a classic engineering reflex: when an agent makes a bad decision, constrain its decision space. But even as the assistant writes this plan, a deeper architectural insight is forming.

The reasoning text shows the assistant thinking through the sub-agent architecture in real time:

"The Go side can handle the heavy lifting—SSH into the instance, collect logs and system metrics, then pass that raw data to a focused LLM that interprets it with knowledge of what normal behavior looks like. This keeps the main agent from making hasty decisions while giving it reliable diagnostic information."

This is the critical insight. The assistant recognizes that the problem is not that the agent has too much freedom, but that it has too little information. A hard guard would prevent the destruction of young instances, but it would also prevent legitimate actions—what if an instance is genuinely broken within the first two hours? The guard is a blunt instrument that sacrifices correctness for safety.

The sub-agent approach is more elegant. It preserves the agent's full decision space but demands that decisions be grounded in evidence. The agent must know before it acts. This shifts the architecture from a constraint-based model ("you cannot do X") to an evidence-based model ("you may do X only after you have proven Y").

The Architecture: A Two-Layer Diagnostic Pipeline

The assistant's reasoning sketches a two-layer architecture that would become the foundation of the diagnostic system.

Layer 1: The Go Endpoint (GET /api/agent/diagnose/{vast_id}). This is the data collection layer. The vast-manager server SSHes into the target instance and collects raw telemetry: entrypoint logs, daemon logs, process listings (ps aux), memory statistics, GPU state, and any error messages from systemd journal or application logs. The Go side handles the messy reality of SSH connectivity—timeouts, authentication failures, machines that are unreachable. It returns a structured blob of raw data.

Layer 2: The Sub-Agent LLM. This is the interpretation layer. A smaller, focused LLM call receives the raw diagnostic data along with context from the codebase about what normal behavior looks like. The assistant envisions the sub-agent returning a structured verdict: "healthy/progressing," "warning: OOM but recovering," or "error: crashed with detail." This verdict is what the main agent sees—not raw log lines, but a distilled, evidence-based assessment.

The assistant explicitly notes that the sub-agent should have "read-only access to curio/cuzk (what we have in ./) codebase, tools to search (grep, ag, read tool) the code." This is crucial: the sub-agent needs to understand what normal looks like for this specific software stack. It needs to know that cuzk_alive=false during the first 1-2 hours is expected, that params_done means the parameter download succeeded and the benchmark is starting, that high memory usage during SRS loading is normal. Without this domain knowledge, the sub-agent would make the same mistake as the main agent.

The Todowrite: A Plan in Motion

The message concludes with a structured todowrite block that crystallizes the plan into three high-priority items:

"Add startup protection to stop_instance AND destroy_vast_instance (reject <2h non-running)" — status: in_progress
"Build GET /api/agent/diagnose/{vast_id} - SSH into instance, collect logs/ps/memory" — status: pending
"Build diagnose_instance tool - sub-agent LLM reads logs and returns structured health" — status: pending The todowrite format is itself revealing. It shows the assistant's working process: a structured todo list embedded in the reasoning, with priority levels and status tracking. This is not just a response to the user—it is the assistant organizing its own execution plan, treating the coding session as a project management environment.

The Immediate Aftermath: A Design Reversal

What happens immediately after [msg 4890] reveals the most interesting part of the story. In [msg 4891], the assistant compiles the startup guard and confirms it builds. But then, in [msg 4894], the user intervenes with a crucial correction:

"Noo it is fine that the agent may destroy shorter running instances, it just needs to ground itself in truth first and never speculate whether instance state is bad or not. Maybe make it possible to call Stop only after the agent has queried the 'state discovery agent' and grounded itself that the instance is indeed in a bad state."

This is the user rejecting the hard-guard approach and endorsing the evidence-based approach. The assistant immediately pivots in [msg 4895], reverting the startup guard and committing fully to the diagnostic sub-agent architecture. The todowrite is updated: "Revert startup guard on stop_instance — wrong approach" becomes the new high-priority item, and the diagnostic endpoint and sub-agent tool are promoted to the core of the plan.

This exchange is a beautiful example of collaborative design. The assistant proposed two solutions—a guard and a diagnostic system—and the user correctly identified which one addressed the root cause. The guard treated the symptom (young instances being killed); the diagnostic system treated the cause (the agent acting on speculation). The assistant, to its credit, recognized the wisdom of this correction immediately and reoriented its entire execution plan.

Assumptions and Knowledge

The message makes several assumptions that are worth examining. First, it assumes that SSH access to the instances is available and reliable. In practice, as later chunks would reveal, SSH connectivity to vast.ai instances can be intermittent, and the diagnostic system would need a fallback to vast API data when SSH fails. Second, it assumes that a smaller LLM can reliably distinguish normal startup behavior from genuine failure. This is a non-trivial AI safety problem—the sub-agent must be robust against both false positives (calling a healthy instance broken) and false negatives (missing a real crash).

The input knowledge required to understand this message is substantial. One must understand the vast.ai instance lifecycle (the registered → params_done → benchmarking → running sequence), the CuZK proving stack (what cuzk_alive means, what SRS loading entails), the existing agent architecture (how stop_instance and destroy_vast_instance work), and the SSH-based monitoring infrastructure. The message assumes all of this context.

The output knowledge created by this message is equally significant. It establishes the architectural blueprint for the diagnostic sub-agent system that would be built over the following messages. It shifts the design philosophy from constraint-based to evidence-based. And it documents a critical lesson in autonomous agent design: speculation is the enemy of reliability, and grounding in evidence is the only sustainable path to trustworthy autonomy.

Conclusion

Message [msg 4890] is a hinge point in the development of this autonomous fleet management system. It captures the moment when a catastrophic failure is transformed into a design insight. The assistant's reasoning moves from "how do we prevent the agent from making this mistake again" to "how do we give the agent the tools to never need to speculate in the first place." The diagnostic sub-agent architecture that emerges from this message—a Go endpoint for data collection paired with an LLM for evidence-based interpretation—represents a principled solution to one of the hardest problems in autonomous systems: the gap between what an agent can observe and what it needs to know to act wisely.