Grounding the Autonomous Agent: Building a Diagnostic Sub-Agent System

In the high-stakes world of autonomous fleet management for GPU proving infrastructure, a single bad decision can cost hours of compute time and significant money. This is the story of message [msg 4897] in a sprawling coding session—a message that represents a critical architectural pivot from brittle hard-coded rules to an evidence-driven, LLM-powered diagnostic layer. It is the moment the assistant stopped trying to constrain the agent's behavior through prohibitions and instead gave it the tools to know before it acted.

The Context: A Fleet-Wide Disaster

Moments before this message, the autonomous fleet management agent had committed a catastrophic error. Observing a fleet of 12 instances that were still in their startup phase—showing cuzk_alive=false and 0% GPU utilization—the agent concluded they were "stuck" and proceeded to destroy every single one of them. These were perfectly healthy machines that simply hadn't finished their 1–2 hour initialization process. The user's reaction was immediate and sharp: the agent had "gotten impatient and killed instances that were progressing just fine" ([msg 4889]).

The assistant's first instinct was the engineer's default: add a guard. In message [msg 4888], it had already begun implementing a hard time-based protection on the stop_instance tool, rejecting any attempt to stop an instance less than two hours old. But the user rejected this approach with a crucial insight: "Noo it is fine that the agent may destroy shorter running instances, it just needs to ground itself in truth first and never speculate whether instance state is bad or not" ([msg 4894]).

This distinction is profound. The problem was not that the agent was too aggressive—it was that the agent was acting on speculation rather than evidence. The user wanted the agent to be empowered to make any decision, including destroying young instances, but only after it had gathered real evidence about their health. This shifted the architectural question from "how do we prevent bad actions?" to "how do we ensure good information?"

The Message: A Parallel Construction

Message [msg 4897] is the assistant's response to this redirection. It is deceptively brief in its visible text—just a single sentence announcing the plan—but it dispatches two parallel task tool calls that together represent a fundamental re-architecture of the agent's decision-making pipeline.

The message reads:

Now build the diagnostic endpoint and sub-agent. Let me do the Go endpoint and Python tool in parallel:

What follows are two task invocations, each spawning a sub-agent session to build a component of the diagnostic system. The assistant dispatches them simultaneously because they are architecturally independent: one builds the backend data collection layer (Go), the other builds the frontend interpretation layer (Python). The parent session blocks until both sub-agents complete, then proceeds with the results.

The Two Components of the Diagnostic System

Component 1: The Go Diagnostic Endpoint (GET /api/agent/diagnose/{vast_id})

The first task builds a server-side endpoint that performs the raw data collection. When called, it SSHes into the specified vast.ai instance and gathers:

• Entrypoint and daemon logs — the startup sequence and any error messages
• Process listings — what's actually running on the machine
• Memory and GPU statistics — resource utilization and health indicators
• System state — overall machine health The endpoint is designed with a fallback: if SSH is broken or unreachable (a common failure mode in cloud GPU instances), it falls back to data available through the vast.ai API, ensuring the diagnostic always returns something useful. This endpoint lives in agent_api.go at route /api/agent/diagnose/ and is registered alongside the other agent management endpoints.

Component 2: The Python diagnose_instance Tool

The second task builds the Python-side tool that the agent's LLM actually calls. This tool:

1. Calls the Go endpoint to get raw diagnostic data
2. Feeds that data to a sub-agent LLM — a separate, focused LLM call with domain knowledge about normal startup sequences, common failure modes, and expected timelines
3. Returns a structured verdict about the instance's health The sub-agent LLM is given context about what constitutes normal behavior for cuzk instances: that cuzk_alive=false is expected during the first 1–2 hours of startup, that params_done state means the instance is progressing normally, and that 0% GPU utilization is not itself a sign of failure during initialization. This domain knowledge is the key differentiator—the main agent was making decisions without this context, while the sub-agent has it baked into its prompt.

The Architectural Shift: From Prohibition to Evidence

The most significant aspect of this message is what it represents architecturally. The assistant had initially reached for a prohibition-based approach: add a rule that says "don't stop instances under 2 hours old." This is the classic engineering solution—hard constraints that prevent bad outcomes by forbidding certain actions.

The user's redirection pushed the system toward an evidence-based approach: instead of forbidding the action, require evidence before the action can be taken. The stop_instance tool was subsequently gated with an HTTP 428 "Precondition Required" status, meaning the main agent must call diagnose_instance first and receive a verdict before it can proceed with stopping or destroying an instance.

This is a fundamentally more robust architecture for several reasons:

1. It preserves agent autonomy — the agent can still make any decision, including destroying young instances, if the evidence supports it
2. It prevents speculation — the agent can no longer act on hunches or incomplete observations
3. It creates an audit trail — every destructive action is preceded by a diagnostic verdict that can be reviewed
4. It separates concerns — the diagnostic sub-agent is specialized for interpretation, while the main agent is specialized for decision-making

The Thinking Process: Parallelization and Independence

The assistant's decision to dispatch both tasks in parallel reveals a deliberate architectural understanding. The Go endpoint and the Python tool are independent in their implementation but tightly coupled in their function. The Go endpoint could be built and tested without the Python tool (it returns raw JSON that can be inspected manually), and the Python tool could be built against a mock endpoint. By dispatching them simultaneously, the assistant maximizes the use of the sub-agent system's parallel execution capability.

This also reflects a mature understanding of the task tool's semantics. The parent session is blocked until both sub-agents complete, but the sub-agents run independently. This is ideal for components that have no circular dependencies—neither task needs the output of the other to complete its work.

Assumptions and Input Knowledge

To understand this message, one must grasp several layers of context:

• The fleet management system consists of a Go backend (vast-manager) that manages instances on vast.ai, a Python agent (vast_agent.py) that makes LLM-driven decisions, and a fleet of GPU instances running cuzk proving software
• The agent architecture uses an LLM (qwen3.5-122b) that calls tools defined in Python, which in turn call Go API endpoints
• Instance startup takes 1–2 hours — a critical domain fact that the main agent lacked and that the diagnostic sub-agent would be given
• The task tool spawns sub-agents that run to completion before returning, allowing parallel construction of independent components The assistant assumes that SSH access to instances is generally available (with a fallback for when it isn't), that the sub-agent LLM can reliably interpret log data, and that the main agent will comply with the precondition requirement (a prompt-level constraint).

Output Knowledge Created

This message creates the diagnostic sub-agent system, which fundamentally changes how the autonomous agent operates. Before this message, the agent could call stop_instance directly based on its own interpretation of fleet observations. After this message, the agent must:

1. Call diagnose_instance(vast_id) to gather evidence
2. Receive a structured verdict from the sub-agent LLM
3. Only then call stop_instance or destroy_vast_instance The system produces structured diagnostic verdicts that distinguish between "healthy/progressing," "warning (e.g., OOM but recovering)," and "error (crashed/failed)." This structured output is consumed by the main agent's decision-making process and is also stored in the conversation history for audit and learning.

Conclusion

Message [msg 4897] is a turning point in the development of this autonomous fleet management system. It represents the moment when the architecture shifted from a brittle system of prohibitions to a robust system of evidence-grounded decision-making. The diagnostic sub-agent system it creates is not just a tool—it is a new layer of intelligence that sits between observation and action, ensuring that the agent's destructive capabilities are only exercised when supported by real evidence.

The parallel construction of the Go endpoint and Python tool demonstrates efficient use of the sub-agent system, while the architectural insight—that the agent should be empowered rather than constrained, but only when grounded in facts—reflects a mature understanding of autonomous system design. In the subsequent messages of the session, this diagnostic system would prove its worth, preventing further speculative destructions and giving the operator confidence that the agent's actions are evidence-based rather than capricious.