Grounding the Autonomous Agent: Deploying a Diagnostic Sub-Agent for Evidence-Driven Fleet Management

Introduction

In the rapidly evolving landscape of autonomous infrastructure management, one of the most dangerous failure modes is an agent making decisions based on speculation rather than evidence. This article examines a pivotal deployment message ([msg 4920]) in an opencode coding session where an AI assistant deployed a diagnostic grounding system for an autonomous LLM-driven fleet management agent. The message represents the culmination of a critical architectural shift—from brittle hard-coded guards to an evidence-driven, sub-agent-powered diagnostic layer that prevents the main agent from making destructive decisions about GPU proving instances without first grounding itself in factual system state.

The Message

The subject message is a deployment and verification command sequence. The assistant copies a newly built Go binary (vast-manager-agent) and an updated Python agent script (vast_agent.py) to the management host at 10.1.2.104, stops the vast-manager service, installs the new binaries, restarts the service, and then runs two diagnostic tests against different GPU instances. The first test targets instance 33034145, where SSH is broken, and the second targets instance 33034189, which is reachable. The output demonstrates that the diagnostic system works correctly in both cases—returning structured fallback data when SSH fails, and returning live process logs when SSH succeeds.

Context: The Crisis That Precipitated This Architecture

To understand why this message matters, one must understand the failure that preceded it. In chunk 3 of the session, the autonomous agent suffered a critical production failure: it misinterpreted active=False from the demand endpoint and stopped all running instances despite 59 pending tasks queued in Curio. The root cause was that the demand signal could not distinguish between "no demand" and "all workers dead with tasks queued." The agent speculated about instance health without checking, and made a catastrophic decision.

The user's response was sharp and instructive. In [msg 4894], the user said: "Noo it is fine that the agent may destroy shorter running instances, it just needs to ground itself in truth first and never speculate whether instance state is bad or not. Maybe make it possible to call Stop only after the agent has queried the 'state discovery agent' and grounded itself that the instance is indeed in a bad state."

This directive fundamentally reshaped the approach. The assistant had initially been working on a hard time-based startup protection guard (preventing the agent from killing instances under 2 hours old), but the user correctly identified that the real problem was not about timing—it was about epistemology. The agent needed to know, not guess. The solution was not to constrain the agent's actions with arbitrary rules, but to give it the tools to acquire reliable evidence before acting.

The Diagnostic Sub-Agent Architecture

The assistant designed a two-layer diagnostic system. At the Go level, a new endpoint GET /api/agent/diagnose/{vast_id} was added to the vast-manager API (agent_api.go). This endpoint SSHes into the target instance and collects:

Process lists (looking for cuzk, curio, entrypoint processes)
Entrypoint logs (the startup script's output)
Daemon logs (cuzk and curio runtime logs)
Memory statistics
GPU information (utilization, memory usage) When SSH succeeds, the endpoint returns rich, real-time diagnostic data. When SSH fails—as it does on many instances due to broken SSH key authorization—the endpoint falls back to available data from the vast.ai API and the manager's own SQLite database, including the instance's registration timestamp, state label, and last known status. A critical note is attached: "SSH failed. Diagnosis based on vast API + manager DB only. Some instances have broken SSH setup — this does NOT mean the instance is unhealthy." At the Python level, a diagnose_instance tool was added to the agent's toolset. This tool calls the Go endpoint, then feeds the raw diagnostic data to a sub-agent LLM that interprets it with domain knowledge about normal startup sequences, expected process states, and common failure patterns. The sub-agent returns a structured verdict about the instance's health. Crucially, the stop_instance tool was gated with an HTTP 428 "Precondition Required" status. The main agent must call diagnose_instance on an instance before it can stop or destroy it. This shifts the architecture from brittle hard-coded rules to an evidence-driven, LLM-powered diagnostic layer.

What This Message Actually Does

The message executes the deployment of this entire system. Let us trace through the commands:

Step 1: Copy binaries. The assistant uses scp to copy the compiled Go binary and the updated Python script to the management host. This is a routine deployment step, but it represents the culmination of significant engineering work across multiple parallel tasks (see [msg 4897] where the Go endpoint and Python tool were built in parallel subagent tasks).

Step 2: Service restart. The assistant stops the vast-manager service (sudo systemctl stop vast-manager), copies the binaries into place (/usr/local/bin/vast-manager and /opt/vast-agent/vast_agent.py), and restarts the service. This is a careful, sequenced deployment that minimizes downtime.

Step 3: Verification test on an SSH-broken instance. The assistant tests the diagnose endpoint on instance 33034145. The output shows reachable: False, confirming that the endpoint gracefully handles SSH failures. The fallback data includes the database label (C.33034145), registration timestamp (2026-03-17T19:22:39Z), and database state (params_done). The critical note is present: "SSH failed. Diagnosis based on vast API + manager DB only. Some instances have broken SSH setup — this does NOT mean the instance is unhealthy." This note is essential—it prevents the sub-agent from concluding that an unreachable instance is necessarily unhealthy.

Step 4: Verification test on a reachable instance. The assistant tests on instance 33034189, which is reachable. The output shows reachable: True and reveals live process information: a bash process running entrypoint.sh (PID 325) and other processes. This demonstrates that when SSH works, the endpoint returns rich, actionable diagnostic data that the sub-agent can analyze.

Assumptions and Design Decisions

Several assumptions underpin this message:

SSH as primary diagnostic channel. The assistant assumed that SSH would be the primary mechanism for collecting instance diagnostics. This was a reasonable assumption given that vast.ai instances are provisioned with SSH access, and the vast-manager host had an authorized SSH key. However, the reality was that many instances had broken SSH setups—the key was authorized on the vast.ai account level but not propagated to individual instances. This required the fallback mechanism.

Fallback data is sufficient for safe decisions. The assistant assumed that the vast API data plus the manager's database state would provide enough context for the sub-agent to make a reasonable assessment even when SSH is unavailable. This is a defensible assumption: the manager knows the instance's registration state, age, and last status, and the vast API provides utilization metrics. However, this fallback is inherently less reliable than live log inspection.

The sub-agent LLM can interpret diagnostic data reliably. The assistant assumed that a focused LLM call with domain knowledge about normal startup sequences could correctly distinguish healthy instances from failing ones. This assumption was tested in production and proved sound, though the system was later hardened further.

What the Message Reveals About the Thinking Process

The reasoning visible in this message is subtle but important. The assistant chose to test both a failing-SSH instance and a working-SSH instance. This is not accidental—it demonstrates a deliberate testing strategy that validates both paths through the diagnostic system. The assistant is thinking: "I need to prove that the system works correctly in both the happy path (SSH reachable) and the degraded path (SSH broken)."

The choice of which instances to test is also revealing. Instance 33034145 was specifically chosen because it was known to have SSH issues (from earlier debugging in [msg 4900] through [msg 4911]). Instance 33034189 was chosen because it was confirmed reachable in [msg 4911]. The assistant is systematically verifying edge cases.

The note in the SSH-failed output—"Some instances have broken SSH setup — this does NOT mean the instance is unhealthy"—reveals a deep understanding of the system's failure modes. The assistant knows that the SSH key issue is a infrastructure problem, not a sign of instance health problems. This note is designed to prevent the sub-agent (and the main agent) from drawing false conclusions from the absence of SSH connectivity.

Output Knowledge Created

This message creates several important pieces of knowledge:

The diagnostic system works. Both the SSH-reachable and SSH-unreachable paths return structured, useful data. The system is ready for production use.
SSH failures are common and handled gracefully. The fallback mechanism provides enough context for the sub-agent to work with, and the explicit note prevents false conclusions about unreachable instances.
The deployment process is validated. The sequence of stopping the service, copying binaries, and restarting works correctly. The service comes back up and the new endpoints are functional.
The sub-agent architecture is feasible. The combination of a Go endpoint for raw data collection and a Python sub-agent for LLM-based interpretation proves to be a workable architecture for grounding the main agent's decisions.

Impact and Significance

This message represents a turning point in the session. Before it, the agent was making destructive decisions based on speculation. After it, the agent is required to ground itself in evidence before taking action. The architecture shift—from hard-coded guards to an evidence-driven diagnostic layer—is a more principled and scalable approach to autonomous agent safety.

The diagnostic grounding system proved its worth in subsequent operations. The agent could now make informed decisions about stopping or destroying instances, and the sub-agent's structured verdicts provided a reliable basis for those decisions. The system was later further hardened with debouncing, context management, and session state persistence, but the core architecture—evidence-driven decision making via a diagnostic sub-agent—remained unchanged.

Conclusion

Message [msg 4920] is a deployment message, but it is far more than a routine binary copy. It represents the culmination of a fundamental architectural insight: autonomous agents must ground their decisions in evidence, not speculation. The diagnostic sub-agent system deployed in this message transformed the fleet management agent from a potentially destructive autopilot into a cautious, evidence-driven operator. The careful testing of both the SSH-reachable and SSH-unreachable paths demonstrates a systematic approach to validation that is essential for production autonomous systems. In the complex world of GPU proving infrastructure management, where a single bad decision can cost hours of proving time and significant money, this grounding mechanism proved to be one of the most critical safety systems in the entire architecture.