Grounding the Autonomous Agent: Building a Diagnostic Sub-Agent System for Evidence-Driven Fleet Management

In the high-stakes world of distributed GPU proving infrastructure, an autonomous agent's most dangerous quality is not incompetence—it is unwarranted certainty. This lesson was learned the hard way when an LLM-driven fleet management agent, observing a cluster of freshly launched instances with zero GPU utilization and no running worker process, concluded they were "stuck" and systematically destroyed every single one. The instances were not stuck; they were in the normal startup phase, downloading parameters and initializing benchmarks—a process that routinely takes one to two hours. The agent had speculated about instance health instead of verifying it, and the fleet paid the price.

The message at the center of this analysis—message index 4900 in the conversation—captures the immediate aftermath of that failure. It is a deceptively simple bash command, executed on the management host, that tests a newly built diagnostic endpoint:

ssh theuser@10.1.2.104 "
echo '=== Test diagnose endpoint ==='; curl -sf http://127.0.0.1:1236/api/agent/diagnose/33034145 | python3 -c '
import json, sys
d = json.load(sys.stdin)
print(f\"reachable: {d.get(\"reachable\")}\")
for k in [\"processes\", \"entrypoint_log\", \"daemon_log\", \"memory\", \"gpu\"]:
    v = d.get(k, \"\")
    print(f\"{k}: {v[:120] if v else \"(empty)\"}...\")
'" 2>&1

The output confirms the endpoint is functional but reveals that the target instance (33034145) is unreachable: reachable: False, with all diagnostic fields empty. This result, while seemingly mundane, represents a critical architectural pivot—from brittle rule-based guards to an evidence-driven, LLM-powered diagnostic layer that fundamentally reshapes how the autonomous agent makes decisions about instance lifecycle.

The Failure That Demanded a New Architecture

To understand why this message was written, one must first understand the failure that precipitated it. In the preceding conversation (see [msg 4884]), the agent had been observing a fleet of twelve freshly launched instances. All showed cuzk_alive=false and gpu_util=0—perfectly normal indicators for instances that had been running for only 0.2 to 1.1 hours. The agent, however, interpreted these signals through a lens of operational urgency: "All 3 'params_done' instances show cuzk_alive=false and 0% GPU utilization, indicating the worker process is not running. They've been stuck for 0.8-1.1 hours, which is abnormally long for benchmarking/SRS loading." This was speculation, not diagnosis. The agent had no evidence that the instances were actually stuck—it simply assumed they were, based on surface-level metrics that were, in fact, entirely consistent with normal startup behavior.

The user's response (see [msg 4894]) was both corrective and instructive: "Noo it is fine that the agent may destroy shorter running instances, it just needs to ground itself in truth first and never speculate whether instance state is bad or not." This single sentence encapsulates a profound design insight for autonomous systems. The problem was not that the agent made a destructive decision—the problem was that it made that decision without evidence. The user proposed a specific architectural remedy: "Maybe make it possible to call Stop only after the agent has queried the 'state discovery agent' and grounded itself that the instance is indeed in a bad state."

This directive triggered a rapid and consequential design pivot. The assistant had initially responded to the crisis with a hard-coded guard—a time-based rule that would reject stop/destroy requests for instances under two hours old (see [msg 4888]). The user's intervention redirected this approach entirely. The correct architecture was not to constrain the agent's actions with arbitrary rules, but to constrain its decision-making process by requiring evidence. The agent must be free to make any decision, but only after it has grounded itself in verifiable facts.

Building the Diagnostic Sub-Agent System

The assistant immediately pivoted. The startup guard was reverted (see [msg 4896]), and two parallel tasks were launched simultaneously (see [msg 4897]): one to build a Go diagnostic endpoint (GET /api/agent/diagnose/{vast_id}) that SSHes into instances to collect logs, process lists, memory stats, and GPU information; and another to build a Python diagnose_instance tool that feeds this raw data to a sub-agent LLM for interpretation.

The Go endpoint, registered at /api/agent/diagnose/ in the vast-manager's route table, performs the heavy lifting of data collection. It SSHes into the target instance, reads entrypoint and daemon logs, captures process output via ps aux, checks memory usage, and queries GPU state. When SSH is unavailable—as the test in our subject message reveals—it returns whatever data it can gather from the vast.ai API as a fallback, clearly marking the instance as unreachable. This graceful degradation is essential for a system operating on remote, potentially unreliable infrastructure.

The Python diagnose_instance tool, defined in the agent's tool set, represents the true innovation. It takes the raw diagnostic data from the Go endpoint and passes it to a focused LLM call—a "sub-agent" that has been primed with domain knowledge about normal startup sequences, expected timelines, and common failure modes. This sub-agent does not make decisions; it only interprets evidence. It returns a structured verdict: whether the instance is healthy and progressing normally, whether there are warnings (such as OOM events with automatic recovery), or whether there is a confirmed error state with specific details. This verdict then becomes the factual basis for the main agent's subsequent decisions.

The gating mechanism is equally elegant. The stop_instance tool is designed to return HTTP 428 (Precondition Required) if the instance has not been recently diagnosed. This forces the main agent to call diagnose_instance first, establishing a mandatory evidence-gathering step before any destructive action can be taken. The architecture enforces the workflow without hard-coding business rules about instance age, GPU utilization thresholds, or any other heuristic that would inevitably prove brittle in edge cases.

The Test Result and Its Implications

The subject message captures the first live test of this new diagnostic system. The assistant SSHes into the management host and curls the diagnose endpoint for instance 33034145—an instance that was presumably among those destroyed in the earlier incident, or one that had been left in a broken state. The result is unambiguous: reachable: False. All diagnostic fields—processes, entrypoint log, daemon log, memory, GPU—are empty.

This result is significant for several reasons. First, it validates that the endpoint is functioning correctly: it returns a structured response even when SSH fails, rather than crashing or hanging. The reachable boolean provides a clear signal that the instance is not accessible, and the empty fields are a truthful representation of the data available. Second, it demonstrates the system's resilience: the endpoint does not throw an error or return an incomplete response—it returns a complete, well-structured diagnostic object that accurately reflects the situation.

Third, and most importantly, this test reveals a deeper operational reality: instances in this infrastructure frequently become unreachable. The diagnostic system is not an academic exercise—it addresses a genuine need. When an instance is unreachable, the diagnostic endpoint's fallback to vast.ai API data (such as status, GPU utilization, and memory usage from the provider's perspective) still provides useful information. The sub-agent LLM can interpret this limited data and offer a probabilistic assessment: "Instance is unreachable via SSH but vast.ai reports 'running' status with 0% GPU util. This could indicate a boot failure or network issue. Recommend waiting 15 minutes and re-diagnosing before taking action."

Assumptions, Knowledge, and Design Philosophy

The diagnostic sub-agent system rests on several key assumptions. The first is that the sub-agent LLM, with appropriate domain priming, can reliably distinguish normal startup behavior from genuine failures. This is a non-trivial assumption—it requires that the sub-agent understand the expected sequence of states (registered → params_done → benchmark → running), the typical duration of each phase (1-2 hours total), and the significance of various log messages. The system prompt for the sub-agent must encode this domain knowledge with sufficient precision that the LLM does not reproduce the same speculation error that the main agent made.

The second assumption is that the gating mechanism (HTTP 428) will reliably prevent the main agent from bypassing the diagnostic step. This depends on the LLM's ability to interpret the 428 response and correctly infer that it must call diagnose_instance first. If the main agent misinterprets the error or attempts to work around it, the gating mechanism fails. The assistant mitigated this risk by making the tool definition for stop_instance explicitly state that a prior diagnose_instance call is required, and by including the 428 precondition in the tool's error response.

The third assumption is that SSH access to instances will be sufficiently reliable for the diagnostic system to function. The test in our subject message reveals that this assumption is not always valid—instance 33034145 was unreachable. The fallback to vast.ai API data mitigates this, but the diagnostic quality is necessarily degraded when direct access is unavailable. The system must operate effectively even with partial information.

The input knowledge required to understand this message is substantial. One must understand the broader context of the autonomous agent's failure, the user's corrective directive, the architectural pivot from rule-based guards to evidence-driven diagnosis, and the mechanics of the Go diagnostic endpoint and Python sub-agent tool. Without this context, the bash command appears to be a trivial test of a simple endpoint. With it, the message becomes a pivotal moment in the evolution of a complex autonomous system—the first verification that a new architectural paradigm is functioning as designed.

The output knowledge created by this message is equally significant. It confirms that the diagnostic endpoint is operational and returns structured responses. It reveals that instance 33034145 is unreachable, providing a real-world data point about infrastructure reliability. It validates the graceful degradation path when SSH is unavailable. And it sets the stage for the next phase of development: refining the sub-agent's diagnostic accuracy, handling edge cases in the gating mechanism, and iterating on the system based on operational experience.

The Thinking Process Behind the Message

The reasoning visible in this message reflects a methodical, test-driven approach to infrastructure development. The assistant did not simply deploy the diagnostic endpoint and assume it worked—it immediately tested it against a real instance, verified the response structure, and confirmed that the fallback behavior functioned correctly. This testing discipline is essential for autonomous systems where failures can cascade rapidly and destructively.

The choice of instance 33034145 for the test is also telling. This was likely an instance that had been destroyed or was in a known broken state—a deliberate choice to test the endpoint's behavior under adverse conditions rather than against a healthy, responsive instance. This reflects a "test with the worst case first" mentality that is well-suited to production infrastructure development.

The message also reveals the assistant's awareness of the broader system architecture. The test validates not just the Go endpoint but the entire diagnostic pipeline: the SSH connection logic, the log collection routines, the JSON response formatting, and the Python parsing code on the client side. Each component must work correctly for the diagnostic system to function, and this single test exercises the entire chain.

Conclusion

Message 4900 appears, at first glance, to be a routine infrastructure test—a bash command that curls an endpoint and prints the result. But in the context of the conversation, it represents a critical architectural inflection point. The autonomous agent had failed catastrophically by making decisions based on speculation rather than evidence. The diagnostic sub-agent system was the response: a layered architecture that separates evidence gathering from decision-making, enforces a mandatory diagnostic step before destructive actions, and uses a focused LLM to interpret raw operational data with domain-specific knowledge.

The test result—reachable: False—is not a failure of the diagnostic system but a validation of its necessity. Instances in this infrastructure are frequently unreachable, and the system must handle that gracefully. The diagnostic endpoint does exactly that: it returns a complete, truthful assessment of what it can and cannot determine, allowing the sub-agent and main agent to make informed decisions with full awareness of the uncertainty.

This message encapsulates a fundamental lesson for autonomous systems engineering: the most dangerous agent is not one that makes bad decisions, but one that makes bad decisions with unwarranted confidence. By requiring evidence before action, the diagnostic sub-agent system transforms the agent from a speculative executor into a grounded decision-maker—and in so doing, makes the entire fleet more resilient to the inevitable uncertainties of distributed infrastructure.