Grounding the Autonomous Agent: The SSH-Fallback Diagnostic Sub-Agent

In the course of building a fully autonomous LLM-driven fleet management agent for a distributed GPU proving infrastructure, a critical architectural insight emerged: the agent must never speculate about instance health. This principle, hard-won after the agent destroyed eight perfectly healthy instances that were in their normal startup phase, drove the creation of a diagnostic sub-agent system. The subject message — a brief but pivotal transition point in that implementation — captures the moment when the assistant pivoted to handle a real-world operational reality: SSH is broken on many vast.ai instances, and the diagnostic system must gracefully degrade without losing its grounding function.

The Message

The complete subject message reads:

[assistant] Now update the Python sub-agent to handle the SSH-failed case with fallback data: [grep] def run_instance_diagnosis Found 1 matches /tmp/czk/cmd/vast-manager/agent/vast_agent.py: Line 1026: def run_instance_diagnosis(vast_id: int) -> dict:

At first glance, this is a simple grep-and-plan message. The assistant states its intent, runs a search to locate the relevant function, and receives confirmation that run_instance_diagnosis lives at line 1026 of vast_agent.py. But this surface simplicity belies the depth of reasoning and context that produced it.

The Crisis That Preceded It

The diagnostic sub-agent system was born from a production incident ([msg 4890]). The autonomous agent, observing instances with cuzk_alive=false and gpu_util=0%, concluded they were broken and stopped them. In reality, these instances were progressing through normal startup phases — parameter downloads, SRS file loading, benchmark execution — where GPU utilization is expected to be zero and the cuzk daemon hasn't started yet. The agent had no way to distinguish "healthy startup" from "stuck failure" because it was making decisions based on surface-level metrics alone.

The user's directive was clear and decisive ([msg 4894]): the agent must be allowed to make any decision, but only after grounding itself in facts. The solution was not to add hard-coded guards (the assistant's initial instinct, which the user correctly rejected), but to build a diagnostic layer that forces the agent to investigate before acting. This led to a two-component architecture: a Go endpoint (GET /api/agent/diagnose/{vast_id}) that SSHes into instances to collect raw diagnostic data, and a Python sub-agent LLM that interprets that data with domain knowledge about normal startup sequences versus actual failures.

The SSH Discovery

When the assistant deployed and tested the diagnostic endpoint ([msg 4900]), the results were sobering. The endpoint returned reachable: false with empty fields for processes, logs, memory, and GPU data. SSH simply could not reach the instances. The assistant initially suspected a missing SSH key on the vast-manager host ([msg 4903]), but the user clarified ([msg 4907]): the key was already authorized on vast.ai; the instances themselves had broken SSH setups. This was not a configuration error but a systemic reality of the vast.ai marketplace — many rented GPU instances ship with SSH in a broken state.

Further investigation ([msg 4908], [msg 4911]) confirmed the pattern: some instances were reachable (e.g., 33034189 at 141.195.21.87), but most returned Permission denied (publickey). The assistant catalogued the fallback options ([msg 4912]): checking the cuzk status API via the portavailc tunnel, using vast.ai API data (memory usage, GPU utilization, status), and reporting what the manager database knows (state, age, registration time).

The Go-Side Enhancement

Before the subject message, the assistant enhanced the Go endpoint in two edits ([msg 4914], [msg 4915]). The SSH-failed case was updated to include a fallback object with vast API data (actual_status, mem_usage, gpu_util, age, label) and manager DB data (state, registered_at, ssh_cmd, bench_rate). A crucial note was added: "SSH failed. Diagnosis based on vast API + manager DB only. Some instances have broken SSH setup — this does NOT mean the instance is unhealthy." The successful SSH case was also enhanced to include this fallback data as supplementary context.

The Subject Message's Role

The subject message is the bridge between the Go-side enhancement and the Python-side consumption of that enhancement. The Go endpoint now returns rich fallback data when SSH fails, but the Python sub-agent function run_instance_diagnosis — which calls the endpoint and feeds the raw data to the sub-agent LLM — was still written assuming SSH would succeed. If SSH failed, the function would receive empty fields and produce a diagnosis based on nothing.

The assistant's reasoning, visible in the transition from [msg 4915] to [msg 4916], is: "The Go side now provides fallback data. The Python side must be updated to use it. The sub-agent LLM should still run even when SSH fails, interpreting the fallback data with the same domain knowledge about startup sequences." The grep confirms the function location, and the next message ([msg 4918]) applies the edit: replacing the SSH-failed early return with a fallback path that still invokes the sub-agent.

Assumptions and Knowledge

The message rests on several assumptions. First, that the sub-agent LLM can produce useful diagnoses from partial data (vast API metrics + DB state) rather than requiring full SSH access. This is a reasonable assumption given that the vast API provides actual_status, memory usage, and GPU utilization — signals that, combined with DB state and age, can distinguish "healthy startup" from "stuck failure" in many cases. Second, that running the sub-agent with fallback data is strictly better than returning an error — the agent at least gets some grounded assessment. Third, that the run_instance_diagnosis function is the correct place to make this change, which the grep confirms.

The input knowledge required to understand this message is substantial. One must know that the diagnostic sub-agent has two components (Go collector + Python interpreter). One must understand that the Go endpoint was just enhanced to include fallback data. One must recognize that run_instance_diagnosis at line 1026 is the function that calls the endpoint and invokes the sub-agent LLM. And one must grasp the broader architectural principle: the agent must ground itself in facts before acting, and the diagnostic system is the mechanism for that grounding.

The output knowledge created by this message is the confirmed function location and the explicit intent to update it. This sets up the subsequent edit and, ultimately, the successful deployment where the diagnostic system correctly handles both SSH-reachable and SSH-unreachable instances ([msg 4920]).

Architectural Significance

This message represents a key architectural decision: the diagnostic sub-agent must be resilient to infrastructure failures. SSH is a fragile transport — instances lose connectivity, keys expire, ports change. Building the diagnostic system to gracefully degrade when SSH fails, rather than treating SSH as a hard requirement, makes the entire agent loop more robust. The fallback data from vast API + manager DB is sufficient for the sub-agent to make grounded assessments in most cases, and the explicit note that "SSH broken does NOT mean unhealthy" prevents the sub-agent itself from speculating.

The deployment test ([msg 4920]) validated this approach. Instance 33034145 (SSH unreachable) returned rich fallback data: state params_done, registered at 19:22, vast status running. Instance 33034189 (SSH reachable) showed actual processes and logs. Both paths fed into the same sub-agent LLM, producing structured verdicts that the main agent could act on. The grounding check on stop_instance was also verified ([msg 4927]): attempting to stop an undiagnosed instance returned HTTP 428 "Precondition Required," forcing the agent to call diagnose_instance first.

Conclusion

The subject message, brief as it is, captures a critical moment of architectural adaptation. The assistant recognized that the diagnostic sub-agent system, designed around SSH-based data collection, needed to handle the real-world reality of broken SSH on vast.ai instances. By updating the Python sub-agent to consume the Go endpoint's fallback data, the assistant ensured that the grounding principle — the agent must never speculate about instance health — holds even when the primary data channel fails. This is the essence of robust autonomous system design: not assuming ideal conditions, but engineering for the failures that will inevitably occur.