The Sub-Agent That Diagnoses Without SSH: A Case Study in Graceful Degradation

The Message

Replace the SSH-failed early return with a fallback that still uses the sub-agent: [edit] /tmp/czk/cmd/vast-manager/agent/vast_agent.py Edit applied successfully.

This message, from an assistant building an autonomous fleet management agent for GPU proving infrastructure, is deceptively brief. On its surface, it reports a single file edit: replacing an early return path in the Python diagnostic sub-agent with a fallback that still invokes the LLM even when SSH connectivity fails. But behind this small change lies a cascade of architectural decisions, real-world debugging, and a fundamental shift in how the system handles uncertainty. Understanding this message requires tracing the reasoning that led to it, the assumptions that were tested and broken, and the output knowledge it created for the broader system.

The Context: Why This Message Was Written

To understand why this edit was necessary, one must understand the crisis that precipitated it. The autonomous fleet agent — a Python-based LLM-driven system that manages GPU instances on vast.ai for SNARK proving — had made a catastrophic error. It had stopped all running instances despite 59 pending tasks, because it misinterpreted active=False as "no demand" rather than "all workers are dead with tasks queued." This failure revealed a fundamental vulnerability: the agent was making irreversible decisions based on speculation rather than evidence.

The user's directive was clear and sharp: the agent must never speculate about instance health. It must ground itself in facts before acting. The solution was a diagnostic sub-agent — a secondary LLM call, fed with raw data from the instance (logs, processes, memory stats, GPU state), that returns a structured verdict about whether the instance is healthy, progressing, or genuinely broken. The main agent would be required to call diagnose_instance before it could invoke stop_instance, enforced by an HTTP 428 precondition on the stop endpoint.

The assistant built this system in two layers. First, a Go endpoint (GET /api/agent/diagnose/{vast_id}) that SSHes into the instance, collects entrypoint logs, daemon logs, process lists, memory usage, and GPU information. Second, a Python tool (diagnose_instance) that calls this endpoint and feeds the raw data to a sub-agent LLM with domain knowledge about normal startup sequences and failure modes.

The Discovery That Changed Everything

When the assistant deployed and tested the diagnostic endpoint, SSH failed. Not because of missing keys — the vast-manager's SSH key was properly configured on vast.ai — but because the specific instances had "messed up SSH setup," as the user explained. The assistant tried direct IP SSH, vast relay SSH, and SSH from different hosts. Some instances were reachable, most were not. The SSH protocol itself was unreliable on this fleet.

This discovery was a moment of architectural reckoning. The original design assumed SSH would be the primary data channel for diagnostics. If SSH was unreliable, the entire sub-agent system would be useless for most instances, and the agent would be back to guessing. The assistant could have accepted this limitation, treating unreachable instances as opaque boxes. Instead, it asked: what data is available when SSH fails?

The answer came from three sources: the vast.ai API (which provides memory usage, GPU utilization, and status), the manager's own SQLite database (which tracks instance state, age, registration status, and SSH commands), and the portavailc tunnel (a reverse connection from instances back to the manager). The assistant enhanced the Go endpoint to gather all of this fallback data and include it in the diagnostic response alongside the SSH error ([msg 4914]). It also added this data to the successful SSH case, recognizing that extra context is always useful ([msg 4915]).

The Subject Message: What Actually Changed

This brings us to message 4918. The Go endpoint now returns rich fallback data when SSH fails, but the Python sub-agent had a problem: its run_instance_diagnosis function had an early return path for unreachable instances. When SSH failed, the function returned a minimal response without invoking the sub-agent LLM at all. The fallback data was being collected on the Go side but never analyzed on the Python side.

The edit replaced that early return with logic that still feeds the available data to the sub-agent LLM. Even without SSH logs or process lists, the sub-agent can interpret vast API metrics, DB state, and the SSH error itself to produce a structured verdict. The LLM has enough domain knowledge to distinguish between "instance is still booting (params_done state, no SSH yet)" and "instance is crashed (running state, no SSH, no GPU activity)" based on the available metadata. This transforms the diagnostic system from a brittle SSH-dependent pipeline into a gracefully degrading multi-source analyzer.

Assumptions Made and Broken

Several assumptions were tested in this sequence. The first was that SSH would be a reliable data channel for diagnostics. This assumption was reasonable given that the vast-manager's SSH key was authorized, but it failed because individual instances had broken SSH configurations — a fleet management problem that the key-level fix couldn't address.

The second assumption was that an SSH failure meant the diagnostic sub-agent had nothing useful to say. The early return pattern reflected this: if we can't SSH, return reachable: false and stop. The edit in message 4918 explicitly rejects this assumption, recognizing that partial data with LLM interpretation is far more valuable than no data at all.

A third, more subtle assumption was that the sub-agent's value came primarily from raw log analysis. The assistant's initial design emphasized SSH-collected data (entrypoint logs, daemon logs, process lists). The fallback design revealed that the sub-agent's real value is in its ability to synthesize diverse signals — API metrics, DB state, error messages — into a coherent judgment, even when the richest data sources are unavailable.

Input and Output Knowledge

To understand this message, one needs input knowledge of: the autonomous agent architecture (main agent + sub-agent), the diagnostic endpoint design (Go SSH collector + Python LLM analyzer), the SSH connectivity failure on specific vast.ai instances, the fallback data sources (vast API, SQLite DB, portavailc tunnel), and the run_instance_diagnosis function's early return pattern.

The output knowledge created by this message is a hardened diagnostic pipeline that works even when SSH fails. The sub-agent can now process instances in any connectivity state, producing structured verdicts from whatever data is available. This directly enables the main agent to make grounded decisions about stopping or destroying instances, fulfilling the user's core requirement: never speculate, always diagnose first.

The Thinking Process

The reasoning visible across this sequence shows a pattern of iterative refinement driven by real-world constraints. The assistant initially built a clean two-layer system (Go collects, Python analyzes) with a clear failure mode (SSH fails → early return). When reality broke the SSH assumption, the assistant didn't patch the symptom — it re-architected the data flow to make the system robust to the failure mode itself.

The key insight was recognizing that the sub-agent LLM is not merely a log parser but a general-purpose reasoning engine. It can work with whatever data it receives. The assistant's edit in message 4918 is the culmination of this insight: instead of treating SSH failure as a dead end, treat it as a reduced-data scenario that the LLM can still handle. This is a design philosophy that values graceful degradation over brittle correctness, and it's what makes the diagnostic sub-agent genuinely useful in a fleet where connectivity is never guaranteed.

Conclusion

Message 4918 is a small edit — a single file change in a Python agent script. But it represents a significant architectural evolution: from an SSH-dependent diagnostic system that fails when its primary channel is unavailable, to a multi-source diagnostic system that adapts to whatever data is available. The edit encodes the lesson that in distributed systems, partial information with intelligent interpretation is almost always better than no information at all. For the autonomous fleet agent, this means it can now make grounded decisions about instance health even when the instances themselves are unreachable — a critical capability for managing GPU proving infrastructure at scale.