Grounding the Autonomous Agent: How a Diagnostic Sub-Agent System Prevented Catastrophic Speculation in GPU Fleet Management

Introduction

On March 17, 2026, an autonomous LLM-driven fleet management agent made a catastrophic decision. It looked at a set of instances showing cuzk_alive=false and gpu_util=0%, concluded they were broken, and destroyed eight perfectly healthy GPU machines that were simply in the middle of their normal startup sequence. This incident—described in the assistant's summary message at <msg id=4929>—catalyzed a fundamental architectural shift in how the agent makes decisions. The assistant moved from brittle hard-coded rules to an evidence-driven, multi-layered diagnostic system that forces the agent to ground its decisions in empirical facts before taking destructive actions.

This article examines that single message in depth: why it was written, the reasoning that shaped the solution, the assumptions that were tested and revised, and the architectural knowledge that emerged from the process.

The Context: An Agent That Couldn't Tell Startup from Failure

The fleet management agent was designed to autonomously scale a cluster of GPU instances running the CuZK proving engine on vast.ai. It monitored Curio SNARK demand, launched and stopped instances, and aimed to keep the fleet cost-efficient while meeting proof generation targets. But the agent had a critical blind spot: it could only observe surface-level metrics like cuzk_alive (a boolean indicating whether the cuzk daemon process was running) and gpu_util (GPU utilization percentage). These metrics are deeply ambiguous during the instance startup phase.

When a fresh GPU instance boots on vast.ai, it goes through a multi-stage initialization process: container image pull, parameter download (often gigabytes of SRS parameters), benchmarking, and finally cuzk daemon startup. During parameter download and benchmarking, cuzk_alive is naturally false—the daemon hasn't started yet. GPU utilization is 0% because the GPU is idle while the CPU downloads files. Memory usage may be high (50-90%) as SRS parameters are loaded. To an agent that only sees the raw metrics, this looks identical to a broken instance: no daemon running, no GPU activity, high memory. The agent, lacking any deeper understanding, assumed the worst and terminated the instances.

This was not a minor bug. It was a fundamental failure of the agent's decision-making architecture. The agent was making life-or-death decisions about compute instances based on speculative interpretations of ambiguous data.

The Reasoning: Why Hard Guards Were the Wrong Approach

The assistant's initial instinct was to implement a hard time-based guard—preventing the agent from killing any instance younger than a certain age. This is a natural engineering response: if the agent can't be trusted to make good decisions about young instances, simply take that option away. But the user pushed back on this approach, and the assistant recognized the deeper problem.

The key insight, articulated in the assistant's message at <msg id=4929>, is that the agent must be allowed to make any decision—including stopping a young instance—but only after grounding itself in facts. A hard guard would prevent the agent from stopping a genuinely broken young instance (e.g., one that crashed immediately after boot). The right solution is not to restrict the agent's options, but to ensure it has the information it needs to make good choices.

This is a profound design principle for autonomous agents. The assistant could have taken the easy path: add a min_age_hours check to stop_instance and call it done. Instead, it recognized that the real problem was epistemic—the agent didn't know enough to distinguish healthy startups from genuine failures—and built a system to give it that knowledge.

The Architecture: Three Layers of Grounding

The diagnostic sub-agent system that the assistant built and summarized in <msg id=4929> operates at three layers:

Layer 1: Data Collection (The Go Endpoint)

The first layer is a Go HTTP endpoint (GET /api/agent/diagnose/{vast_id}) that SSHes into the target instance and collects raw diagnostic data: running processes, entrypoint log tail, daemon log tail, memory usage, GPU state, and dmesg output. This is the empirical foundation—raw facts from the machine itself, not aggregated metrics from the vast.ai API.

The assistant discovered during implementation that SSH connectivity to vast.ai instances is unreliable. Some instances accept SSH connections, others reject them despite having the correct public key configured. The user confirmed this at <msg id=4907>: "The vast-manager pubkey is setup in vast, the only reason it's not authorized is some instances have messed up ssh setup." This meant the diagnostic system needed a fallback path.

When SSH fails, the endpoint falls back to data from two alternative sources: the vast.ai API (providing actual_status, mem_usage, gpu_util, and instance age) and the manager's local SQLite database (providing the instance's registration state, registration timestamp, and benchmark rate). The response explicitly notes that "SSH broken does NOT mean unhealthy"—a crucial signal to prevent the sub-agent from misinterpreting the absence of SSH data as evidence of a problem.

Layer 2: Interpretation (The Python Sub-Agent)

The raw data from Layer 1 is fed to a Python sub-agent—a secondary LLM call with specialized domain knowledge about CuZK instance startup. This sub-agent is the heart of the system. It knows what's normal:

Layer 3: Enforcement (The HTTP 428 Precondition)

The final layer is enforcement. The stop_instance endpoint checks whether a diagnose action has been logged for the target instance within the last 10 minutes. If not, it returns HTTP 428 "Precondition Required" with a message telling the agent to call diagnose_instance first. The assistant verified this worked correctly at <msg id=4927>, where a stop attempt on instance 33037353 (which had no recent diagnosis) was cleanly blocked with HTTP 428.

This gating mechanism is elegant because it doesn't hard-code any business logic about when stopping is allowed. It simply enforces a procedural requirement: you must look before you leap. The agent remains free to stop any instance—but only after it has gathered and interpreted the evidence.

The Verification: Testing the Full Flow

The assistant didn't just build the system and declare it done. It ran a series of verification tests that reveal the careful engineering mindset behind the implementation.

First, it tested the diagnose endpoint on two instances: one with SSH reachable (33034189) and one without (33034145). Both returned useful data. The SSH-reachable case showed actual processes and log tails. The SSH-unreachable case returned fallback data from the vast API and manager DB, with an explicit note that SSH failure doesn't imply unhealthiness. Both responses were rich enough for the sub-agent to produce a meaningful verdict.

Second, it tested the grounding enforcement. An initial test at <msg id=4922> seemed to show the stop going through without a diagnosis—but the assistant correctly identified that this was because the earlier curl test had already logged a diagnose action for that instance. The system was working as designed. A subsequent test on a fresh instance (33037353) at <msg id=4927> confirmed the HTTP 428 block.

Third, the assistant verified the sub-agent's interpretation by examining the structured verdicts. The sub-agent correctly identified that an instance in params_done state with param download at 99% was progressing normally—exactly the kind of judgment that the main agent had failed to make when it destroyed eight instances.

Assumptions, Mistakes, and Lessons Learned

Several assumptions were tested and revised during this work:

Assumption: SSH will work for diagnosis. This was wrong. Many vast.ai instances have broken SSH setups despite having the correct public key configured. The assistant discovered this empirically at <msg id=4911>, where three different SSH attempts (direct IP, vast relay, and another direct) all failed with "Permission denied." This led to the fallback mechanism, which turned out to be essential.

Assumption: Hard time-based guards are the right approach. The assistant initially proposed a startup age guard. The user's rejection of this approach (implicitly, through the direction to build the diagnostic system instead) led to a much better architecture. The lesson is that restricting agent capabilities is inferior to giving agents better information.

Mistake: The initial diagnose endpoint logged actions that satisfied the grounding check for subsequent stop attempts. This was actually correct behavior—the system was working as designed—but it meant that a human manually testing the endpoint could inadvertently authorize a stop. The 10-minute window is a reasonable compromise between usability and safety.

Mistake: The stop endpoint initially had no grounding check at all. The assistant had to add the HTTP 428 gating after building the diagnose endpoint. This is a good example of building the enforcement mechanism after the data collection mechanism—a sensible order that ensured the enforcement had something to enforce against.

Input Knowledge Required

To understand the diagnostic sub-agent system, one needs knowledge of several domains:

Output Knowledge Created

The diagnostic sub-agent system produced several forms of reusable knowledge:

Conclusion

The diagnostic sub-agent system described in <msg id=4929> represents a critical turning point in the fleet management agent's evolution. The assistant moved from a world where the agent made catastrophic decisions based on superficial metrics to a world where every destructive action is preceded by evidence gathering and expert interpretation. The architecture is notable for its layered design, its graceful handling of unreliable infrastructure, and its philosophical commitment to grounding over restriction.

The message itself is a summary—a status report after deployment and verification—but it captures the essence of a much deeper engineering journey. The assistant didn't just add a safety check; it fundamentally rethought how an autonomous agent should relate to the systems it controls. The agent is no longer allowed to speculate. It must ground itself in facts.