Grounding the Autonomous Fleet Manager: Building a Diagnostic Sub-Agent for Evidence-Driven Decision Making

Introduction

In the lifecycle of any complex autonomous system, there comes a moment when the designers must confront a fundamental question: how do we prevent the agent from making catastrophic decisions based on incomplete information? For the autonomous GPU fleet management agent built in this coding session, that question was answered with a single architectural intervention: a diagnostic grounding system that forces the agent to collect empirical evidence before taking any destructive action. This chunk of the session documents the complete implementation of that system—from the initial discovery that the UI code lived in a different file, through the careful sequencing of six coordinated edits across two files, to the final compilation verification that confirmed everything was wired together correctly.

The narrative arc of this chunk is a masterclass in defensive design for LLM-based autonomous systems. It demonstrates how an AI assistant, guided by a detailed user specification, can implement a multi-layered safety architecture that combines tool definitions, execution handlers, sub-agent LLM reasoning, system prompt rules, API-level enforcement, and UI transparency into a coherent whole. Along the way, the assistant navigates file structure discoveries, ordering dependencies, LSP errors, and the tension between sequential editing and holistic understanding—all while maintaining a disciplined commitment to verification and closure.

The Crisis That Demanded Grounding

The diagnostic grounding system was born from a painful production incident. Earlier in the session, the autonomous agent had misinterpreted a demand signal and stopped all running GPU instances despite 59 pending SNARK proof tasks queued for proving. The root cause was that the agent could not distinguish "no demand" from "all workers are dead but tasks are still queued." It made a life-or-death decision about expensive GPU instances based on speculation rather than evidence [9].

This failure exposed a fundamental architectural flaw. The agent had tools to launch and destroy instances, but it had no reliable way to determine what was actually happening on a running machine. Was it loading SRS (a 44 GiB pinned memory operation taking 30-45 minutes)? Was it downloading parameters (10-30 minutes)? Had it crashed with a CUDA error? The agent had no way to know—it was operating on thin API signals like cuzk_alive and state fields that could mean multiple things depending on context [9].

The user's directive was clear and firm: the agent must never stop or destroy an instance based on speculation. It must diagnose first. This principle drove the design of the diagnostic grounding system that this chunk brings into being [9].

The Specification: A Blueprint for Grounded Decision-Making

The user's specification in [msg 0] laid out a complete blueprint for the diagnose_instance tool. The architecture had three layers:

Data Collection Layer (Go backend): The GET /api/agent/diagnose/{vast_id} endpoint SSHes into instances to collect raw logs, process lists, memory stats, GPU status, dmesg OOM messages, and uptime. It has a fallback to vast.ai API data when SSH is broken [9].

Interpretation Layer (Python sub-agent): The run_instance_diagnosis function takes this raw data, formats it into a structured diagnostic context, and sends it to an LLM sub-agent with the DIAG_SYSTEM_PROMPT—a meticulously crafted prompt that encodes deep domain knowledge about the cuzk prover's normal startup sequence [9].

Enforcement Layer (HTTP 428 precondition): The stop_instance tool is gated with an HTTP 428 "Precondition Required" status. The API rejects stop requests unless the agent has called diagnose_instance within the last 10 minutes. This shifts the architecture from "the agent must follow rules" to "the agent must gather evidence before acting" [9].

The DIAG_SYSTEM_PROMPT is arguably the most important artifact in the entire system. It describes a five-phase normal startup sequence that takes 1-2 hours total: memcheck/memprobe (5-10 min), params download (10-30 min), benchmark with SRS loading (30-45 min), benchmark result reporting, and finally production running. The prompt explicitly tells the sub-agent that cuzk_alive=false during SRS loading is NORMAL, that GPU util 0% during startup phases is NORMAL, and that "overlay rename" errors in the daemon log are non-fatal. It also defines what is truly abnormal: OOM kill messages, CUDA errors, segfaults, entrypoint stuck in a registration loop for more than 10 minutes, and instances older than 2 hours with no progress past the "registered" state [9].

The First Step: Reading Before Building

The assistant began by following the user's instruction to "read the file first." In [msg 1], the assistant read the beginning of /tmp/czk/cmd/vast-manager/agent/vast_agent.py, seeing the module docstring and the start of imports. In [msg 2], the assistant strategically read the context window management section near the end of the file (lines 1296-1305), revealing how the agent manages its token budget through estimate_tokens and SUMMARIZE_THRESHOLD [1][16].

This reading strategy was methodical: rather than reading a 1300+ line file sequentially, the assistant sampled key sections to build a cognitive map of the file's architecture. The context window management section was particularly relevant because the diagnose_instance tool would produce substantial output that must be managed within the conversation's token budget [16].

The Discovery That Changed the Plan

After completing the file read, the assistant made a critical discovery. The user's specification had assumed that the UI tools panel lived in vast_agent.py, referencing a renderConversation() function that was supposed to contain the "Agent Tools (12 available)" section. But the assistant noted in [msg 3]: "I notice there's no renderConversation() or UI tools panel in this file" [2].

Rather than proceeding under the user's assumption, the assistant performed targeted grep searches. A search for renderConversation found two matches, both in a completely different file: /tmp/czk/cmd/vast-manager/ui.html. A subsequent grep for Agent Tools.*available in [msg 4] confirmed the exact line at line 2141 of ui.html [3][11].

This discovery was a pivotal moment. It meant the assistant would need to edit two files instead of one, and it had to understand the structure of the UI code before making changes. In [msg 5], the assistant read the renderConversation() function in ui.html starting at line 2126, gaining the structural context needed to plan the UI edit [11].

The Pivot Point: Planning All Edits

With full context established, the assistant declared in [msg 6]: "Now I have full context. Let me apply all edits." The assistant laid out a six-item plan in dependency order [8]:

Add diagnose_instance to TOOL_DEFINITIONS
Add to execute_tool
Add the DIAG_SYSTEM_PROMPT and run_instance_diagnosis function
Update system prompt CRITICAL RULES
Update stop_instance description
Update UI tools panel This ordering reveals a dependency-aware approach to implementation. The tool must be defined before it can be dispatched, the function must exist before it can be called, the prompt rules must mandate the tool's use, the tool description must reinforce the mandate, and finally the UI must reflect the new capability [8].

The Six Edits: Building the System Layer by Layer

The assistant executed the six edits across messages [msg 6] through [msg 13], each building on the previous one.

Edit 1 ([msg 6]): Added the diagnose_instance entry to TOOL_DEFINITIONS, defining its schema with a vast_id integer parameter and a description that emphasized its mandatory role: "REQUIRED before stopping any instance. SSH into an instance and analyze its logs, processes, and resource usage. Returns a structured health verdict from a diagnostic sub-agent" [8][15].

Edit 2 ([msg 8]): Added the handler in execute_tool that routes diagnose_instance calls to the run_instance_diagnosis function. This edit created a deliberate LSP error—"run_instance_diagnosis" is not defined—because the function didn't exist yet. The assistant acknowledged this in [msg 9] with "Good — the function doesn't exist yet," treating the LSP error as an expected waypoint rather than a failure [7][13].

Edit 3 ([msg 9]): Added the DIAG_SYSTEM_PROMPT constant and the run_instance_diagnosis function, placing them right before execute_tool. The function orchestrates the entire diagnostic workflow: it calls the Go API endpoint to collect raw data, formats a diagnostic context, invokes the sub-agent LLM with the specialized prompt, parses the JSON verdict using a regex fallback, and returns a structured result with status, phase, summary, details, and action fields. If the instance is unreachable, it returns status: "error" with action: "wait"—encoding the lesson that SSH failures can be temporary and should not trigger destruction [13].

Edit 4 ([msg 10]): Updated the system prompt's CRITICAL RULES section with the directive: "NEVER stop or destroy an instance based on speculation. You MUST call diagnose_instance() first. The stop_instance API will REJECT your request if you haven't diagnosed the instance in the last 10 minutes." This rule transforms the diagnostic tool from an optional capability into a mandatory workflow, encoded in the agent's core operating instructions [14].

Edit 5 ([msg 11]): Updated the stop_instance tool description to read: "Stop and destroy a running GPU instance. REQUIRES a prior diagnose_instance() call — the API will reject the stop if you haven't diagnosed the instance in the last 10 minutes. Never stop based on speculation about instance health." This creates a double-binding constraint: the system prompt tells the agent what to do, and the tool description tells the agent what the API will enforce [6].

Edit 6 ([msg 12]): Updated the UI tools panel in ui.html, changing "Agent Tools (12 available)" to "Agent Tools (13 available)" and adding a visual card for diagnose_instance with a blue color and the description "SSH diagnostics + LLM sub-agent health verdict (required before stop)." This ensured that human operators monitoring the system through the web interface could see the new tool and understand its purpose [10][12].

The Verification Step: Closing the Loop

With all edits applied, the assistant performed a compilation verification in [msg 14], running python3 -c "import py_compile; py_compile.compile('/tmp/czk/cmd/vast-manager/agent/vast_agent.py', doraise=True)". This was not optional—the user's specification had explicitly required it. The choice of py_compile with doraise=True was the most rigorous syntax-level check available, catching everything from missing parentheses to encoding errors without executing the module [4].

In [msg 15], the assistant reported "File compiles cleanly" and provided a structured summary of all eight changes across the two files. The summary was organized by file, then by logical dependency within each file, with line numbers and positional references that made each change easy to locate and verify. This summary message closed the loop, confirming that the work was done and providing the user with a complete audit trail [5].

Architectural Decisions and Design Philosophy

Several deliberate architectural choices are visible across this implementation.

Sub-agent pattern: Rather than having the main agent reason about raw diagnostic data directly (which would consume its limited context window and risk distraction), a separate LLM call is made with a specialized prompt. This is a form of cognitive specialization: break a complex task into smaller, focused sub-tasks, each handled by a specialized agent [9].

Conservative by design: The run_instance_diagnosis function has an explicit early-return for unreachable instances that recommends "wait" rather than "destroy." The code comment states: "Don't destroy just because SSH fails — could be temporary." This encodes the lesson learned from the earlier production crash—the agent must not jump to conclusions [9].

Defense in depth: The enforcement of the diagnostic precondition operates at three levels simultaneously. The system prompt tells the agent what to do. The tool description reminds the agent of the API's requirements. And the HTTP 428 precondition physically rejects ungrounded stop requests. This multi-layered approach ensures that even if one layer fails (e.g., prompt engineering is bypassed), the others provide backup [6][14].

Graceful degradation: The JSON parsing in run_instance_diagnosis uses a three-tier strategy: structured JSON parsing, regex extraction for markdown-wrapped JSON, and raw content passthrough as a final fallback. This acknowledges that LLM outputs are not guaranteed to be well-formed and builds resilience into the system [9].

The Broader Significance

This chunk represents a philosophical shift in the agent's architecture: from a system that made decisions based on thin API signals to one that grounds every destructive action in empirical evidence collected from the target machine. The stop_instance tool now carries a precondition that forces the agent to diagnose before destroying. The diagnostic sub-agent acts as a layer of epistemic humility—a check against the agent's tendency to over-interpret ambiguous signals.

For anyone building autonomous systems with LLMs, this chunk offers valuable lessons. First, don't trust the LLM to know things it hasn't observed—give it tools to collect evidence. Second, don't rely on prompt engineering alone—build API-level enforcement for critical safety constraints. Third, specialize: use sub-agents for focused tasks rather than expecting a single agent to handle everything. Fourth, be conservative: when in doubt, wait and monitor rather than act. And fifth, be transparent: let human operators see what tools are available and how decisions are made.

The diagnostic grounding system built in this chunk is, in essence, a blueprint for building trust in an autonomous system. By forcing the agent to ground its decisions in evidence, by providing specialized diagnostic reasoning, and by enforcing safety constraints at multiple levels, the system becomes one that human operators can trust to manage expensive GPU infrastructure. And in the world of distributed proving, where a single wrong decision can cost hours of compute time and delay critical proofs, that trust is everything.