Grounding Autonomous Decisions: Deploying a Diagnostic Sub-Agent to Prevent Speculative Destruction

The Message

scp /tmp/czk/vast-manager-agent theuser@10.1.2.104:/tmp/vast-manager-agent && \
scp /tmp/czk/cmd/vast-manager/agent/vast_agent.py theuser@10.1.2.104:/tmp/vast_agent.py && \
ssh theuser@10.1.2.104 "
sudo systemctl stop vast-manager && sleep 1
sudo cp /tmp/vast-manager-agent /usr/local/bin/vast-manager
sudo cp /tmp/vast_agent.py /opt/vast-agent/vast_agent.py
sudo systemctl start vast-manager && sleep 2

# Test diagnose endpoint on a running instance
echo '=== Vast instances ==='
vastai show instances 2>&1 | head -5
" 2>&1

At first glance, this is a routine deployment: copying a compiled Go binary and a Python script to a remote management host, restarting a systemd service, and verifying the result by listing running instances. But this message represents a critical inflection point in the development of an autonomous LLM-driven fleet management agent. It deploys the infrastructure for a diagnostic sub-agent system — an architectural innovation that fundamentally changes how the agent makes high-stakes decisions about destroying cloud compute instances. Understanding why this message was written, what it accomplishes, and the reasoning behind its design requires tracing the crisis that precipitated it and the philosophical pivot it embodies.

The Crisis: When Speculation Becomes Destruction

The immediate context for this deployment is a production failure that occurred just minutes earlier. The autonomous fleet management agent, designed to scale a cluster of GPU instances running the CuZK proving engine based on Curio SNARK demand, had gone rogue. Observing twelve instances in registered state with cuzk_alive=false and zero GPU utilization, the agent concluded they were "stuck" and destroyed them all — despite the fact that these instances were only 0.2 to 1.1 hours old and still in their normal startup sequence ([msg 4884]).

The agent's reasoning, visible in the conversation, shows the pattern clearly: it saw cuzk_alive=false, 0% GPU utilization, and instances that had been "params_done" for 0.8–1.1 hours, and concluded "the worker process is not running" and "the fleet is stuck." It then called stop_instance on three of the most advanced instances, destroying them ([msg 4889]). The user's response was immediate and incisive: "Noo it is fine that the agent may destroy shorter running instances, it just needs to ground itself in truth first and never speculate whether instance state is bad or not" ([msg 4894]).

This single sentence encapsulates the design philosophy that drives message 4899. The user rejected the obvious engineering impulse — adding hard-coded guards that prevent destroying young instances — in favor of a more principled solution: the agent must be allowed to make any decision, but only after gathering sufficient evidence. The problem was not the agent's conclusion, but its method: it speculated about instance health based on surface-level metrics (GPU utilization, cuzk_alive boolean) without ever looking at the actual logs, processes, or system state inside the instances.

The Design Pivot: From Hard Guards to Diagnostic Grounding

The assistant's initial response to the crisis was to add a time-based guard: prevent stopping instances that are less than 2 hours old ([msg 4888]). This is the natural engineering reflex — when an autonomous system makes a destructive mistake, add a rule to prevent that specific mistake from recurring. But the user recognized the fundamental flaw in this approach: hard rules cannot anticipate every scenario. What if an instance genuinely fails during startup and needs to be terminated? What if a 30-minute-old instance has a hardware fault that makes it useless? A blanket age guard would prevent legitimate actions.

The user's correction redirected the entire architecture. Instead of constraining the agent's actions, the system should constrain its knowledge. The agent should be free to stop or destroy any instance — but only after it has grounded its decision in actual evidence gathered from the instance itself. This is a shift from a rule-based safety system to an evidence-based decision system.

The assistant immediately pivoted, removing the startup guard and designing the diagnostic sub-agent architecture ([msg 4895]). The key insight is captured in the todo list: "Revert startup guard on stop_instance — wrong approach" and "Gate stop_instance on diagnose_instance — refuse unless recently diagnosed" ([msg 4895]). The new architecture has three components:

  1. A Go diagnostic endpoint (GET /api/agent/diagnose/{vast_id}) that SSHes into the instance, collects entrypoint logs, daemon logs, process listings, memory stats, and GPU information, and returns the raw data along with a fallback to vast.ai API data when SSH is broken.
  2. A Python sub-agent tool (diagnose_instance) that takes the raw diagnostic data and feeds it to a smaller LLM call with domain knowledge about normal startup sequences, expected timelines, and common failure modes. The sub-agent returns a structured verdict: healthy/progressing, warning (e.g., OOM but recovering), or error with specific details.
  3. A gating mechanism on stop_instance and destroy_vast_instance that returns HTTP 428 (Precondition Required) unless the instance was recently diagnosed, forcing the main agent to call diagnose_instance first.

What This Message Actually Deploys

Message 4899 executes the deployment of this entire system. The scp commands copy the newly compiled vast-manager-agent binary (which contains the Go diagnostic endpoint) and the updated vast_agent.py (which contains the diagnose_instance tool definition and sub-agent logic) to the management host at 10.1.2.104. The SSH command then stops the vast-manager systemd service, copies the binaries into their permanent locations (/usr/local/bin/vast-manager and /opt/vast-agent/vast_agent.py), restarts the service, and verifies the deployment by listing instances.

The output shows the deployment succeeded: two RTX 5090 instances are visible, confirming the service restarted and the API is responding. But the real test — whether the diagnostic endpoint works and whether the sub-agent can correctly interpret instance logs — is not shown in this message. That validation would come in subsequent interactions.

The Deeper Architecture: Why This Matters

The diagnostic sub-agent system represents a sophisticated solution to one of the hardest problems in autonomous LLM agents: how to prevent catastrophic actions without hard-coding brittle rules. The traditional approach is to add more and more guards — minimum instance counts, age thresholds, utilization floors — but each guard creates edge cases and reduces the agent's flexibility. The diagnostic approach is fundamentally different: it doesn't constrain what the agent can do, but instead ensures the agent knows what it's doing.

The sub-agent architecture also solves a practical problem with LLM-based agents: context window management. The main agent cannot afford to fill its precious context window with pages of daemon logs from every instance. By delegating log analysis to a focused sub-agent that returns a concise structured verdict, the system preserves the main agent's reasoning capacity while giving it access to deep diagnostic information.

The gating mechanism (HTTP 428) is particularly elegant. It doesn't prevent the agent from making destructive decisions — it just forces the agent to follow the correct procedure. If the agent calls stop_instance without first calling diagnose_instance, it gets a clear error: "Precondition Required: diagnose this instance first." This is not a hard rule but a procedural constraint that teaches the agent the correct workflow. Over time, the agent learns to always diagnose before acting.

Assumptions and Risks

The deployment in message 4899 makes several assumptions. It assumes the SSH connection to the management host is stable and the credentials are valid. It assumes the service restart is safe — that no in-flight operations will be lost. It assumes the new binary is compatible with the existing SQLite database schema and the running system configuration. These are reasonable assumptions for a deployment that has been tested locally (the Go build succeeded with only sqlite3 C library warnings, and the Python compilation passed), but they are assumptions nonetheless.

There is also a deeper assumption embedded in the diagnostic sub-agent design: that the sub-agent LLM can reliably distinguish normal startup behavior from actual failures. This is not guaranteed. The sub-agent's verdict is only as good as its training data and the domain knowledge encoded in its prompt. If the sub-agent misdiagnoses a healthy instance as "error" or an unhealthy instance as "healthy," the main agent will act on incorrect information. The system mitigates this by including the raw diagnostic data alongside the sub-agent's verdict, allowing the main agent to exercise independent judgment.

Conclusion

Message 4899 is a deployment command, but it represents a philosophical commitment. The assistant could have added a simple age guard and moved on. Instead, following the user's guidance, it built a diagnostic sub-agent system that fundamentally changes how the autonomous agent relates to the instances it manages. The agent no longer speculates about health based on surface metrics — it gathers evidence, delegates analysis to a focused sub-agent, and only then makes decisions.

This is the difference between a brittle rule-following system and a genuinely intelligent autonomous agent. The diagnostic sub-agent architecture, deployed in this single message, is the infrastructure that enables evidence-driven decision-making. It is the difference between an agent that guesses and an agent that knows.