The 428 That Changed Everything: Building Evidence-Driven Guardrails for an Autonomous Fleet Agent

A Single HTTP Status Code That Proved an Architecture

In the sprawling, multi-chunk saga of building an autonomous LLM-driven fleet management agent for a distributed GPU proving infrastructure, most breakthroughs arrive quietly—a clean compilation, a successful SSH handshake, a Docker image that builds without errors. But occasionally, a single HTTP status code speaks louder than any log line. Message [msg 4928] is that rare moment: a message whose entire substance is the triumphant observation that a 428 Precondition Required response successfully blocked a destructive action. The message itself is deceptively brief:

[assistant] 428 Precondition Required — the grounding check blocked the stop. The error message tells the agent to call diagnose_instance first. [todowrite] {"todos":[{"content":"Revert startup guard on stop_instance — wrong approach","priority":"high","status":"completed"},{"content":"Build GET /api/agent/diagnose/{vast_id} — SSH collect logs/ps/memory from instance","priority":"high","status":"completed"},{"content":"Build diagnose_instance Python tool — sub-agent LLM reads logs, returns structured verdict","priority":"high","status":"completed"},{"content":"Gate stop_in...

This is not merely a status update. It is the culmination of a fundamental architectural pivot—a pivot from brittle, hard-coded rules to an evidence-driven, LLM-powered diagnostic layer—and the message's real weight lies in what it proves about that pivot.

The Problem That Demanded a New Architecture

To understand why this 428 response matters, one must understand the failure that preceded it. The autonomous agent, designed to manage a fleet of vast.ai GPU instances running the CuZK proving engine, had suffered a critical production failure: it misinterpreted active=False and stopped all running instances despite 59 pending tasks queued in Curio. The demand signal could not distinguish between "no demand" and "all workers dead with tasks queued." This was a catastrophic failure of reasoning, not a bug in code.

The assistant's initial response was to build a hard time-based guard—a rule preventing the agent from killing instances that had been running for less than a certain duration. This was the instinctive engineering response: add a check, prevent the bad outcome. But the user redirected this approach with surgical precision in [msg 4894]:

"Noo it is fine that the agent may destroy shorter running instances, it just needs to ground itself in truth first and never speculate whether instance state is bad or not. Maybe make it possible to call Stop only after the agent has queried the 'state discovery agent' and grounded itself that the instance is indeed in a bad state."

This is a profound design insight. The user recognized that the problem was not the action (stopping instances) but the epistemology (making decisions without evidence). The agent could be permitted any action, as long as it first grounded itself in observable facts. This shifted the entire architecture from a permission-based model ("you may not do X") to an evidence-based model ("you may do anything, but only after you have verified the facts").

Building the Diagnostic Grounding System

The assistant executed this vision across several rounds of development. First, a Go endpoint was built: GET /api/agent/diagnose/{vast_id}. This endpoint SSHes into a vast.ai instance and collects raw diagnostic data—process lists, entrypoint logs, daemon logs, memory usage, GPU state. When SSH is unreachable (a common problem on vast.ai, where some instances have broken SSH setups), it falls back to vast API data and the manager's internal database state, providing a rich fallback payload rather than an empty error.

Second, a Python sub-agent tool was built: diagnose_instance. This tool calls the Go endpoint, then feeds the raw diagnostic data to a secondary LLM—a "diagnostic sub-agent"—that interprets the data with domain knowledge about normal startup sequences versus actual failures. The sub-agent returns a structured verdict: is the instance healthy, starting up, or genuinely broken?

Third, and most critically, the stop_instance tool was gated. The Go handler for stopping instances was modified to check for a recent diagnose action in the agent's action log. If no diagnosis exists for that instance within a configurable time window, the endpoint returns HTTP 428 Precondition Required—a status code that tells the calling agent "you must satisfy a precondition before this request can be processed." The error message explicitly instructs the agent to call diagnose_instance first.

What Message 4928 Actually Proves

The message at [msg 4928] is the moment this entire architecture is validated. The assistant had just executed a test: it attempted to call stop_instance on instance 33037353 without having diagnosed it first. The response was HTTP 428. The grounding check worked.

The message's brevity belies its significance. The assistant writes "428 Precondition Required — the grounding check blocked the stop. The error message tells the agent to call diagnose_instance first." This is a statement of verification, not of implementation. The system is no longer a set of hopes and code; it is a running, tested reality. The todowrite that follows is not a plan but a retrospective—every item is marked status: "completed", tracing the full arc from the reverted startup guard through the built endpoints to the gated tool.

The Todowrite as a Design Document

The todowrite embedded in this message is worth close examination. It lists four items:

1. Revert startup guard on stop_instance — wrong approach: The assistant explicitly acknowledges that the initial hard-coded guard was the wrong approach. This is intellectual honesty—the willingness to discard one's own work when a better design emerges.
2. Build GET /api/agent/diagnose/{vast_id} — SSH collect logs/ps/memory from instance: The data collection layer. This is the raw evidence-gathering mechanism.
3. Build diagnose_instance Python tool — sub-agent LLM reads logs, returns structured verdict: The interpretation layer. Raw data is meaningless without interpretation; the sub-agent LLM provides the domain-aware analysis.
4. Gate stop_instance with HTTP 428 precondition: The enforcement layer. The tool itself refuses to act without prior diagnosis. This four-layer architecture—acknowledgment of wrong approach, data collection, interpretation, enforcement—represents a mature design pattern for autonomous systems. The agent is not prevented from making decisions; it is required to make informed decisions.

Assumptions and Their Validation

The system makes several assumptions, and message [msg 4928] validates one of the most critical: that the HTTP 428 precondition mechanism is enforceable at the API level. The assistant had earlier tested the stop endpoint on an instance that had been diagnosed (via a curl test), and it passed through successfully. The test at [msg 4927] targeted an instance without a recent diagnosis, and the 428 response confirmed that the gating logic correctly distinguishes between the two cases.

Another assumption, validated in earlier messages, is that the diagnostic data is useful even when SSH fails. The fallback mechanism—using vast API data and the manager's internal database—was tested at [msg 4920], showing that an SSH-unreachable instance still returns meaningful state information (db_state: params_done, db_registered_at: 2026-03-17T19:22:39Z). The note appended to the response is particularly thoughtful: "Some instances have broken SSH setup — this does NOT mean the instance is unhealthy." This prevents the sub-agent from misinterpreting an SSH failure as evidence of a broken instance.

The Deeper Significance: From Speculation to Evidence

The most profound shift represented by this message is epistemological. The earlier agent failure—stopping all instances despite pending tasks—was a failure of reasoning based on incomplete signals. The agent saw active=False and inferred "no demand," when the reality was "all workers are dead and tasks are piling up." This is the classic autonomous system failure mode: acting on proxy signals without understanding their limitations.

The diagnostic grounding system forces the agent to move from speculation to evidence. Before it can stop an instance, it must have recent, concrete diagnostic data about that instance's state. The sub-agent LLM interprets that data with domain knowledge, producing a structured verdict. The main agent can then make its decision based on that verdict rather than on vague signals like active=False.

This is not merely a technical improvement; it is a philosophical one. The system is being designed to privilege evidence over inference, to require grounding before action. The HTTP 428 status code is the enforcement mechanism for this philosophy—a technical artifact that encodes a design principle.

Output Knowledge and the Road Ahead

Message [msg 4928] creates several forms of output knowledge. First, it provides concrete verification that the grounding architecture works as designed—the 428 response is empirical proof. Second, the todowrite serves as a compact design document, tracing the full arc of the diagnostic grounding system from conception to completion. Third, it establishes a pattern that can be extended to other tools: if stop_instance requires diagnosis, perhaps launch_instance should require demand verification, and perhaps destroy_instance should require a more stringent multi-diagnosis consensus.

The message also implicitly defines the contract between the main agent and the diagnostic sub-agent. The main agent must call diagnose_instance before stop_instance. The diagnostic sub-agent must return a structured verdict. The Go backend must enforce the precondition. This three-party contract—agent, sub-agent, API—is the backbone of the entire grounding system.

Conclusion

Message [msg 4928] appears, at first glance, to be a trivial status update: a test passed, a status code was received. But it is far more. It is the moment a complex, multi-layered architectural vision was validated in production. It represents the successful transition from a rule-based guard system to an evidence-driven, LLM-powered diagnostic layer. It proves that the agent can be trusted to make destructive decisions—as long as it grounds itself in facts first. And it demonstrates that sometimes the most important engineering insight is not about what to prevent, but about what to require before action is taken.

The 428 Precondition Required status code, often overlooked as a niche HTTP feature, becomes in this context a powerful architectural tool: a way to encode design philosophy directly into the API contract. The message that celebrates its successful deployment is a milestone in the journey from brittle automation to robust, evidence-driven autonomy.