Closing the Loop: The Observation Update That Completed an Autonomous Agent's Rescue

In the aftermath of a catastrophic production failure where an autonomous GPU cluster manager systematically stopped every running instance despite 59 pending tasks, the assistant was methodically working through a multi-layer fix. The subject message — a deceptively simple grep-and-edit step — represents the final link in a chain of corrections that would determine whether the agent could actually see the new safety signals being built for it. The message reads:

Now update build_observation to include the new fields: [grep] def build_observation Found 1 matches /tmp/czk/cmd/vast-manager/agent/vast_agent.py: Line 225: def build_observation(run_id: int, demand: dict, fleet: dict, config: dict) -> str:

This is not a decision-making message. It is an execution message — a statement of intent followed by a tool invocation to locate the exact code that needs changing. But to understand why this seemingly trivial step was critical, we must first understand the disaster that precipitated it.

The Disaster: When "No Demand" Means "All Workers Are Dead"

The autonomous agent had been running successfully for hours, scaling instances up and down based on Curio SNARK demand. Then, in a single observation cycle, it saw active=False and 0 throughput over the last 15 minutes. Following its programming, it concluded that demand had evaporated and began killing instances to save money. It stopped machine after machine — stop_instance({"vast_id":33008932,"reason":"Demand inactive..."}) — each call returning ok: true. Within minutes, the fleet was decimated.

But demand had not evaporated. There were 59 PSProve tasks queued and waiting. The workers were simply dead — all vast instances showed exited status on the vast.ai platform, likely due to GPU driver faults or external termination. The agent's active flag was computed purely from recent completion counts: if zero proofs completed in the last 15 minutes, active was False. This single boolean could not distinguish between "no one wants work" and "all workers have died and tasks are piling up." The agent, lacking this distinction, made the worst possible decision: it killed the remaining instances, ensuring that even if the crashed ones recovered, there would be nothing to recover.

The Multi-Layer Fix

The assistant's diagnosis was precise: the demand signal needed a richer vocabulary. The fix would span four layers, each requiring modification:

  1. The Go API endpoint (agent_api.go): Add demand_queued (boolean indicating any pending tasks exist) and workers_dead (boolean indicating pending tasks with zero alive workers) to the DemandResponse struct. These fields give the agent semantic understanding: "there are tasks waiting but no one to do them."
  2. The fast-path logic (vast_agent.py, no_action_needed): The function that decides whether to skip the LLM call entirely must never return "no action needed" when workers_dead is true or demand_queued is true. The agent must always engage its reasoning when there's a potential emergency.
  3. The system prompt (SYSTEM_PROMPT_TEMPLATE): The LLM's behavioral rules must include explicit guidance about the new signals: "If workers_dead is true, this is an emergency — never scale down, prioritize restoring capacity."
  4. The observation string (build_observation): The compact text that the LLM sees each run must include the new fields, otherwise the LLM has no way to act on them. By the time the subject message was written, layers 1, 2, and 3 had already been edited and deployed. The assistant had added the Go fields, hardened the fast-path, and updated the prompt. Only one piece remained: ensuring the observation string — the actual text the LLM reads to understand the current state — included demand_queued and workers_dead.

Why This Message Matters

The build_observation function is the agent's sensory interface. It constructs a compact (~150 token) string that summarizes demand, fleet state, throughput, and configuration for the LLM each cycle. If the new fields were not added to this string, the LLM would never see them. The system prompt could say "if workers_dead is true, never scale down" until the heat death of the universe, but the LLM would never encounter the condition because the observation wouldn't report it.

This is a classic systems-design pitfall: adding a signal to the data layer and the policy layer but forgetting the presentation layer. The assistant's methodical approach — working through every consumer of the demand signal — ensured no gap remained. The grep for def build_observation was the verification step: "Does this function exist? Where is it? Good, now I can edit it."

The message also reveals the assistant's working style. It doesn't assume the function is where it expects it to be. It greps first, confirming the exact location and signature before proceeding. The function signature — (run_id: int, demand: dict, fleet: dict, config: dict) -> str — confirms that demand is already passed as a parameter, so adding demand.get("demand_queued", False) and demand.get("workers_dead", False) to the string construction is straightforward.

Input Knowledge Required

To fully understand this message, one must grasp several layers of context:

Output Knowledge Created

This message creates several forms of knowledge:

The Thinking Process

The assistant's reasoning, visible across the sequence of messages, follows a clear pattern:

  1. Observe failure: Agent killed instances despite pending tasks.
  2. Diagnose root cause: active=False is ambiguous — can't distinguish "no demand" from "workers dead."
  3. Design solution: Add demand_queued and workers_dead signals.
  4. Identify all consumers: Go API → fast-path → system prompt → observation string.
  5. Fix each consumer in order: API first (data source), then fast-path (gate), then prompt (policy), then observation (presentation). The subject message is step 5 in action. The assistant has already fixed the data, the gate, and the policy. Now it's fixing the presentation layer. The grep is a deliberate, low-ceremony verification: "Let me confirm the function exists before I try to edit it." This methodical approach — trace the signal through every layer, fix each one, verify before editing — is what separates a robust fix from a fragile one. The assistant could have stopped after updating the API and the fast-path, assuming the LLM would figure things out. But it didn't. It ensured the LLM would actually see the new signals in its observation, closing the loop completely.

Conclusion

The subject message, for all its brevity, captures a critical moment in the evolution of an autonomous system. It is the moment when a fix transitions from "good enough" to "complete." The assistant had already addressed the data layer, the decision gate, and the behavioral rules. But without updating the observation string — the LLM's window into the world — those fixes would have been invisible to the agent's reasoning core. By methodically working through every consumer of the demand signal and ensuring each one was updated, the assistant closed the loop and ensured the agent would never again mistake "all workers are dead" for "no one wants work."