Chunk 32.4

This chunk focused on three major areas: building a diagnostic grounding system to prevent the agent from making destructive decisions based on speculation, overhauling context management to handle duplicates and no-op runs cleanly, and debugging a critical failure where a session reset completely broke the agent loop. **Diagnostic Grounding & Agent Behavior Correction** The assistant initially tried a hard time-based guard to prevent the agent from killing young instances, but the user correctly redirected the approach—the agent must be allowed to make any decision, but only after grounding itself in facts. This led to the creation of a **Diagnostic Sub-Agent system**: a Go endpoint that SSHes into instances to collect raw logs, processes, and system state (with a fallback to vast API data when SSH is broken), paired with a Python sub-agent LLM that interprets the data with domain knowledge about normal startup sequences versus actual failures. The `stop_instance` tool was gated with an HTTP 428 precondition, forcing the main agent to call `diagnose_instance` first, shifting the architecture from brittle hard-coded rules to an evidence-driven, LLM-powered diagnostic layer. **Context Management, Verdict System, and Duplicate Suppression** The user reported duplicate parallel agent runs (timer + path unit collisions), idle observations polluting the conversation, and a lack of structured feedback. The assistant implemented a **file lock** to prevent parallel invocations, added a structured `<verdict>{"action": bool, "state_changed": bool}</verdict>` block to the LLM's output, and used it to prune no-action runs from the prompt context while keeping them visible in the UI (grayed out with a "skipped in prompt" badge). Duplicate intermediate assistant messages (pre-tool narration vs. final answer) were suppressed in both the prompt and UI, keeping only the final response per run. The `schedule_next_check` tool was made a no-op for its default 240–360s range since the 5-minute heartbeat already covers that window. **Context Overflow Crisis, Reset Bug, and Event Debouncing** The chunk concluded with a severe operational incident: the agent's context ballooned to 38k tokens because compaction failed, and hitting the "Reset Session" button broke the agent entirely—it stopped responding to both the cron timer and manual triggers. The assistant diagnosed the root causes: 20 stale "pending wakes" were accumulating without being marked processed, causing the trigger file to be touched repeatedly and spawning bursts of redundant event-driven runs. The `run_id` was being derived from conversation history, which broke after a reset. Fixes included a `POST /api/agent/mark-wakes` endpoint to clear processed wakes, a **debounce mechanism** in the Go `triggerAgent()` function to suppress burst file modifications, and making the `run_id` monotonic via persistent session state instead of fragile conversation history. This stabilized the agent loop, restoring normal operation after resets and eliminating the re-trigger churn. The themes highlight the immense complexity of building reliable autonomous agents, where context management, event handling, and diagnostic grounding must be meticulously engineered to prevent catastrophic decisions based on incomplete information or system race conditions.

Grounding, Verdict, and Recovery: Engineering Reliability into an Autonomous LLM Fleet Agent 2680 words

Message Articles

Subagent Sessions