Taming the Autonomous Agent: Debugging Duplicate Runs in an LLM-Driven Fleet Manager

Introduction

On March 17, 2026, an autonomous LLM-driven fleet management agent—responsible for scaling GPU proving infrastructure across a distributed cluster of vast.ai instances—was suffering from a subtle and debilitating operational failure. The agent appeared to be running in duplicate, sometimes in parallel, flooding its own context window with redundant information, and worst of all, becoming completely unresponsive after a session reset. The system that was supposed to autonomously manage a fleet of GPU machines running cryptographic proof workloads had become its own maintenance burden.

The message under analysis—message index 5017 in the conversation—is the assistant's final summary of a deep diagnostic and repair session. It is not a message that introduces new code or deploys a fix. Rather, it is a moment of synthesis: the assistant lays out what was actually happening, what was changed, and why each change matters. It is a post-mortem delivered in real-time, and it reveals the immense complexity of building reliable autonomous agents where subtle race conditions, context pollution, and state management inconsistencies can silently degrade performance to the point of total failure.

This article examines that single message in depth: the reasoning that produced it, the assumptions it rests on, the mistakes it corrects, and the broader lessons it offers for anyone building LLM-driven autonomous systems.

The Crisis That Preceded the Summary

To understand message 5017, one must understand the crisis that provoked it. The agent had crossed 31,000 tokens of context (and in reality, 38,000 tokens on the backend), and the compaction system—designed to keep the prompt lean—had failed to run. The user then clicked the "Reset Session" button, which cleared the session state. But instead of recovering, the agent went completely silent. It stopped responding to the cron timer. It stopped responding to the manual "Trigger run now" button. The autonomous fleet manager had effectively flatlined.

This was not a simple bug. It was a cascade failure rooted in multiple interacting subsystems: context management, event triggering, state persistence, and the agent's own conversational loop. The assistant's task was to diagnose why the agent stopped responding, fix the underlying causes, and—crucially—prevent the duplicate-run problem that had been eroding trust in the system even before the reset.

Message 5017 is the capstone of that diagnostic effort. It is the assistant explaining, with clarity and precision, what was actually happening beneath the surface.

The Root Causes: What Was Actually Happening

The assistant identifies three primary causes for what the user experienced as "double/parallel" runs:

1. Stale Scheduled Wakes Accumulating Indefinitely

The agent's scheduling system allowed it to request future wake-ups—reminders to check on instance progress, re-evaluate fleet state, or follow up on pending actions. Each wake was stored as a row in a database table with a status field. But critically, once a wake was due and processed, its status was never updated to processed. This meant that every run of the agent would find the same stale wakes, log "Processing 20 scheduled wakes...", and re-process them. The wakes were never consumed. They accumulated indefinitely, like undelivered mail piling up in a dead letter office.

The assistant discovered 20 stale pending wakes that had been re-processed on every single agent run, contributing nothing but noise and wasted tokens. The fix was straightforward: after detecting and handling due wakes, mark them as processed in the database. After one triggered run under the new code, the count dropped from 20 to 13—the remaining 13 were legitimate future wakes, not stale ones.

2. Intermediate and Final Assistant Messages Both Polluting Context

The agent's conversation history stored every message the LLM produced during a run. But the LLM often produced multiple messages: an initial reasoning message (the "Agent Reasoning" block visible in the conversation), followed by tool calls, followed by a final answer message. Both messages carried the same run_id. The context management system was including both messages in the prompt, doubling the token cost of each run and filling the context window with redundant content.

The fix was to filter the conversation history so that only the final assistant message per run (identified by run_id) was included in the prompt context. The intermediate messages remain in the database for audit purposes, but they are excluded from the LLM's input.

3. Burst Duplicate Triggering from State Changes

The agent used a systemd.path unit to trigger runs immediately when important events occurred (human messages, state changes). But state changes could happen in rapid succession—multiple instances transitioning states within seconds—causing the path unit to fire repeatedly. Each trigger touched a trigger file, which the agent's timer service monitored. The result was a burst of redundant agent runs, each a few seconds apart, each processing the same state.

The assistant implemented a debounce mechanism in the Go triggerAgent() function. P0 triggers (highest priority, e.g., human messages) are debounced to at most one per 2 seconds. P1 triggers (state changes) are debounced to at most one per 10 seconds. This eliminated the bursty duplicate activations while still ensuring that legitimate events are not missed.

The Fixes: Engineering Details

Beyond the three root causes, the assistant made several additional improvements that collectively hardened the agent's operational loop.

Monotonic Run IDs via Session State

After a session reset, the agent was deriving its run_id from the conversation history—specifically, by taking the maximum existing run_id and adding one. But if the conversation was cleared (as happened during the reset), the maximum was zero, and the agent would announce "This is run #1" even though it had run hundreds of times before. This was harmless for behavior but confusing for observability. The fix was to persist a monotonic run counter in the session state database, independent of conversation history.

No-Op Runs: Visible but Not Costly

The agent sometimes runs and decides no action is needed. These "no-op" runs were previously excluded from the prompt entirely, but the user wanted them visible in the UI for audit purposes. The assistant implemented a verdict-based filtering system: runs with action=false, state_changed=false, and no tool calls are kept in the UI (with reduced opacity and a "skipped in prompt" badge) but excluded from the LLM prompt. This preserves the audit trail while keeping the context window lean.

Discouraging Redundant Schedule Requests

The agent had a schedule_next_check tool that it was using to request follow-up in ~5 minutes. But the system already had a 5-minute heartbeat timer. Every such request was redundant—it created a wake that would fire at roughly the same time as the next scheduled run. The assistant updated the prompt to tell the agent not to use schedule_next_check for routine 5-minute follow-up, and the backend now treats 240–360 second requests as no-ops.

The Thinking Process: What the Reasoning Reveals

The assistant's reasoning block in message 5017 is unusually reflective. It begins with a self-assessment: "I've reduced the system to 13 future wakes, which is good progress." This is not just a status update—it is the assistant validating its own work against the user's reported symptoms. The reasoning then moves through a checklist of what was changed and why, before arriving at a key insight: "The duplicates came from many old scheduled wakes plus the assistant pair, and that's now been fixed."

What is striking about this reasoning is the causal chain the assistant has reconstructed. The user experienced "double/parallel" runs, but the assistant correctly identified that this was not a single bug. It was the confluence of three independent failure modes:

Stale wakes created the appearance of redundant work (every run logged "Processing 20 wakes").
Duplicate assistant messages created the appearance of the LLM answering twice.
Burst triggers created actual duplicate runs, but only occasionally. Each of these had a different root cause, a different fix, and a different engineering domain (database state management, context filtering, event debouncing). The assistant's ability to decompose the user's vague symptom ("double runs") into these three distinct mechanisms is a testament to the depth of its diagnostic reasoning.

Assumptions and Knowledge Required

To understand message 5017, the reader needs significant background knowledge:

The agent architecture: The system uses a Python agent that runs on a 5-minute timer (vast-agent.timer) and an event-driven path unit (vast-agent-trigger.path) for immediate triggers. Both feed into the same agent loop.
The context management system: Conversation history is stored in SQLite, loaded at the start of each run, and filtered before being sent to the LLM. Token budget is tracked.
The wake scheduling system: The agent can request future wake-ups via schedule_next_check, which inserts rows into a database table. The pending-wakes API endpoint returns due wakes.
The verdict system: Each agent run produces a structured <verdict>{"action": bool, "state_changed": bool}</verdict> block that determines whether the run is considered a no-op.
The session state: A persistent key-value store in SQLite that survives conversation resets. The assistant makes several assumptions in its analysis: 1. That stale wakes are the primary driver of the "double run" feeling, even though they don't actually cause duplicate LLM calls. This is a correct assumption—logging "Processing 20 wakes" every run creates the perception of wasted work. 2. That debouncing is safe for event-driven triggers. This assumes that no two distinct events within the debounce window are important enough to warrant separate runs. For P1 (state changes), a 10-second window is reasonable because state transitions are not instantaneous. 3. That the 5-minute heartbeat timer is sufficient for routine checks, making schedule_next_check(300) redundant. This assumes the timer is reliable and has adequate coverage.

Mistakes and Incorrect Assumptions

While the assistant's analysis is largely sound, there are subtle points worth examining:

The assumption that stale wakes were the main cause of the "parallel" feeling may have been slightly off. The user reported that the agent was "running in duplicate" and "in parallel." Stale wakes don't cause parallel execution—they cause redundant logging. The actual parallel execution came from burst triggers. The assistant correctly identified both, but the framing in message 5017 emphasizes stale wakes first.

The debounce implementation has a subtle limitation: it uses file modification time (mtime) to determine when the last trigger occurred. If the system clock is not monotonic (e.g., if it jumps backward due to NTP correction), the debounce could fail. In practice, this is unlikely on a well-configured server, but it is an unstated assumption.

The filtering of intermediate assistant messages assumes that the final message per run is always the most useful one for the LLM to see. This is generally true, but there are edge cases: if the final message is a short acknowledgment ("Done.") and the intermediate reasoning contains important context, the LLM loses that information. The assistant does not address this trade-off.

Output Knowledge Created

Message 5017 creates several forms of knowledge:

A verified causal model of the duplicate-run problem. The assistant has confirmed that stale wakes, duplicate messages, and burst triggers were the three mechanisms at play.
A set of engineering patterns for preventing similar issues: mark processed items as processed, debounce event handlers, filter redundant context, persist state outside of conversation history.
An operational baseline: 13 future wakes (down from 20) is now the expected steady state. Any increase would indicate new scheduling requests, not accumulation.
A proposed UI improvement: the assistant suggests adding a "Wakes" section to the UI with pending/due/future views and cancel buttons, which would give operators direct visibility into the scheduling subsystem.

Broader Implications

The problems diagnosed in message 5017 are not unique to this particular agent. They are endemic to autonomous LLM-driven systems:

Context pollution is the silent killer of agent reliability. Every redundant message, every stale wake, every verbose tool output eats into the token budget and degrades decision quality.
Event-driven architectures with LLM agents are prone to thrashing. Without debouncing, state changes trigger cascades of redundant runs.
State management across resets is fragile. If run IDs, session state, or conversation history are derived from each other, a reset in one domain breaks all dependent domains.
Observability vs. context efficiency is a fundamental tension. The user wants to see everything; the LLM needs to see only what matters. The verdict-based filtering system is a clever resolution, but it adds complexity. The assistant's fixes—marking wakes as processed, debouncing triggers, filtering duplicate messages, persisting monotonic run IDs—are not glamorous. They are plumbing. But they are the kind of plumbing that determines whether an autonomous system runs for weeks without intervention or collapses under its own weight within hours.

Conclusion

Message 5017 is a masterclass in diagnosing and repairing subtle failures in autonomous LLM-driven systems. It demonstrates that the hardest problems in agent reliability are not about model capability—they are about state management, event handling, and context hygiene. The assistant's ability to decompose a vague symptom ("double runs") into three distinct root causes, each with a clean fix, is a model for how to build robust autonomous agents.

The message also reveals something important about the assistant's own reasoning: it is not just fixing bugs, but building a theory of the system's behavior. The reasoning block shows the assistant testing its own understanding, validating assumptions against observed data, and proposing future improvements. This is not rote debugging. It is engineering judgment applied to a complex, distributed, LLM-driven system—and it works.