Hardening the Autonomous Agent: Eliminating Duplicate Runs, Context Pollution, and Event Thrashing

Introduction

Building a reliable autonomous LLM-driven agent is hard. Building one that manages a fleet of GPU machines running cryptographic proof workloads—where every misstep costs real money in compute credits and lost proving capacity—is exponentially harder. In the final stretch of a marathon coding session that spanned deploying budget-integrated pinned pools, fixing a critical wait -n supervisor crash bug, and constructing a fully autonomous fleet management agent from scratch, the assistant encountered a class of failure that threatens every autonomous system: the silent, gradual degradation caused by duplicate work, context pollution, and event thrashing.

The message under analysis—message index 5017 in the conversation—is the assistant's synthesis of a deep diagnostic and repair effort. It is not a message that introduces new code or deploys a feature. It is a post-mortem delivered in real-time: the assistant laying out what was actually happening beneath the surface, what was changed, and why each change matters. This article examines that message in depth, tracing the causal chain from vague user reports of "double/parallel runs" to a precise set of engineering fixes that transformed a fragile, thrashing agent into a stable, debounced, and context-efficient operational loop.

The Crisis: An Agent That Stopped Responding

To understand message 5017, one must understand the crisis that preceded it. The autonomous fleet management agent—built to scale GPU instances on vast.ai based on Curio SNARK demand—had been running for several days. It had survived multiple production incidents: over-provisioning instances, accidentally stopping all running machines due to a flawed demand signal, and suffering context overflow that ballooned to 38,000 tokens. But the most alarming failure came when the user clicked the "Reset Session" button to clear the agent's conversation history. Instead of recovering gracefully, the agent went completely silent. It stopped responding to the 5-minute cron timer. It stopped responding to the manual "Trigger run now" button. The autonomous fleet manager had flatlined.

The assistant's task was to diagnose why the agent stopped responding, fix the underlying causes, and—crucially—prevent the duplicate-run problem that had been eroding trust in the system even before the reset. Message 5017 is the capstone of that effort: the assistant explaining, with clarity and precision, the three distinct failure modes that had converged to create the user's experience of a broken agent.

Root Cause 1: Stale Scheduled Wakes Accumulating Indefinitely

The agent's scheduling system allowed it to request future wake-ups—reminders to check on instance progress, re-evaluate fleet state, or follow up on pending actions. Each wake was stored as a row in a database table with a status field. But critically, once a wake was due and processed, its status was never updated to processed. This meant that every run of the agent would find the same stale wakes, log "Processing 20 scheduled wakes...", and re-process them. The wakes were never consumed. They accumulated indefinitely, like undelivered mail piling up in a dead letter office.

The assistant discovered 20 stale pending wakes that had been re-processed on every single agent run, contributing nothing but noise and wasted tokens. The fix was straightforward but transformative: after detecting and handling due wakes, mark them as processed in the database. After one triggered run under the new code, the count dropped from 20 to 13—the remaining 13 were legitimate future wakes, not stale ones. This single change immediately eliminated the most visible symptom of the "duplicate run" problem: the agent no longer appeared to be doing redundant work on every cycle.

This fix also addressed a subtler issue: the stale wakes were touching the trigger file every time they were processed, which in turn triggered the systemd.path unit, causing additional redundant agent activations. By consuming wakes properly, the assistant broke a feedback loop that had been silently amplifying the agent's workload.

Root Cause 2: Intermediate and Final Assistant Messages Both Polluting Context

The agent's conversation history stored every message the LLM produced during a run. But the LLM often produced multiple messages: an initial reasoning message (the "Agent Reasoning" block visible in the conversation), followed by tool calls, followed by a final answer message. Both messages carried the same run_id. The context management system was including both messages in the prompt, doubling the token cost of each run and filling the context window with redundant content.

This was a classic context pollution problem. The LLM was being asked to reason about a conversation where it could see its own intermediate thoughts alongside its final conclusions. This not only wasted tokens—it potentially confused the model by presenting multiple versions of the same decision-making process.

The fix was to filter the conversation history so that only the final assistant message per run (identified by run_id) was included in the prompt context. The intermediate messages remain in the database for audit purposes, but they are excluded from the LLM's input. The UI was also updated to hide earlier assistant messages from the same run, presenting a cleaner view to the operator.

Root Cause 3: Burst Duplicate Triggering from State Changes

The agent used a systemd.path unit to trigger runs immediately when important events occurred—human messages, state changes on instances, or other P0/P1 events. But state changes could happen in rapid succession: multiple instances transitioning states within seconds, causing the path unit to fire repeatedly. Each trigger touched a trigger file, which the agent's timer service monitored. The result was a burst of redundant agent runs, each a few seconds apart, each processing the same state.

The assistant implemented a debounce mechanism in the Go triggerAgent() function. P0 triggers (highest priority, e.g., human messages) are debounced to at most one per 2 seconds. P1 triggers (state changes) are debounced to at most one per 10 seconds. This eliminated the bursty duplicate activations while still ensuring that legitimate events are not missed. The debounce uses file modification time (mtime) to determine when the last trigger occurred, providing a simple and reliable guard against thrashing.

Additional Hardening: The Full Set of Fixes

Beyond the three root causes, the assistant made several additional improvements that collectively hardened the agent's operational loop.

Monotonic Run IDs via Session State

After a session reset, the agent was deriving its run_id from the conversation history—specifically, by taking the maximum existing run_id and adding one. But if the conversation was cleared (as happened during the reset), the maximum was zero, and the agent would announce "This is run #1" even though it had run hundreds of times before. This was harmless for behavior but confusing for observability. The fix was to persist a monotonic run counter in the session state database, independent of conversation history. Now, even after a reset, the agent correctly continues the sequence from where it left off.

No-Op Runs: Visible but Not Costly

The agent sometimes runs and decides no action is needed. These "no-op" runs were previously excluded from the prompt entirely, but the user wanted them visible in the UI for audit purposes. The assistant implemented a verdict-based filtering system: runs with action=false, state_changed=false, and no tool calls are kept in the UI (with reduced opacity and a "skipped in prompt" badge) but excluded from the LLM prompt. This preserves the audit trail while keeping the context window lean.

Discouraging Redundant Schedule Requests

The agent had a schedule_next_check tool that it was using to request follow-up in approximately 5 minutes. But the system already had a 5-minute heartbeat timer via vast-agent.timer. Every such request was redundant—it created a wake that would fire at roughly the same time as the next scheduled run. The assistant updated the prompt to tell the agent not to use schedule_next_check for routine 5-minute follow-up, and the backend now treats 240–360 second requests as no-ops. This prevents the wake table from being cluttered with redundant entries that would otherwise need to be processed and marked as consumed.

The Reasoning Process: Decomposing a Vague Symptom

What is most striking about message 5017 is the assistant's ability to decompose the user's vague symptom—"double/parallel runs"—into three distinct mechanisms, each with a different root cause, a different fix, and a different engineering domain:

1. Stale wakes (database state management): The agent appeared to be doing redundant work because it logged "Processing 20 wakes" on every run, even though those wakes had already been handled.
2. Duplicate assistant messages (context filtering): The LLM appeared to be answering twice because both its intermediate reasoning and its final answer were included in the prompt.
3. Burst triggers (event debouncing): The agent was sometimes actually running in parallel because state-change events fired in rapid succession, each touching the trigger file. The assistant's reasoning block in message 5017 reflects this decomposition. It begins with a self-assessment: "I've reduced the system to 13 future wakes, which is good progress." It then walks through each root cause and its corresponding fix, before arriving at the key insight: "The duplicates came from many old scheduled wakes plus the assistant pair, and that's now been fixed." This is not rote debugging. It is engineering judgment applied to a complex, distributed, LLM-driven system—identifying which symptoms are causes and which are effects, tracing feedback loops, and implementing targeted fixes that address the underlying mechanisms rather than their surface manifestations.

Broader Implications for Autonomous Agent Design

The problems diagnosed in message 5017 are not unique to this particular agent. They are endemic to autonomous LLM-driven systems, and the patterns the assistant established are broadly applicable:

Context pollution is the silent killer of agent reliability. Every redundant message, every stale wake, every verbose tool output eats into the token budget and degrades decision quality. The assistant's verdict-based filtering and intermediate-message suppression are essential patterns for any agent that maintains a persistent conversation history.

Event-driven architectures with LLM agents are prone to thrashing. Without debouncing, state changes trigger cascades of redundant runs. The assistant's priority-based debounce (2s for P0, 10s for P1) is a simple but effective pattern that should be standard in any agent that responds to external events.

State management across resets is fragile. If run IDs, session state, or conversation history are derived from each other, a reset in one domain breaks all dependent domains. The assistant's decision to persist monotonic run IDs in session state—independent of conversation history—is a pattern that prevents identity confusion after any kind of state reset.

Observability vs. context efficiency is a fundamental tension. Operators want to see everything; the LLM needs to see only what matters. The verdict-based filtering system—keeping no-op runs in the UI while excluding them from the prompt—is a clever resolution, but it adds complexity. The proposed addition of a "Wakes" section in the UI represents a continued commitment to transparency and operator control over the agent's internal scheduling.

Conclusion

Message 5017 is a masterclass in diagnosing and repairing subtle failures in autonomous LLM-driven systems. It demonstrates that the hardest problems in agent reliability are not about model capability—they are about state management, event handling, and context hygiene. The assistant's ability to decompose a vague symptom into three distinct root causes, each with a clean fix, is a model for how to build robust autonomous agents.

The fixes themselves—marking wakes as processed, debouncing triggers, filtering duplicate messages, persisting monotonic run IDs—are not glamorous. They are plumbing. But they are the kind of plumbing that determines whether an autonomous system runs for weeks without intervention or collapses under its own weight within hours. By moving from a fragile system prone to duplicate work and context overflow to a debounced, filtered, and stateful architecture, the assistant significantly hardened the agent's operational loop, establishing a robust foundation for autonomous, cost-effective cluster management.

The proposed "Wakes" UI section—with pending, due, and future views and cancel buttons—represents a continued commitment to transparency. In an autonomous system, operators need visibility into every subsystem that can affect behavior. The agent's scheduling system, once invisible, would become directly observable and controllable. This is the final lesson of message 5017: in autonomous systems, trust is earned through transparency, and reliability is built one fix at a time.