The Double-Response Bug: Diagnosing and Eliminating Duplicate Agent Messages in an LLM-Powered Fleet Manager

Introduction

In the course of building a fully autonomous LLM-driven fleet management agent for a distributed GPU proving infrastructure, a subtle but deeply frustrating bug emerged: the agent appeared to produce "double responses" to user messages and system events. The operator would see two nearly identical replies from the agent, creating confusion about which response was authoritative and whether the agent was making duplicate decisions. This article examines a single message ([msg 4973]) in which the assistant summarizes its investigation and resolution of this double-response problem. The message is a concise post-mortem delivered after a series of diagnostic steps, code edits, and deployments, and it captures the moment where two distinct root causes were identified and fixed simultaneously.

The message reads as follows:

I found two separate issues behind the "double responses" feeling:

>

- Real parallel starts: timer + path unit can both start the agent around the same time - Per-run duplicate assistant messages: the model often emits one assistant message before tools and another final assistant message after tools

>

I fixed both.

>

What changed: - vast_agent.py now takes an exclusive file lock on /var/lib/vast-manager/agent.lock - If timer + path unit fire together, one run proceeds, the other logs Another agent instance is already running — exiting. - The system prompt now explicitly says: - if the model is about to call tools, do not produce a long natural-language answer first - save the full explanation for the final response after tools are done - Conversation/context building now suppresses intermediate assistant messages - only the last assistant message per run is kept in the LLM context - The UI conversation view now also hides earlier assistant messages in the same run - so you see the final assistant answer, not the pre-tool narration + final narration pair

>

So the agent may still get two service activations in systemd logs, but only one does work, and the UI/context now prefer the final answer per run instead of showing both.

This message is deceptively simple. On its surface, it is a summary of fixes applied. But beneath that summary lies a rich story of diagnostic reasoning, architectural assumptions about LLM behavior, the interplay between systemd service management and Python process synchronization, and the challenge of building a reliable autonomous agent where the "agent" is simultaneously a shell service, a Python script, and an LLM with its own conversational patterns. This article unpacks each layer of that story.

The Context: An Autonomous Agent Under Construction

To understand why this message matters, one must understand what the agent is and how it operates. The agent is a Python script (vast_agent.py) that runs on a management host and makes autonomous decisions about a fleet of GPU instances on vast.ai. It uses an LLM (in this case, qwen3.5-122b) to analyze fleet state, demand signals, and operational conditions, then issues tool calls to launch, stop, or diagnose instances. The agent is triggered by two mechanisms: a systemd timer that fires every 5 minutes (a heartbeat), and a systemd path unit that fires whenever a trigger file is touched (for event-driven activation on state changes or human messages).

This dual-trigger architecture was designed to give the agent both periodic awareness and immediate responsiveness. But it introduced a race condition: the timer and the path unit could fire within milliseconds of each other, spawning two concurrent Python processes that both attempted to run the same agent logic. The agent's conversation history is stored in a SQLite database, and both processes would read the same pending notifications, call the same LLM, and produce similar responses. The result was the "double response" phenomenon.

The Diagnostic Journey: Separating Signal from Noise

The message at [msg 4973] is the culmination of a diagnostic chain that began several messages earlier. The user's directive at [msg 4960] was simple: "continue investigating the double responses." What followed was a deep forensic examination of the agent's runtime behavior.

The assistant's initial reasoning (visible in [msg 4961]) shows a careful exploration of hypotheses. The assistant considered whether pending notifications from human messages were being processed multiple times across runs, whether the path unit and timer were overlapping, and whether the lock file mechanism was sufficient. The reasoning blocks reveal a methodical mind working through the problem space: "It's possible that despite having a lock, the assistant's messages could still be duplicated within the same run." This is a critical insight — the assistant recognized that even if process-level parallelism was handled, there could be within-run duplication caused by the LLM's own conversational patterns.

The diagnostic process hit a snag when a complex SSH command failed due to zsh escaping issues ([msg 4961]). The assistant adapted, simplifying the approach into multiple smaller bash calls ([msg 4963]), and successfully retrieved journalctl logs showing overlapping agent runs at 20:34, before the file lock was in place. The logs revealed that runs #72 and #73 completed within seconds of each other — clear evidence of parallel service activation.

But the deeper investigation in [msg 4964] revealed something subtler: even after the file lock was implemented (which prevented parallel execution), the "double response" problem persisted in a different form. The assistant noted that "duplicate responses per run persist" and that run 77 had "just one assistant, along with tool outputs." This suggested that the LLM itself was producing multiple assistant messages within a single run — one narrative response before calling tools, and another final response after tools completed. This is a well-known behavior pattern in LLMs: when given a system prompt that encourages step-by-step reasoning, models often produce a natural-language "thinking" message before executing tool calls, then a separate final answer.

The Two Root Causes

The message at [msg 4973] crystallizes this investigation into two distinct root causes.

Root Cause 1: Real Parallel Starts. The systemd timer and path unit could both trigger the agent script simultaneously. Before the file lock was added, both processes would run to completion, each reading the same pending notifications, calling the LLM with the same context, and producing duplicate responses in the conversation history. The lock fixed this at the process level: one run acquires the exclusive lock, the other exits immediately with a log message.

Root Cause 2: Per-Run Duplicate Assistant Messages. Even with the lock preventing parallel execution, a single agent run could produce two assistant messages — one before tool calls and one after. This happened because the LLM, when given a complex task with tool access, naturally produces a narrative response explaining what it plans to do, then executes tools, then produces a final summary. Both messages were being stored in the conversation history and presented in the UI, creating the perception of a double response even when only one logical decision was made.

The distinction between these two root causes is important. The first is a systems engineering problem: process synchronization in a distributed service architecture. The second is an LLM interaction design problem: how to structure the model's output to avoid redundant communication. Fixing only one would have left the user still seeing double responses, just from a different source.

The Fixes: Four Layers of Defense

The assistant implemented four coordinated fixes, each targeting a different layer of the system.

Layer 1: Process-Level Mutual Exclusion. The vast_agent.py script now acquires an exclusive file lock (fcntl.flock with LOCK_EX) on /var/lib/vast-manager/agent.lock at startup. If a second process attempts to start while the first is running, the lock acquisition fails (with LOCK_NB for non-blocking), and the second process logs "Another agent instance is already running — exiting." This is a standard pattern for ensuring singleton execution, but its placement here is critical: it was added after the assistant tested the lock mechanism manually ([msg 4955]) and confirmed that a systemd-triggered run was already holding the lock ([msg 4957]). The lock file approach was chosen over alternatives like systemd's ExecCondition or a flock wrapper because it keeps the synchronization logic co-located with the agent code, making it easier to maintain and test.

Layer 2: Prompt Engineering. The system prompt was amended with an explicit instruction: if the model is about to call tools, it should not produce a long natural-language answer first. Instead, it should save the full explanation for the final response after tools are done. This is a behavioral constraint placed directly on the LLM, leveraging the model's instruction-following capability to suppress the pre-tool narrative. The assistant applied this patch at [msg 4968], adding the rule to the "CRITICAL RULES" section of the system prompt. This is a pragmatic acknowledgment that LLMs have conversational habits that must be explicitly overridden for reliable tool-use scenarios.

Layer 3: Context Filtering. The conversation-building logic in vast_agent.py was modified to suppress intermediate assistant messages when constructing the LLM context. Only the last assistant message per run is included in the prompt sent to the model. This prevents the LLM from seeing its own earlier draft responses and potentially building on them, which could create confusion. The patch at [msg 4969] replaced the simple enumeration of all messages with logic that computes the last assistant message ID per run and skips earlier ones. This is a pragmatic compromise: it preserves the full decision history in the database for auditing, but feeds only the final, authoritative response to the model.

Layer 4: UI Rendering. The JavaScript rendering loop in ui.html was similarly modified to hide earlier assistant messages within the same run. The patch at [msg 4971] introduced a lastAssistantIdByRun map that tracks the final assistant message ID for each run, and the rendering loop skips any assistant message that is not the last for its run. This ensures that the human operator sees a clean, non-redundant conversation view, even if the database contains the full history.

Assumptions and Their Validity

Several assumptions underpin the fixes in this message, and it is worth examining them critically.

Assumption: The LLM will follow the prompt instruction to suppress pre-tool narrative. This is the most fragile assumption. LLMs are not deterministic, and prompt instructions — especially negative instructions ("do not produce...") — can be unreliable. The model might still occasionally produce a pre-tool narrative despite the instruction. The context filtering (Layer 3) serves as a safety net: even if the model produces an intermediate message, it will be filtered out of the prompt for future runs and hidden from the UI. This layered defense is wise engineering.

Assumption: The last assistant message per run is the most authoritative. This is generally true for the agent's current design, where the model produces tool calls and then a final verdict. But if the model were to produce a final message that contradicts earlier reasoning, the system would lose that nuance. The assistant briefly considered this concern in [msg 4970]: "I'm thinking about whether to skip tool calls as it seems they only keep the last assistant message. This might lead to losing important decisions from earlier if only the final verdict is kept." The conclusion was that the final message should include all necessary explanation, and the system prompt was adjusted accordingly.

Assumption: The file lock is sufficient for process synchronization. The lock prevents concurrent execution, but it does not prevent sequential duplicate runs. If the timer fires, the lock is acquired, the run completes, the lock is released, and then the path unit fires (because the trigger file was touched during the run), a second run could start immediately after the first. This would still produce duplicate responses, just not in parallel. The assistant acknowledged this implicitly by noting that "the agent may still get two service activations in systemd logs, but only one does work." The context filtering handles the sequential case: the second run would see the first run's final message in its context and make a different decision (or decide no action is needed).

Input Knowledge Required

To fully understand this message, one needs knowledge in several domains:

• Systemd service management: Understanding how timer units and path units work, and how they can race when triggered by the same event.
• File locking in Python: The fcntl.flock mechanism and the semantics of LOCK_EX and LOCK_NB.
• LLM interaction patterns: The tendency of instruction-following models to produce narrative responses before and after tool calls, and the techniques for controlling this behavior through prompt engineering.
• Conversation context management: The challenge of maintaining a coherent history for an LLM across multiple runs, and the trade-offs between completeness and token budget.
• The agent's architecture: The specific roles of vast_agent.py, the Go backend (vast-manager), the SQLite conversation store, and the UI rendering in ui.html. Without this knowledge, the message reads as a simple list of fixes. With it, one can appreciate the careful reasoning that led to each decision.

Output Knowledge Created

This message creates several pieces of output knowledge that are valuable for anyone building similar autonomous agent systems:

1. Duplicate responses have multiple root causes. The same symptom ("I see two responses") can arise from process-level races, LLM behavioral patterns, or both. Diagnosing requires examining logs, conversation history, and model outputs separately.
2. Layered defenses are necessary. No single fix is sufficient. The file lock prevents parallel execution, but without context filtering, a single run's duplicate messages would still pollute the history. Without prompt engineering, the model would continue producing redundant narratives. Each layer addresses a different failure mode.
3. LLM behavior can be shaped but not guaranteed. The prompt instruction to suppress pre-tool narrative is a soft constraint. The hard constraint is the context filtering that removes intermediate messages regardless of whether the model complied. This pattern — prompt for best-effort behavior, filter for guaranteed correctness — is broadly applicable.
4. UI and context should be decoupled from storage. The database retains the full conversation history for auditing and debugging, but the LLM context and the UI both apply filtering to present a clean view. This separation of concerns allows the system to be both complete (in storage) and clean (in presentation).

The Thinking Process Visible in the Reasoning

The assistant's reasoning blocks throughout this diagnostic chain reveal a particular style of problem-solving. The thinking is iterative and hypothesis-driven: each reasoning block proposes a theory, considers evidence, and refines the approach. In [msg 4961], the assistant cycles through three distinct hypotheses (pending notification duplication, path/timer overlap, trigger file modification) before settling on the need to examine actual logs. In [msg 4964], the assistant recognizes that overlapping runs occurred before the lock was in place, and that even after the lock, duplicate responses persisted — forcing the investigation to shift from process-level to message-level causes.

The reasoning also shows a healthy awareness of the assistant's own limitations. In [msg 4964], the assistant writes: "I'm wondering if I could also detect and remove duplicate runs from before—maybe that's a bit complex?" This moment of self-reflection — recognizing when a proposed solution is too complex — leads to the simpler, more targeted approach of filtering within runs rather than attempting to deduplicate across runs retrospectively.

Conclusion

The message at [msg 4973] is a masterclass in concise technical communication. In a few paragraphs, it summarizes a multi-hour diagnostic journey, identifies two distinct root causes, describes four coordinated fixes across three layers of the system (process synchronization, prompt engineering, context management, and UI rendering), and honestly acknowledges the residual limitation (systemd may still show two activations). The fixes are not theoretical — they were built, deployed, and verified in production, as shown by the build and deployment steps in [msg 4972].

For anyone building autonomous LLM agents — especially agents that must operate reliably in production environments — this message offers a valuable lesson: the "agent" is never just the LLM. It is the LLM plus the process management, the context handling, the UI, the lock files, and the systemd services. Reliability emerges not from any single component but from the careful integration of all of them, with layered defenses that catch failures at multiple levels. The double-response bug was not a bug in the LLM or a bug in the Python script or a bug in systemd — it was a bug in the interaction between all three, and fixing it required touching all three.