The Diagnostic Mind: Tracing Duplicate Responses in an Autonomous LLM Fleet Agent

Introduction

In the complex tapestry of building an autonomous LLM-driven fleet management system, few problems are as maddening as the duplicate response. It is the kind of bug that erodes trust silently: the operator sees the same observation twice, the same decision twice, the same log line twice, and begins to wonder whether the system is hallucinating, the infrastructure is broken, or both. In message [msg 4961] of this opencode session, the assistant finds itself precisely at this inflection point — having just deployed a file-lock mechanism to prevent parallel agent runs, only to be told by the user to "continue investigating the double responses." The lock was supposed to solve the problem. It did not. And now the assistant must dig deeper.

This message is a window into the diagnostic reasoning of an AI system debugging its own runtime infrastructure. It is not a message about writing code or deploying features; it is a message about thinking about what is broken — formulating hypotheses, weighing evidence, planning investigations, and confronting the gap between expected behavior and observed reality. The message contains four distinct reasoning blocks, each exploring a different causal theory, followed by a bash command intended to gather forensic evidence. And crucially, that bash command fails — a failure that itself reveals something about the brittleness of the very infrastructure the agent is trying to debug.

The Full Text of the Message

The subject message reads as follows:

[assistant] ## Agent Reasoning Investigating duplicate responses

>

I need to keep looking into the issue of duplicate responses. I'll check the logs for any duplicate agent calls. It's possible that despite having a lock, the assistant's messages could still be duplicated within the same run. I should figure out if the path trigger and timer are both running sequentially, which could cause the same question to be processed twice. The user's reports of double responses mean I need to investigate further and identify the root cause. Implementing deduplication or event coalescing might be necessary.

>

## Agent Reasoning Analyzing notification pending issues

>

I'm thinking that pending notifications from the same human message might be sticking around across runs. Since we consider all run_id=0 user messages with IDs greater than the last assistant run's max message ID, if notifications aren't marked as consumed after processing, each subsequent run could view them as pending. The logic seems okay, but perhaps the human message could trigger the path unit and timer too closely together, causing overlap. However, if the first run processes the message, the assistant messages should have higher IDs, preventing the second run from seeing them as pending.

>

## Agent Reasoning Investigating possible duplicates

>

I'm considering whether the duplicate response could be caused by both the path unit service and timer service using the same agent script, even if there's a weird log unit name. I need to check if there are cases of duplicate assistant content in conversations around similar times. I'll inspect recent systemd logs and look for any instances where the trigger file was touched multiple times due to the same event, which might cause multiple starts. The lock should prevent parallel runs, but if the first run modifies the trigger file, it could unintentionally restart the service and lead to duplicates if notifications are still pending.

>

## Agent Reasoning Investigating run issues

>

It seems like the first run failed due to a lack of user query and maybe retriggered. I'm wondering if this could be causing a duplicate run because of event paths and timers. I should probably check the actual logs now. I'll look into the recent conversations around the same timestamps and run IDs to see if a similar user message triggered two runs. Time to inspect using the tools at my disposal!

>

[bash] ssh theuser@10.1.2.104 " echo '=== Recent agent runs from journals ===' journalctl -u vast-agent -u vast-agent-trigger.service --since '30 min ago' --no-pager 2>&1 | grep -E '=== Agent run started ===|completed in|Another agent instance|Found [0-9]+ pending notifications|No action needed|human message|triggered via' | tail -80 ... " 2>&1 zsh:1: no matches found: ...

The Context: A Lock That Wasn't Enough

To understand why this message exists, we must trace back to the user's report in [msg 4930]. The user had observed a troubling pattern: "duplicate calls to cron observe, so seems a lot of the time we call the agent in two parallel streams(?) and responses also duplicate?" The user suggested two remedies: a semaphore to prevent parallel runs, and a structured JSON return from the LLM so that no-action observations could be cleanly pruned from the conversation history.

The assistant responded in [msg 4931] with a comprehensive plan. Three fixes were identified:

  1. A file lock to prevent the systemd timer and systemd path unit from running the agent script simultaneously.
  2. A structured JSON verdict appended to the LLM's response, containing action and state_changed boolean fields.
  3. Pruning logic to delete no-action runs from the conversation history, keeping the context window clean. The assistant implemented all three changes across a series of edits ([msg 4941] through [msg 4948]), deployed them, and tested them. The file lock worked — when a second agent instance tried to start while the first was running, it correctly exited with "Another agent instance is already running — exiting." The verdict system also worked: the log showed "Verdict: action=True state_changed=True" for a run that took action. But then the user said: "continue investigating the double responses" ([msg 4960]). The implication was clear: despite the lock, duplicates were still occurring. The lock prevented parallel runs, but duplicates could still arise from sequential runs — two invocations happening one after the other, each processing the same event. The problem was subtler than a simple race condition.

The Reasoning Process: Four Hypotheses, One Investigation

What makes [msg 4961] particularly interesting is the structure of its reasoning. The assistant does not simply execute a command; it first thinks through four distinct hypotheses, each building on the previous one.

Hypothesis 1: Duplication Within a Single Run

The first reasoning block starts with a crucial insight: "It's possible that despite having a lock, the assistant's messages could still be duplicated within the same run." This is a sophisticated observation. The lock prevents two Python processes from running simultaneously, but it does nothing about the internal logic of a single run. If the LLM produces two responses within one invocation — perhaps because the tool-calling loop iterates twice — those responses would both be persisted to the conversation history, creating the appearance of duplicates.

This hypothesis gets at a fundamental distinction in distributed systems: concurrency versus sequential repetition. A lock solves concurrency (two things at the same time) but not sequential repetition (the same thing happening twice in a row). The assistant is correctly identifying that the problem space is larger than the lock can address.

Hypothesis 2: Stale Pending Notifications

The second reasoning block explores a more subtle mechanism: "pending notifications from the same human message might be sticking around across runs." The assistant describes the notification logic in detail: the system considers all run_id=0 user messages with IDs greater than the last assistant run's max message ID as "pending." If notifications are not marked as consumed after processing, each subsequent run would see them as new, causing the same human message to be processed repeatedly.

This is a classic state-management bug. The assistant recognizes that the notification system lacks a "mark as read" mechanism. Each run re-discovers the same pending notifications, re-processes them, and generates a new response. The lock does nothing to prevent this because the runs are sequential, not parallel.

However, the assistant also identifies a potential counterargument: "if the first run processes the message, the assistant messages should have higher IDs, preventing the second run from seeing them as pending." This is a subtle point about the ID assignment scheme. If assistant messages are assigned monotonically increasing IDs, then after the first run creates its response, the "last assistant run's max message ID" would advance past the pending notification, and the second run would not see it. But this only works if the ID assignment is correct — and as later events in the session reveal, the ID scheme was indeed fragile.

Hypothesis 3: Trigger File Re-triggering

The third reasoning block introduces yet another mechanism: "if the first run modifies the trigger file, it could unintentionally restart the service." This is a feedback-loop problem. The agent's event-driven triggering works through a systemd path unit that watches a trigger file. If the agent, during its execution, writes to or touches that trigger file, it could cause the path unit to fire again, spawning a new invocation as soon as the current one finishes.

This is a genuinely dangerous class of bug — a self-reinforcing trigger loop. The assistant correctly identifies that "the lock should prevent parallel runs, but if the first run modifies the trigger file, it could unintentionally restart the service and lead to duplicates if notifications are still pending." The lock only prevents overlap; it does not prevent sequential re-triggering.

Hypothesis 4: First Run Failure and Retrigger

The fourth reasoning block considers a simpler explanation: "the first run failed due to a lack of user query and maybe retriggered." If the first run encountered an error (perhaps because there was no user message to process, or the LLM returned an empty response), it might have exited without properly marking notifications as processed. The systemd path unit, seeing the trigger file still active, would then fire again, causing a duplicate.

This hypothesis is notable because it acknowledges that the agent's own failure modes can cascade. A failed run does not clean up after itself, leaving the system in a state that guarantees a duplicate run.

The Failed Command: A Meta-Debugging Lesson

After the four reasoning blocks, the assistant executes a bash command to gather forensic evidence. The command is ambitious: it SSHes into the management host, queries systemd journals for recent agent runs, fetches the conversation history from the vast-manager API, and runs a Python script to detect duplicate observations.

But the command fails spectacularly. The shell (zsh) reports: "no matches found" — the multi-line heredoc and special characters in the command string were mangled during transmission. The <<'PY' heredoc delimiter, the pipe characters, the grep patterns — all got interpreted by the local shell before being sent to the remote host.

This failure is deeply instructive. It reveals the brittleness of the assistant's tool-use infrastructure when dealing with complex shell commands. The assistant is trying to debug a distributed system — the agent runs on a remote host, the conversation data lives in a Go API server, the logs are in systemd journals — but the debugging tool itself (the bash command) is fragile. The quoting, escaping, and heredoc handling required to pass a multi-line Python script through SSH is non-trivial, and the assistant gets it wrong.

This is not a trivial mistake. It is a symptom of the same class of problem the assistant is trying to debug: complex distributed systems have failure modes at every layer. The agent's runtime infrastructure (SSH, systemd, shell scripting) is itself a source of bugs, not just the agent's decision-making logic.

Assumptions and Their Consequences

The assistant makes several assumptions in this message, some explicit and some implicit:

  1. The lock is working correctly. The assistant assumes that the file-lock mechanism is functioning as designed. But earlier in the conversation ([msg 4954]), the lock had a permission issue — the directory was owned by root and the agent ran as a non-root user. This was fixed by changing directory permissions, but the fix was ad-hoc. If the permissions change is not persisted across reboots or deployments, the lock could silently fail.
  2. The notification ID scheme is monotonic. The assistant assumes that message IDs increase monotonically and that the "last assistant run's max message ID" can be reliably used to determine which notifications are new. This assumption is challenged later in the session when a session reset breaks the ID scheme entirely ([chunk 32.4]).
  3. The systemd path unit and timer are the only triggering mechanisms. The assistant considers these two sources of duplicate runs but does not consider that the trigger file could be modified by other processes — for example, by the vast-manager API itself when processing a user message.
  4. The first run's failure was due to "lack of user query." This is a hypothesis, not a confirmed fact. The assistant does not verify this by checking the logs of the failed run before forming the hypothesis.
  5. The command will work as written. The assistant assumes that the complex multi-line bash command with heredocs and pipes will survive SSH transport and shell interpretation. This assumption is immediately falsified by the zsh error.

Input Knowledge Required

To understand this message, the reader needs knowledge of:

Output Knowledge Created

This message creates:

The Broader Significance

[msg 4961] is a microcosm of the challenges involved in building reliable autonomous AI systems. The assistant is not just debugging a bug; it is debugging the interaction between multiple subsystems — the Python agent script, the Go API server, the systemd timer and path units, the SSH transport, the shell environment, and the LLM itself. Each subsystem has its own failure modes, and the duplicate response problem could originate in any of them.

The message also illustrates a pattern that recurs throughout this segment: the assistant's reasoning is thorough and multi-faceted, but its execution is sometimes fragile. The four reasoning blocks show genuine diagnostic sophistication — the assistant considers race conditions, state management, feedback loops, and cascading failures. But the bash command that follows fails because of shell quoting issues. This gap between reasoning and execution is a recurring theme in autonomous systems: thinking is easier than doing.

The user's response to this message is not shown in our window, but the subsequent chunks ([chunk 32.4] and [chunk 32.5]) reveal that the root causes were eventually found: stale scheduled wakes accumulating without being marked processed, intermediate and final assistant messages both polluting the LLM prompt, and bursty state-change events triggering the path unit multiple times. The assistant's hypotheses in [msg 4961] were on the right track — particularly the second hypothesis about pending notifications not being consumed — but the full diagnosis required deeper investigation into the wake-processing logic and the event-debouncing mechanism.

Conclusion

Message [msg 4961] captures a moment of diagnostic tension in the development of an autonomous fleet management agent. The assistant has deployed a fix (the file lock) that addressed one class of duplicate (parallel runs) but not the deeper class (sequential re-processing of the same events). The four reasoning blocks show the assistant working through increasingly sophisticated hypotheses, from within-run duplication to stale notifications to trigger-loop feedback. The failed bash command is a reminder that debugging distributed systems requires not just good hypotheses but also robust tooling — and that the debugging infrastructure itself must be hardened against failure.

In the end, the duplicate response problem was solved not by any single fix but by a combination of mechanisms: debouncing, notification consumption tracking, monotonic run IDs, and context pruning. Each of these fixes addressed one of the hypotheses the assistant formulated in this message. The reasoning was sound; the execution needed iteration. And that, perhaps, is the most important lesson of all.