Engineering Reliability into an Autonomous LLM Agent: The Verdict System and File Lock

Introduction

In the complex ecosystem of autonomous AI agents, few challenges are as deceptively difficult as preventing the agent from getting in its own way. When an LLM-powered system is designed to observe, decide, and act on a live production fleet of GPU instances, the difference between a helpful assistant and a destructive liability often comes down to the subtle engineering of its operational loop—the mechanisms that prevent duplicate work, prune irrelevant history, and extract structured decisions from unstructured language. Message <msg id=4931> in this conversation captures a pivotal moment where the assistant, responding to sharp user feedback, redesigned the core architecture of an autonomous fleet management agent to solve exactly these problems. This message is not an implementation—it is a planning document, a diagnosis, and a blueprint rolled into one, revealing the assistant's reasoning about three interrelated reliability issues that threatened the agent's viability in production.

Context: The Agent's Journey So Far

To understand the significance of this message, one must appreciate what came before it. The autonomous agent in question manages a fleet of GPU instances on vast.ai, running Filecoin SNARK proving workloads (cuzk/curio). Over the preceding chunks of this segment, the assistant had built this agent from scratch, iterating through multiple architectural phases: a simple cron-based observer, an event-driven trigger via systemd path units, a diagnostic sub-agent system for grounding decisions in facts, and a sophisticated context management system using SQLite-based rolling conversations.

The agent had already survived several critical failures. Most notably, it had once misinterpreted active=False as "no demand" and destroyed all running instances despite 59 pending tasks—a catastrophic error that required adding demand_queued and workers_dead flags to prevent the agent from conflating "no demand" with "all workers dead." It had suffered from context overflow (ballooning to 38k tokens), session reset bugs that broke the agent loop entirely, and bursty event-driven triggers that spawned redundant runs.

By the time we reach <msg id=4931>, the agent has just received its most sophisticated feature: a diagnostic grounding system that forces the agent to SSH into instances and collect raw logs before making destructive decisions like stopping instances. The stop_instance tool is gated with HTTP 428 "Precondition Required," forcing the agent to call diagnose_instance first. This was the culmination of the previous chunk's work—a shift from brittle hard-coded rules to an evidence-driven, LLM-powered diagnostic layer.

The User's Report: Three Problems in Production

The immediate trigger for <msg id=4931> is the user's message at <msg id=4930>, which reports a cluster of operational issues that had become apparent in production:

"There seem to be duplicate calls to cron observe, so seems a lot of the time we call the agent in two parallel streams(?) and responses also duplicate? Should have some semaphore. Also on the wait/idle no-action checking we don't remove the last message correctly, presumably because there is no sentinel that we can match on easily. The agent in cron calls should be prompted to additionally return a json return like {"action": bool, "meaningful_state_change": bool}, then we parse for this json within the larger json response."

The user identifies three distinct problems:

  1. Duplicate parallel agent runs: The systemd timer (which runs every 5 minutes) and the systemd path unit (which triggers on state-change events) can fire simultaneously, causing two agent instances to run in parallel. Both produce observations and responses, polluting the conversation history and potentially causing conflicting actions.
  2. Idle/no-action observations not cleaned up: When the agent runs but determines no action is needed (e.g., everything is stable), its observation and response still get appended to the conversation history. Over time, these "no-op" messages accumulate, consuming precious context window space and diluting the signal-to-noise ratio of the conversation. The existing fast-path code tried to avoid appending observations when no action was needed, but there was no clean way to identify and remove them after the fact.
  3. Need for structured JSON return: The system was trying to parse the agent's natural language output to determine whether action was taken. This is inherently fragile—LLM outputs are variable, and detecting "did the agent do something meaningful" from prose is error-prone. The user proposes having the agent return a structured JSON block that can be reliably parsed. These three problems are interconnected. Duplicate runs produce duplicate observations. No-op runs produce observations that shouldn't exist. And without a structured verdict, there's no reliable way to distinguish meaningful runs from noise.

The Assistant's Reasoning: A Three-Pronged Analysis

The assistant's response in <msg id=4931> begins with a structured "Agent Reasoning" section that demonstrates a clear analytical process. The assistant breaks down the user's report into the same three issues and immediately begins designing solutions:

Fix 1: File Lock for Parallel Run Prevention

The assistant identifies the root cause of duplicate runs: "the systemd timer AND the systemd path unit can trigger simultaneously, causing two parallel agent runs that duplicate observations and responses." The proposed solution is elegantly simple: a file-level lock using fcntl.flock at the start of the Python script. If another instance is already running, the script exits immediately. This prevents both the timer and path unit from triggering simultaneously because whichever process acquires the lock first wins, and the other exits harmlessly.

This is a classic operating-system-level concurrency control mechanism. The assistant chooses fcntl.flock over alternatives like a PID file or a database-level lock because:

Fix 2 & 3: Structured JSON Verdict and No-Op Pruning

The assistant recognizes that Fixes 2 and 3 are deeply intertwined. The core insight is: rather than trying to detect after the fact whether an observation should be kept, have the agent itself declare what happened in a structured format that can be reliably parsed.

The design works as follows:

  1. The system prompt is modified to instruct the LLM to end every response with a structured JSON verdict block containing fields like action (bool) and state_changed (bool).
  2. After the LLM responds, the Python wrapper parses the response for this JSON block.
  3. If action is false and state_changed is false (a no-op run), the wrapper removes all messages from that run from the conversation history.
  4. If meaningful action occurred, the messages are kept for context. This solves both problems at once: the structured JSON provides a reliable sentinel for detecting no-op runs, and the pruning keeps the conversation history clean. The assistant's reasoning reveals a key insight: "The real problem is that even when the LLM runs due to notifications, it might still conclude no action was needed—and that response clutters the conversation history." This is a profound observation about autonomous agents. Notifications (state-change events) don't always require action—sometimes the agent observes a change and decides it's benign. But the very act of running the agent produces messages that become part of the permanent context. Over time, these accumulate and waste tokens, potentially pushing out genuinely useful context.

Assumptions and Their Consequences

The assistant's reasoning in <msg id=4931> makes several assumptions, some of which prove problematic during implementation:

Assumption 1: The file lock will be straightforward. The assistant assumes that opening a lock file and calling fcntl.flock will work without issues. In practice, the lock file's directory has restrictive permissions (owned by root), and the agent sometimes runs as a non-root user. This causes a PermissionError that requires fixing directory permissions to 777. Additionally, the assistant initially doesn't consider that a previous agent run might crash without releasing the lock, leading to a "stuck lock" scenario where the lock file exists but no process holds it. The fix for this—removing the stale lock file—works because open(lock_path, "w") creates a new inode, and the old file descriptor is gone.

Assumption 2: The LLM will reliably produce structured JSON. The assistant assumes that instructing the LLM to end with a JSON verdict block will produce consistently parseable output. This is a reasonable assumption given modern LLM instruction-following capabilities, but it's not guaranteed. The assistant later adds parsing logic that must handle cases where the JSON is malformed, embedded in prose, or missing entirely. The system prompt engineering becomes critical here—the instructions must be precise enough that the model complies reliably.

Assumption 3: Pruning no-op runs from conversation history is safe. The assistant assumes that removing messages from no-op runs won't break the conversation flow or lose important context. This is mostly true—if nothing happened, there's nothing to remember. However, there's a subtle edge case: what if a no-op run contains important observations that inform future decisions? For example, the agent might note "all instances healthy, no action needed" which is useful context for the next run. The assistant's design sacrifices this nuance for cleanliness, which is a reasonable trade-off given the token budget constraints.

Assumption 4: The user wants all three fixes implemented simultaneously. The assistant treats the three issues as a single package to be implemented together. This is efficient but risky—if one fix has a bug, it could mask or complicate the others. The assistant's approach of implementing all three, building, and deploying in a single cycle is ambitious.

Input Knowledge Required

To fully understand <msg id=4931>, the reader needs knowledge of:

  1. The agent architecture: The agent runs as a Python script triggered by two systemd units—a timer (every 5 minutes) and a path unit (on state-change events). It maintains a rolling conversation in SQLite, accessed via a Go API (vast-manager). The conversation has a 30k token window with compaction for stale tool outputs.
  2. The systemd path unit: A systemd .path unit watches a trigger file for modifications. When the file is touched (by the Go API on state-change events), systemd starts the agent service. This is in addition to the regular timer, creating the potential for parallel invocations.
  3. The conversation history model: Messages in the conversation have id, run_id, role, timestamp, and content fields. The run_id groups messages from a single agent invocation. The assistant needs to identify and delete all messages belonging to a no-op run.
  4. The existing fast-path logic: The agent already has code that tries to avoid appending observations when no action is needed. This code is acknowledged as insufficient—it doesn't reliably detect no-op runs.
  5. The API surface: The Go API has endpoints for conversation management (GET /api/agent/conversation, POST /api/agent/conversation). The assistant needs to add a DELETE /api/agent/conversation/{id} endpoint to support message deletion.
  6. The system prompt: The existing system prompt instructs the agent on its role, available tools, and operational rules. The assistant needs to modify this prompt to add the verdict requirement.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. The verdict system design: A pattern for extracting structured decisions from LLM outputs by having the model emit a parseable JSON block. This is a general technique applicable to any LLM-powered agent that needs to make go/no-go decisions.
  2. The file lock pattern: Using fcntl.flock for preventing parallel agent invocations. This is a robust, OS-level concurrency control mechanism that works across process boundaries and automatically releases on crash.
  3. The no-op pruning strategy: A method for keeping conversation history clean by removing messages from runs that didn't produce meaningful action. This prevents token waste and maintains signal quality in the context window.
  4. The integration pattern: The assistant designs how the Python agent, Go API, and systemd units interact to implement these fixes. The Python script acquires the lock, runs the LLM, parses the verdict, and calls the Go API to delete messages. The Go API provides the DELETE endpoint.
  5. A diagnostic framework: The assistant's reasoning process itself demonstrates how to analyze production issues in an autonomous agent system—identify root causes (parallel triggers, no sentinel), design minimal fixes (file lock, JSON verdict), and validate assumptions through testing.

The Thinking Process: A Window into Agent Reasoning

What makes <msg id=4931> particularly valuable is the visibility it provides into the assistant's reasoning process. The "Agent Reasoning" sections show a step-by-step analytical approach:

  1. Problem identification: The assistant parses the user's report into three distinct issues, demonstrating the ability to decompose a complex complaint into actionable components.
  2. Root cause analysis: For the duplicate runs issue, the assistant correctly identifies that the systemd timer and path unit can trigger simultaneously. This requires understanding the systemd architecture and recognizing that two independent triggers can race.
  3. Solution design: For each issue, the assistant designs a solution, considering trade-offs. The file lock is chosen over alternatives like PID files or database semaphores. The JSON verdict is chosen over heuristic detection of no-op runs.
  4. Integration thinking: The assistant considers how the solutions interact. The verdict system enables the pruning system. The file lock prevents the duplicate runs that would make pruning more complex.
  5. Implementation planning: The assistant creates a todowrite with three todos, establishing a clear execution plan: add file lock, add verdict/pruning, build/deploy/test. The reasoning also reveals the assistant's awareness of the system's brittleness. The phrase "even when the LLM runs due to notifications, it might still conclude no action was needed—and that response clutters the conversation history" shows an understanding that the agent's own operation produces side effects that can degrade its performance over time. This is a meta-cognitive insight—the agent recognizing that its own existence in the conversation is a resource that must be managed.

Conclusion

Message <msg id=4931> represents a critical architectural pivot in the development of an autonomous fleet management agent. Faced with three interconnected reliability issues—duplicate parallel runs, no-op pollution of conversation history, and the absence of structured decision output—the assistant designs a coherent set of fixes that address all three simultaneously. The file lock prevents race conditions at the OS level. The structured JSON verdict provides a reliable signal for decision detection. The pruning mechanism keeps the conversation history clean and token-efficient.

What makes this message noteworthy is not just the technical solutions but the reasoning process behind them. The assistant demonstrates the ability to decompose complex production issues, identify root causes, design minimal interventions, and plan integrated implementation. The assumptions made—about file permissions, LLM reliability, and the safety of pruning—are reasonable but tested through subsequent implementation, revealing the iterative nature of engineering reliable autonomous systems.

In the broader context of the conversation, this message is the moment where the agent's operational loop transitions from fragile to robust. The fixes implemented here—particularly the verdict system and file lock—become foundational to the agent's reliability, enabling it to run continuously without degrading its own context or conflicting with itself. For anyone building autonomous LLM agents, the patterns established in this message—structured verdict extraction, OS-level concurrency control, and context-aware pruning—are directly applicable lessons in the art of keeping agents from getting in their own way.