The Lock That Wouldn't Let Go: Debugging Parallel Agent Runs in an Autonomous Fleet Manager

[assistant] Now test the full agent:
[bash] ssh theuser@10.1.2.104 "
export AGENT_LLM_BASE_URL=[REDACTED_API_ENDPOINT]
export AGENT_LLM_API_KEY=[REDACTED]
export AGENT_LLM_MODEL=qwen3.5-122b
export VAST_MANAGER_URL=http://127.0.0.1:1236
export AGENT_LOG_FILE=/tmp/vast-agent-test.log
python3 /opt/vast-agent/vast_agent.py 2>&1
" 2>&1
2026-03-17 20:51:07 [INFO] vast-agent starting (pid=639452)
2026-03-17 20:51:07 [INFO] Another agent instance is already running — exiting.

At first glance, this message looks like a success. The file lock works. The agent correctly detects that another instance is already running and exits gracefully. But this message is not a celebration — it is the culmination of a frustrating debugging session where a seemingly simple mechanism kept failing in ways that exposed deeper architectural tensions between testing, production, and the inherent complexity of building autonomous systems.

The Problem: Duplicate Agents and the Need for a Semaphore

The story begins with the user's observation in [msg 4930]: "There seem to be duplicate calls to cron observe, so seems a lot of the time we call the agent in two parallel streams(?) and responses also duplicate? Should have some semaphore."

The autonomous fleet management agent — a Python script that uses an LLM to make scaling decisions for a GPU proving cluster — was triggered by two systemd mechanisms: a timer (every 5 minutes) and a path unit (on urgent events like state changes). When both fired simultaneously, two agent processes would run in parallel, each making observations, calling the LLM, and appending duplicate messages to the conversation history. This race condition polluted the agent's memory, wasted tokens, and could lead to conflicting decisions.

The user's request was precise: add a semaphore to prevent parallel runs, and also add a structured JSON return ({"action": bool, "meaningful_state_change": bool}) so the system could cleanly determine whether a run had taken meaningful action and prune idle observations from the conversation history.

The Assistant's Response: A Three-Pronged Fix

In [msg 4931], the assistant laid out a comprehensive plan. Three issues needed fixing:

Race condition: The timer and path unit could trigger simultaneously, causing duplicate runs.
Idle observations polluting conversation: No-action runs left clutter with no clean way to identify and remove them.
Structured return needed: The system needed a parseable JSON verdict from the LLM to decide what to keep. The assistant's design decisions were thoughtful. For the race condition, a file lock using fcntl.flock was chosen — a Unix-native mechanism that automatically releases when the process exits, avoiding the risk of stale locks. For the structured return, the system prompt would be modified to instruct the LLM to end every response with a <verdict>{"action": bool, "state_changed": bool}</verdict> block, which the Python script would parse to decide whether to prune the run's messages from the conversation.

Implementation and Deployment

The assistant implemented the changes across multiple files. The file lock was added to the main() function of vast_agent.py, checking the lock at startup and exiting immediately if another instance held it. The system prompt was updated with instructions for the verdict block. A new Go endpoint was added to the vast-manager API to support deleting individual conversation messages. The code was built, deployed to the management host, and tested.

The initial test in [msg 4950] showed the lock working: "Another agent instance is already running — exiting." But this was the production timer, not a stale lock. The assistant waited 30 seconds and tried again in [msg 4951], but got no output — the timer run was still in progress. In [msg 4952], the same "already running" message appeared. The lock was stuck.

The Debugging Descent

What followed was a classic debugging spiral. In [msg 4953], the assistant checked what was holding the lock using fuser, found no process, and force-removed the lock file. But in [msg 4954], the lock was still reported as held — even after removing the file. This is where the assistant made an incorrect assumption: that removing the file and reopening would create a fresh lock. The actual problem was a permission error — the lock file was being created in /var/lib/vast-manager/ which was owned by root, but the agent was sometimes running as the non-root user theuser. The open() call was failing with PermissionError, and the exception handler was treating this as "lock already held."

In [msg 4955], the assistant fixed the permissions by running sudo chmod 777 /var/lib/vast-manager/ and verified the lock worked manually. Then came the subject message: the full agent test.

The Subject Message: A Test That Reveals More Than Intended

The subject message is the assistant's attempt to verify the entire system after the permission fix. The command is straightforward: SSH into the management host, set environment variables (LLM endpoint, API key, model, manager URL), and run the agent script.

The output is anticlimactic: "Another agent instance is already running — exiting."

On the surface, this is the lock working correctly. But the message is more interesting for what it reveals about the state of the system at this moment. The lock is being held by a legitimate systemd timer run — the production agent is doing its job. The assistant's manual test is blocked by the very mechanism it just deployed. This is both a validation of the fix and a demonstration of a practical problem: how do you test a system that runs on a timer without conflicting with the timer?

Assumptions and Their Consequences

Several assumptions shaped this message. The assistant assumed that the lock file mechanism would be straightforward — open a file, acquire an exclusive lock, release on exit. This assumption was correct in principle but failed in practice due to the permission issue. The assistant also assumed that removing the stale lock file would resolve the problem, but the real issue was that the open() call itself was failing silently, not that the lock was held.

A deeper assumption was that the lock would naturally release when the holding process exited. This is true for fcntl.flock — the kernel automatically releases the lock when the file descriptor is closed, which happens on process exit. But the permission error meant the lock was never actually acquired — the error was being misinterpreted.

Input Knowledge Required

To understand this message, the reader needs knowledge of:

Unix file locking: How fcntl.flock works, that locks are associated with file descriptors and released on process exit.
Systemd timer and path units: How the agent is triggered by both a timer (periodic) and a path unit (event-driven), creating the race condition.
The agent architecture: A Python script that runs as a single invocation, uses an LLM to make decisions, and persists conversation history via a Go API.
The production context: A GPU proving cluster on vast.ai where instances take 1-2 hours to start and the agent manages scaling decisions.
The debugging history: The previous messages showing the stuck lock, the permission investigation, and the manual lock test.

Output Knowledge Created

This message creates knowledge about the state of the system at a specific point in time:

The file lock mechanism is operational and blocking parallel runs.
A production agent run is in progress, preventing manual testing.
The permission fix was necessary but not sufficient for testing — the production schedule interferes.
The system is now more robust against duplicate runs, but testing requires coordination with or temporary suspension of the production timer.

The Thinking Process

The assistant's thinking in this message is visible in the structure of the command. The environment variables are set explicitly, suggesting the assistant is aware that the systemd service might have different defaults. The timeout wrapper (present in earlier tests but absent here) has been dropped, perhaps because the assistant expects the lock check to be fast. The command is a single SSH invocation, suggesting the assistant wants a clean, self-contained test.

The choice to test immediately after the permission fix, without checking whether a timer run was in progress, reveals a subtle thinking error: the assistant is focused on verifying the code change and doesn't consider that the production system might interfere. This is a common pattern in debugging — the developer becomes so focused on the specific bug that they forget the broader context.

Conclusion

The subject message captures a moment of apparent success that is actually a moment of continued struggle. The lock works — but it works so well that it blocks the very test meant to verify it. This is the paradox of building autonomous systems: every safeguard you add makes the system more reliable in production but harder to debug in development. The file lock, the verdict system, the conversation pruning — these are all mechanisms that increase operational stability at the cost of making the system more opaque and harder to test manually.

The message is a testament to the complexity of building reliable autonomous agents. Each fix reveals new edge cases. Each assumption is tested against reality. And sometimes, the system works exactly as designed — and that's precisely the problem.