When a Lock Is Worth a Thousand Words: Testing Parallel Run Prevention in an Autonomous GPU Agent

The Message in Context

The subject message ([msg 4952]) is deceptively simple. It contains a single bash command that SSHes into a management host and executes the vast_agent.py script:

[assistant] [bash] ssh theuser@10.1.2.104 "
export AGENT_LLM_BASE_URL=[REDACTED]
export AGENT_LLM_API_KEY=[REDACTED]
export AGENT_LLM_MODEL=qwen3.5-122b
export VAST_MANAGER_URL=http://127.0.0.1:1236
export AGENT_LOG_FILE=/tmp/vast-agent-test.log
python3 /opt/vast-agent/vast_agent.py 2>&1
" 2>&1
2026-03-17 20:49:38 [INFO] vast-agent starting (pid=639307)
2026-03-17 20:49:38 [INFO] Another agent instance is already running — exiting.

The output spans just two log lines:

2026-03-17 20:49:38 [INFO] vast-agent starting (pid=639307)
2026-03-17 20:49:38 [INFO] Another agent instance is already running — exiting.

Yet this brief exchange sits at the intersection of three critical engineering challenges: preventing race conditions in autonomous systems, verifying that a fix actually works in production, and understanding the temporal dynamics of long-running LLM-driven agents. The message is a test — a manual probe to confirm that a newly implemented file-lock mechanism correctly prevents duplicate parallel agent invocations. And it succeeds. But the success itself reveals deeper truths about the system's behavior and the complexity of building reliable autonomous infrastructure management.

Why This Message Was Written

The immediate trigger was a production bug reported by the user in [msg 4930]: the agent was being invoked in duplicate parallel streams. The systemd timer (a heartbeat firing every 5 minutes) and the systemd path unit (triggered by urgent events like human messages or state changes) could fire simultaneously, causing two agent processes to run in parallel. Both would load the same conversation history, both would call the LLM, and both would append their observations and decisions to the conversation — creating duplicate entries, wasting tokens, and potentially causing conflicting actions.

The assistant diagnosed three root causes ([msg 4931]): the race condition between timer and path unit, the lack of a structured return value from the LLM to distinguish action-taking from idle observations, and the absence of a mechanism to prune no-op runs from the conversation history. The file lock was the first and most critical fix — a simple, brutal, effective solution to the parallelism problem.

The assistant implemented the lock by adding a file-based mutex at the top of the main() function in vast_agent.py (<msg id=4941-4942>). If another instance is already running (detected via fcntl.flock or similar OS-level file locking), the new process prints a message and exits immediately. No queuing, no coordination, no handoff — just a hard stop. This is appropriate for a system where the timer is a heartbeat and missing one cycle is acceptable because the next cycle will pick up any pending work.

After building and deploying the fix (<msg id=4949-4950>), the assistant ran a first test ([msg 4950]) that confirmed the lock worked: the manual invocation found an existing agent instance and exited. But the assistant wanted to verify the system more thoroughly. In [msg 4951], it waited 30 seconds and tried again — but the grep-filtered output was empty, suggesting the agent run had completed (or the grep pattern missed the relevant lines). Now, in [msg 4952], the assistant runs the test once more, and again the lock blocks the new invocation.

What the Output Reveals

The two log lines tell a story that goes beyond "the lock works." The timestamp 20:49:38 is significant. The previous test in [msg 4950] ran at approximately 20:48:24. The test in [msg 4951] ran after a 30-second sleep (so around 20:48:54). Now at 20:49:38, roughly 74 seconds after the first test, the lock is still held by another agent instance.

This reveals something important about the agent's runtime characteristics: a single agent invocation can take well over a minute. The agent loads conversation history, calls the LLM (which may take 30-60 seconds for a model like qwen3.5-122b), processes tool calls, and persists results. During this entire window, the lock is held, and any concurrent invocation — whether from the systemd timer, the path unit, or a manual test — is rejected.

This is the correct behavior for the race condition fix. But it also introduces a subtle operational consideration: if the agent takes, say, 3 minutes to complete a cycle, and the systemd timer fires every 5 minutes, there is a 2-minute window where no agent runs. That's fine for a heartbeat-based system. But if the agent takes 6 minutes (possible with slow LLM responses or many tool calls), the timer would fire while the agent is still running, and that timer-triggered run would be silently dropped. The next run would happen 5 minutes later — meaning the system could go up to 10 minutes without a successful observation cycle.

This is not necessarily a problem — the agent is designed to be event-driven as well, with the path unit triggering on urgent events. But it's a design tension worth noting: the lock prevents duplicate work at the cost of potentially skipping scheduled cycles.

Assumptions and Potential Issues

The file lock approach makes several assumptions that deserve scrutiny:

Assumption 1: Lock cleanup is reliable. If the agent crashes (SIGKILL, OOM, segfault), the operating system will release the file lock automatically when the process dies. This is true for fcntl.flock on Linux — locks are associated with the file descriptor and are released when the process terminates. However, if the agent is killed by a signal that doesn't clean up file descriptors (unlikely but possible with certain kernel configurations), the lock could persist. A stale lock would block all future agent runs until manually cleared, effectively killing the autonomous management system.

Assumption 2: Skipping a timer cycle is acceptable. The systemd timer fires every 5 minutes as a heartbeat. If a run is skipped because the lock is held, the next run in 5 minutes will handle any pending work. This is reasonable for a system that manages GPU instances with startup times of 1-2 hours — a 5-minute delay in scaling decisions is negligible. However, for urgent events (like a detected OOM or a human alert), the path unit provides a separate trigger path that could also be blocked by the lock.

Assumption 3: The lock file path is consistent. Both the timer and path unit invocations must use the same lock file path. If there's any discrepancy (e.g., different working directories, different environment variables), the lock would be ineffective. The assistant's implementation uses a fixed path like /tmp/vast-agent.lock, which is consistent across invocations.

Assumption 4: No other coordination mechanism is needed. The file lock is a mutual exclusion mechanism — it prevents concurrent execution but doesn't provide any other guarantees. There's no queue, no priority, no way for an urgent event to interrupt a running observation cycle. If the agent is in the middle of a routine 5-minute check when a critical alert arrives, the alert-triggered run will be blocked until the routine check completes. This could delay response to urgent situations by minutes.

The Broader Engineering Context

This message is one step in a much larger effort to build a reliable autonomous agent for GPU cluster management. The file lock is just one component of a multi-faceted solution that also includes:

• Structured JSON verdicts ([msg 4943]): The LLM is now prompted to return a JSON block at the end of its response containing {"action": bool, "state_changed": bool}. This allows the Python wrapper to cleanly determine whether the agent took any meaningful action, and if not, prune the run's messages from the conversation history.
• Context pruning ([msg 4944]): No-op runs are removed from the conversation to prevent context bloat. This keeps the LLM's prompt within the token budget and prevents the agent from getting confused by its own idle observations.
• Debouncing (developed in subsequent chunks): Event-driven triggers are debounced to prevent bursty state-change events from spawning multiple rapid agent invocations. The file lock is the simplest and most foundational of these fixes. Without it, the other improvements would still be vulnerable to race conditions — two parallel agents could both decide to prune the same messages, or both could attempt conflicting scaling actions.

Conclusion

Message [msg 4952] appears to be a routine test — a developer verifying that a fix works. But it's also a window into the challenges of building autonomous systems that must be both reactive and reliable. The file lock mechanism it tests is a blunt instrument: it prevents parallel execution by fiat, with no nuance or prioritization. But for a system where the cost of duplicate runs is wasted tokens, polluted context, and potentially conflicting actions, blunt is exactly what's needed.

The two log lines of output confirm that the lock works. But they also reveal that the agent takes significant time to complete a cycle — time during which the system is effectively single-threaded for agent operations. This is a design trade-off that the assistant and user will need to monitor as the fleet grows and the agent's responsibilities expand. For now, the lock is a solid foundation. The next challenges — debouncing, context management, and diagnostic grounding — will build on it.