The File Lock That Saved the Fleet: Preventing Duplicate Agent Runs in Autonomous GPU Cluster Management

Message at a Glance

[assistant] [edit] /tmp/czk/cmd/vast-manager/agent/vast_agent.py
Edit applied successfully.

This four-line message — a tool call result confirming that a file edit was applied to vast_agent.py — appears deceptively simple. In reality, it represents the first of three critical fixes deployed to stabilize an autonomous LLM-driven fleet management agent that was suffering from a race condition so severe it threatened the reliability of the entire GPU proving infrastructure. The edit added a file lock to the agent's main() function, solving a problem where the systemd timer and systemd path unit were triggering duplicate parallel agent runs, causing duplicate observations, duplicate responses, and a gradual corruption of the agent's conversational context.

The Crisis: Duplicate Parallel Agent Runs

To understand why this tiny edit mattered, one must understand the architecture of the autonomous fleet agent. The agent ran as a Python script (vast_agent.py) invoked by two separate systemd mechanisms: a timer that triggered every 5 minutes for routine observation cycles, and a path unit that triggered immediately on urgent events such as human messages or instance state changes. These two mechanisms were designed to be complementary — the timer provided heartbeat reliability, the path unit provided responsiveness.

The problem, as the user reported in [msg 4930], was that these two mechanisms frequently collided: "There seem to be duplicate calls to cron observe, so seems a lot of the time we call the agent in two parallel streams(?) and responses also duplicate." When a state-change event occurred just before the 5-minute timer was due to fire, both triggers would launch the Python script simultaneously. Two instances of the agent would then load the same conversation history, make the same observations, call the same tools, and append duplicate responses to the conversation database. This not only wasted LLM tokens and API calls but, more insidiously, polluted the agent's persistent memory with redundant content that gradually pushed the context window toward overflow.

The Reasoning: Why a File Lock Was the Right Choice

The assistant's thinking, visible in [msg 4931], shows a clear diagnostic process. The assistant identified three distinct issues from the user's report: (1) duplicate parallel agent runs from timer/path unit collisions, (2) idle no-action observations not being cleaned up from the conversation history, and (3) the need for a structured JSON return from the LLM to enable programmatic pruning of no-op runs.

For issue #1, the assistant considered several approaches. A semaphore via the vast-manager API would have been architecturally "cleaner" but would introduce network dependency and latency. A PID file check would be fragile across crashes. The assistant settled on a file lock — a simple, operating-system-level mutual exclusion mechanism that would work regardless of how the script was invoked. The reasoning was pragmatic: "I'll use a file lock at the start of the Python script to prevent parallel runs—if another instance is already running, the script exits immediately, which stops both the timer and path unit from triggering simultaneously."

The file lock approach had several virtues. It was simple to implement (a few lines of Python using the fcntl module or a lock file pattern). It required no changes to the systemd configuration. It was race-condition-free at the OS level — the kernel guarantees atomicity of file locking. And it was self-cleaning: if a process crashed while holding the lock, the kernel would release it automatically.

The Implementation: What the Edit Actually Changed

The edit applied in [msg 4942] modified the main() function of vast_agent.py to acquire an exclusive file lock at startup. The exact diff is not visible in the message itself (the tool call content was sent in the preceding message [msg 4941]), but the pattern is standard: at the very beginning of main(), before loading configuration or connecting to any services, the script attempts to open a lock file (typically /tmp/vast_agent.lock or similar) and acquire an exclusive lock. If the lock cannot be acquired because another instance is already running, the script logs a warning and exits with code 0, avoiding any disruptive failure.

This placement — at the very start of main() — was deliberate. The lock needed to be acquired before any work began, to prevent even partial duplicate execution. It also needed to be released when the script exited, which Python's fcntl.flock handles automatically when the file descriptor is closed or the process terminates.

Assumptions and Their Validity

The assistant made several assumptions in choosing this approach. First, that the lock file would reside on a filesystem visible to both triggering mechanisms — a safe assumption since both the timer and path unit run on the same host. Second, that the lock acquisition would be fast enough not to delay the timer-missed path unit trigger — also safe, since file locking is a kernel operation completing in microseconds. Third, that a crashed agent would release the lock — guaranteed by POSIX semantics for flock.

One subtle assumption was that the file lock alone would be sufficient to prevent all duplicate runs. In practice, the lock prevents simultaneous execution but does not prevent sequential duplicate runs where the timer fires, the agent runs and finishes, and then the path unit fires moments later for the same event. However, this was an acceptable gap — sequential duplicates are far less harmful than parallel ones, and the structured verdict system (changes #2 and #3, applied in subsequent messages) would handle the cleanup of redundant observations.

Input Knowledge Required

To understand this message, one needs knowledge of: the systemd timer/path unit architecture for triggering periodic and event-driven tasks; the Python file locking API (fcntl.flock, fcntl.LOCK_EX); the agent's conversational architecture where each run loads and appends to a persistent conversation history; and the specific race condition where two independent scheduling mechanisms can collide. One also needs to understand the broader context of the autonomous fleet agent — that it manages GPU instances on vast.ai for Filecoin SNARK proving, that instance startup takes 1–2 hours, and that the agent had previously caused a catastrophic incident where it destroyed 8 healthy instances due to speculative reasoning (the very problem that led to the diagnostic grounding system built earlier in this chunk).

Output Knowledge Created

This message created a production-hardened agent entry point that prevents duplicate parallel execution. The file lock became a foundational reliability layer upon which the other two fixes (structured verdict parsing and no-op run pruning) could be built. It also established a pattern for future agent hardening: simple OS-level primitives (file locks, kill -0 polling, file-based triggers) proved more reliable than complex distributed coordination for this single-host deployment.

The Broader Significance

The file lock fix, while individually small, was part of a larger architectural maturation of the autonomous agent. Earlier in this chunk ([msg 4928]), the assistant had built a diagnostic sub-agent system to prevent the agent from making destructive decisions based on speculation — gating the stop_instance tool with an HTTP 428 precondition that forced the agent to ground its decisions in empirical evidence from SSH log collection and LLM-based analysis. Now, in [msg 4942], the assistant was hardening the agent's operational loop against race conditions. The two fixes addressed complementary failure modes: the diagnostic system prevented bad decisions, the file lock prevented bad execution.

This pattern — first build the decision guardrails, then harden the execution infrastructure — reflects a mature approach to autonomous system design. The diagnostic system ensured the agent would not kill healthy instances even if it ran; the file lock ensured the agent would not run twice simultaneously even if triggered. Together, they transformed the agent from a fragile prototype prone to catastrophic errors into a robust production system capable of unsupervised fleet management.

Conclusion

Message [msg 4942] is a testament to the fact that in production autonomous systems, the most impactful fixes are often the simplest. A four-line file lock, added to the main() function of a Python script, eliminated an entire class of race-condition bugs that were silently degrading the agent's reliability. It required no new dependencies, no architectural changes, and no coordination with other services — just a clear understanding of the problem, a pragmatic choice of solution, and the discipline to place the lock at the exact right point in the execution flow. In the high-stakes world of GPU cluster management, where a single agent mistake can cost thousands of dollars in wasted compute or lost proving capacity, such simplicity is not a compromise — it is the highest form of engineering sophistication.