The Lock That Saved the Agent: Deploying Concurrency Control in an Autonomous Fleet Manager

scp /tmp/czk/vast-manager-agent theuser@10.1.2.104:/tmp/vast-manager-agent && \
  scp /tmp/czk/cmd/vast-manager/agent/vast_agent.py theuser@10.1.2.104:/tmp/vast_agent.py && \
  ssh theuser@10.1.2.104 "
sudo systemctl stop vast-manager && sleep 1
sudo cp /tmp/vast-manager-agent /usr/local/bin/vast-manager
sudo cp /tmp/vast_agent.py /opt/vast-agent/vast_agent.py
sudo systemctl start vast-manager && sleep 2

# Test the lock and verdict system
echo '=== Test 1: Run agent ==='
export AGENT_LLM_BASE_URL=[REDACTED_API_ENDPOINT]
export AGENT_LLM_API_KEY=[REDACTED]
export AGENT_LLM_MODEL=qwen3.5-122b
export VAST_MANAGER_URL=http://127.0.0.1:1236
export AGENT_LOG_FILE=/tmp/vast-agent-test.log
timeout 60 python3 /opt/vast-agent/vast_agent.py 2>&1 | tail -15
" 2>&1
=== Test 1: Run agent ===
2026-03-17 20:48:24 [INFO] vast-agent starting (pid=639224)
2026-03-17 20:48:24 [INFO] Another agent instance is already running — exiting.

Two lines of log output. A 60-second timeout that never needed to expire. A deployment that, on its surface, looks trivial — copy binaries, restart a service, run a quick smoke test. But those two log lines represent the culmination of a deep debugging session into one of the most insidious problems in autonomous systems: the race condition that silently corrupts state.

Message [msg 4950] is the deployment message for a set of critical fixes to an LLM-driven fleet management agent. The agent, built to autonomously manage a cluster of GPU instances on vast.ai for Filecoin SNARK proving, had developed a debilitating problem: duplicate parallel runs, idle observations cluttering its context window, and no clean way to distinguish between a run that took meaningful action and one that merely observed the world and decided nothing needed to change. The user had reported these issues in [msg 4930], and the assistant spent the intervening messages ([msg 4931] through [msg 4949]) designing and implementing three interconnected fixes. Message [msg 4950] is the moment those fixes hit production.

Why This Message Was Written: The Context of Crisis

To understand why this deployment mattered, one must understand the fragility of the system it was protecting. The fleet management agent operated as a single-invocation Python script, triggered every five minutes by a systemd timer. But there was a second trigger path: a systemd path unit that would fire the agent immediately on urgent events like human messages or state changes. These two trigger mechanisms — the heartbeat timer and the event-driven path unit — could fire simultaneously, spawning two parallel agent processes that would each load the same conversation history, each observe the same fleet state, and each produce duplicate responses. The result was a conversation log polluted with redundant entries, an LLM context window bloated with duplicate observations, and an agent that appeared to be working twice as hard while actually accomplishing nothing extra.

The user's diagnosis in [msg 4930] was precise: "There seem to be duplicate calls to cron observe, so seems a lot of the time we call the agent in two parallel streams(?) and responses also duplicate? Should have some semaphore." The user also identified a second problem: idle observations — runs where the agent decided no action was needed — were accumulating in the conversation history with no clean way to remove them. And the user proposed the solution: have the agent return a structured JSON verdict that could be parsed programmatically to determine whether a run produced meaningful action.

The assistant's reasoning in [msg 4931] shows a clear understanding of the three issues and a methodical plan to fix them: a file lock to prevent parallel runs, a structured JSON verdict block appended to every LLM response, and logic to prune no-action runs from the conversation history. The thinking is explicit about why the verdict system matters: "The real problem is that even when the LLM runs due to notifications, it might still conclude no action was needed — and that response clutters the conversation history."

How the Fixes Were Designed and Implemented

The assistant implemented three changes across two codebases — a Go API server and a Python agent script — in a carefully orchestrated sequence.

The File Lock (Fix 1): The simplest and most critical fix was added to the main() function of the Python agent. Before any logic runs, the agent attempts to acquire a file lock. If another instance is already running, the script exits immediately with a log message. This prevents the timer and path unit from ever executing in parallel, regardless of how they are triggered. The lock file acts as a semaphore, ensuring exactly one agent process runs at a time.

The Structured JSON Verdict (Fix 2): The assistant modified the system prompt template to instruct the LLM to always end its response with a structured JSON block containing two boolean fields: action (whether the agent took any action) and state_changed (whether the fleet state meaningfully changed). This gives the Python wrapper a machine-parseable signal at the end of every LLM response, eliminating the need to heuristically determine whether a run was productive.

The Conversation Pruning Logic (Fix 3): After each agent run completes, the Python script parses the LLM's response for the JSON verdict block. If action is false and state_changed is false — meaning the agent observed the fleet and decided nothing needed to be done — the script deletes the messages from this run from the conversation history via a new DELETE endpoint on the Go API. This keeps the conversation history clean and prevents the LLM's context window from being polluted with idle observations.

The assistant also added a DELETE /api/agent/conversation/{id} route to the Go API server ([msg 4946]), fixing a compilation error in the process ([msg 4948]), and verified both the Go binary and Python script compiled successfully ([msg 4949]).

The Deployment and Its Significance

Message [msg 4950] executes the deployment. The assistant uses scp to copy the compiled Go binary and updated Python script to the management host, stops the vast-manager service, replaces the binaries, restarts the service, and then runs a test. The test is notable for what it checks and what it does not check.

The test runs the agent with a 60-second timeout and captures the output. The result is immediate: "Another agent instance is already running — exiting." This confirms the file lock is working — the test environment already had a lock file from a previous run (or the test itself created one and the second invocation detected it). But the test does not verify the verdict system or the conversation pruning logic. It does not check whether the DELETE endpoint works. It does not simulate a full agent cycle with the new system prompt.

This selective testing reveals the assistant's priorities: the file lock was the most critical fix because duplicate parallel runs were actively corrupting the conversation history and wasting LLM tokens. The verdict system and pruning logic, while important, are optimizations that could be verified in subsequent tests. The assistant chose to deploy and validate the highest-risk fix first, a pragmatic engineering decision.

Assumptions and Potential Blind Spots

The deployment makes several assumptions worth examining. First, it assumes the file lock is sufficient to prevent all race conditions — but what if the agent crashes without releasing the lock? The lock file would persist, preventing any future agent runs until manually cleared. Second, it assumes the management host is reachable and the service restart is clean — but the sleep 2 after starting the service is a fragile heuristic. Third, it assumes the LLM will reliably produce the structured JSON verdict — but LLMs are not deterministic, and a model that forgets to append the verdict block could break the pruning logic.

The assistant also did not test the verdict parsing logic in this deployment. The test only verified the file lock. If the verdict parsing had a bug — say, a regex that failed on edge cases or a JSON parser that threw exceptions on malformed output — it would not have been caught. The conversation pruning logic, which depends on the DELETE endpoint, was also not tested.

Input and Output Knowledge

To understand this message, one needs knowledge of: the systemd timer and path unit architecture that creates the parallel-trigger problem; the concept of file locking as a concurrency control mechanism; the structure of the LLM conversation history and how it is stored and retrieved; the Go HTTP API that manages agent state; and the Python agent's execution flow from main() through the LLM tool loop to message persistence.

The message produces several pieces of output knowledge: the file lock mechanism works and prevents duplicate runs; the deployment pipeline (scp, service stop, binary replacement, service start) is functional; the agent can be invoked in a test environment and produces expected log output; and the system is now ready for more comprehensive testing of the verdict and pruning features.

The Thinking Process Visible in the Message

The assistant's thinking is visible in the structure of the deployment command itself. The test command is carefully constructed: it exports all necessary environment variables, sets a timeout to prevent runaway processes, pipes output through tail -15 to capture only the most relevant lines, and runs the agent directly rather than through the systemd service. This reveals a methodical approach to verification — isolate the component under test, control for variables, and capture clean output.

The choice to test only the file lock, rather than running a full end-to-end test of all three fixes, reveals a prioritization mindset. The assistant could have written a complex test script that verified the verdict parsing, the DELETE endpoint, and the conversation pruning. Instead, it chose the simplest possible test that validated the most critical fix. This is the thinking of an engineer who understands that in production systems, the highest-risk failure mode must be addressed first, and that perfect verification can wait.

Conclusion

Message [msg 4950] is a deployment message that, on its surface, looks routine. But it represents the resolution of a class of bugs that plague autonomous systems: race conditions, state pollution, and the lack of structured feedback between an LLM and its orchestration layer. The file lock, the structured verdict, and the conversation pruning logic together form a coherent solution to the problem of running an LLM-driven agent on a timer in a world full of events. The deployment succeeded, the lock worked, and the agent was one step closer to reliable autonomy.