From Silent Crashes to Autonomous Operations: Building an LLM-Driven Fleet Management Agent for GPU Proving Infrastructure

Introduction

In the high-stakes world of distributed GPU proving infrastructure, where machines scattered across the globe synthesize cryptographic proofs for Filecoin's storage verification network, a single production outage can cascade into hours of lost capacity and delayed proofs. This article chronicles a remarkable engineering journey that began with a silent, fleet-wide crash and culminated in a fully autonomous LLM-driven fleet management agent—a system capable of observing GPU cluster demand, making scaling decisions, and launching cloud instances without human intervention.

The narrative spans a critical pivot in an opencode coding session: from reactively debugging why four out of six GPU nodes had silently died, to proactively building an autonomous agent powered by the qwen3.5-122b language model. Along the way, the assistant and user iteratively refined the agent through multiple production incidents—over-provisioning, accidentally stopping all instances due to a flawed demand signal, context overflow bugs, and race conditions—ultimately establishing a robust foundation for autonomous, cost-effective cluster management.

Part I: The Silent Crash and the Discovery That Changed Everything

The crisis began when multiple nodes running the cuzk proving daemon simply vanished. No panics, no CUDA errors, no OOM kills in the logs—the processes had been erased from existence mid-operation, their log files ending mid-sentence. The supervisor loop, a bash script in entrypoint.sh designed to automatically restart cuzk on failure, had failed catastrophically. The root cause was a subtle but devastating bug in bash 5.2's wait -n builtin: when a child process exits and is reaped by the kernel before wait -n is called, bash can lose track of the exit status, causing wait -n to block forever on the remaining children. The supervisor was effectively catatonic—alive but useless.

The assistant diagnosed this bug and implemented a fix, replacing the unreliable wait -n with a robust polling loop using kill -0 to check process liveness at 5-second intervals. A new Docker image was built and pushed. But the deeper question remained: why were the processes dying in the first place?

The answer came from the user, who posed a brilliant hypothesis: could vast.ai be enforcing a separate memory limit via a host-side watchdog, distinct from the cgroup limits visible inside the container? The assistant confirmed this by querying the vast.ai API and discovering the mem_limit field—a value that differed significantly from both physical RAM and cgroup limits on every machine. On the 2 TB machines, cgroups reported approximately 1032 GB, but vast.ai's mem_limit was only 1009 GB. The cgroup was set higher than the actual enforced limit. The daemon's detect_system_memory() function, which read the cgroup to determine the memory budget, was using an inflated value. The daemon would allocate memory up to ~95% of the cgroup limit, inadvertently exceeding vast.ai's real cap, and the host-side watchdog would deliver a silent SIGKILL.

This discovery was the catalyst for everything that followed. The user recognized that reactive debugging was insufficient—the infrastructure needed intelligent, automated management that could anticipate constraints rather than merely respond to failures. The directive was clear: build an autonomous agent to manage the fleet, scale it based on Curio SNARK demand, and alert humans only when necessary.

Part II: Architecting an Autonomous Agent from Scratch

The assistant responded to this directive with a remarkable burst of engineering. Before writing a single line of new code, the assistant conducted systematic reconnaissance: reading the existing vast-manager codebase, exploring the Curio PostgreSQL database schema, and evaluating the qwen3.5-122b model's tool-calling capabilities through a synthetic subagent assessment that passed all six tests.

The architecture that emerged was a clean separation of concerns. A comprehensive Go API (agent_api.go, 1,357 lines) was built to expose 12 endpoints covering demand monitoring, fleet status, instance lifecycle management (with safety guards against excessive concurrent launches), alerting, and per-machine performance tracking. A Python autonomous agent (vast_agent.py, 697 lines) was created to run on a 5-minute systemd timer, using the LLM to make scaling decisions by calling the Go API.

The assistant dispatched two parallel subagent tasks to construct these components simultaneously—a deliberate architectural choice that reflected a clear understanding of the dependency graph. The Go file defined endpoints that the Python agent would later call, but both could be written concurrently against a shared interface contract. The Go/Python split was pragmatic: Go excelled at robust HTTP servers with connection pooling and type-safe API definitions, while Python offered superior ergonomics for LLM interaction and rapid prompt iteration without recompilation.

The system was deployed to the management host at 10.1.2.104, and the first end-to-end test revealed a routing bug—the agent hit a 404 when calling /api/agent/offers because the Go API had registered the endpoint at /api/offers. The assistant fixed the missing route, rebuilt, redeployed, and re-ran the test. This time, the agent successfully observed 8 pending PSProve tasks, identified an RTX 5090 offer at $0.361/hr with 0.985 reliability, and launched instance 33007738. The first autonomous instance launch was a milestone.

Part III: The Pivot to Simplicity—Learning to Think in Hours, Not Minutes

The celebration was short-lived. The user's very next message delivered a devastating piece of operational feedback: "Note on rates, be careful—starting an instance can take hours. Pending tasks can be really volatile and are NOT a useful metric, PSProve takes 4 minutes and each machine churns out one every 30s pipelined."

This single message shattered the assumption that the agent's logic was sound. The assistant's reasoning reveals the dawning realization: those 8 pending tasks would be cleared in approximately 4 minutes by the existing worker, while the newly launched instance would take 1-2 hours to start contributing. The agent had just wasted money launching capacity that would arrive long after the demand had evaporated.

The user clarified the correct approach in a follow-up message: "The agent should ideally: scale down the cluster when there is no activity for 1h+, scale up to 500proofs/h as target." When the assistant began designing complex auto-targeting logic involving PI controllers and exponential moving averages, the user immediately corrected course: "There should be no complex auto targets." The assistant internalized this with refreshing clarity: "Right—keep it simple. The agent is a 122B model, not a control system."

The redesigned decision framework was distilled to three simple rules:

Part IV: Building Trust Through Production Data

The user's next refinement introduced the concept of provenance into the agent's decision-making. The directive was precise: "Agent should prefer instances which we saw working, especially ones that we've seen perform in curio. Agent should, on cron, check on instances—proof/error counts in curio, write to a ~100 line .md file with instance summary that is used to weigh whether the instance is good."

This was a shift from predictive trust (what a machine might do based on benchmarks) to empirical trust (what a machine has actually done in production). The assistant built a /api/agent/perf endpoint that queried Curio's harmony_task_history for per-machine completion and error counts, and updated the Python agent to maintain a markdown file summarizing instance performance. The LLM would read this file before making launch decisions, preferring machines with proven track records over unknown gambles.

The assistant also recognized a critical blind spot: the agent's fleet summary showed current capacity but ignored instances that were still loading. With 1 running instance producing 40 proofs/hour and 2 instances loading, the agent saw a gap of 460 proofs/hour against the 500 target and might launch more instances unnecessarily. Rather than adding complex rules, the assistant improved the data—the fleet summary was updated to explicitly state: "1 running, 2 loading (will add capacity in 1-2h)." This gave the LLM the information it needed to make forward-looking decisions without imposing rigid logic.

Part V: Hardening the Loop—Rate Limits, Context Management, and Operational Stability

As the agent began operating autonomously, real-world friction emerged. The most instructive bug was the self-reinforcing rate limit. The rate limiter counted ALL launch actions—including rejected ones—within the 15-minute window. When the agent hit the limit, its rejected attempts were recorded as actions, which kept the count above the threshold, causing further rejections, which were recorded as more actions. The agent was permanently locked out by its own guard mechanism. The fix was a one-line SQL change: filter on result = 'success' rather than counting all attempts.

The agent's operational loop was further hardened through multiple iterations. A file lock was added to prevent parallel invocations when the systemd timer and event-driven path unit fired simultaneously. A structured verdict system was implemented, where the LLM's output included a <verdict> block indicating whether it took action, allowing no-op runs to be pruned from the prompt context while remaining visible in the UI. Duplicate intermediate assistant messages were suppressed, keeping only the final response per run in the context window.

When a session reset broke the agent entirely—it stopped responding to both the cron timer and manual triggers—the assistant diagnosed the root causes: stale "pending wakes" accumulating without being marked processed, causing the trigger file to be touched repeatedly and spawning bursts of redundant event-driven runs. The fix included a debounce mechanism in the Go triggerAgent() function to suppress burst file modifications, and making the run_id monotonic via persistent session state rather than fragile conversation history.

Conclusion: The Architecture of Autonomous Infrastructure

The journey from silent crashes to autonomous operations spanned a remarkable arc of engineering. What began as a debugging session into bash's wait -n bug and vast.ai's hidden memory enforcement evolved into the design, construction, and iterative hardening of a fully autonomous LLM-driven fleet management agent.

Several principles emerge from this narrative. First, domain knowledge is not optional—the assistant's technically sophisticated agent initially made poor decisions because it lacked understanding of the workload's temporal dynamics. Second, improve the data, not the rules—when the agent might over-provision due to invisible loading instances, the fix was to make those instances visible in the fleet summary, not to add complex control logic. Third, safety mechanisms must be reflexive-aware—the self-reinforcing rate limit demonstrated that guardrails counting their own failures create feedback loops that permanently paralyze the system.

The final system that emerged from this session was not a brittle script with hard-coded thresholds, but a genuinely autonomous agent capable of observing fleet state, reasoning about demand, making scaling decisions, and learning from production performance data. It was a system that had earned the trust to spend real money on GPU instances, and had been hardened through real production incidents to handle the edge cases that only operational experience reveals.

In the end, the most valuable outcome was not the agent itself, but the design philosophy it embodied: trust the LLM, but give it complete, well-structured data; keep the rules simple and the perception rich; and when something breaks, improve the information flow rather than adding more rules. This philosophy, forged through production fire, is the lasting legacy of this session.