From Crash to Autonomy: Building an LLM-Driven GPU Fleet Manager

Introduction

In the span of a single coding session, a production crisis catalyzed a strategic transformation. What began as a forensic investigation into silent GPU daemon crashes on vast.ai evolved into the design, construction, and deployment of a fully autonomous LLM-driven fleet management agent — a system capable of observing proving demand, acquiring and releasing GPU instances, alerting human operators, and learning from operational feedback without direct human intervention. This article traces that arc: from the discovery of a host-side memory watchdog that was silently terminating processes, through the specification and implementation of a Python-based autonomous agent, to the critical user feedback that reshaped the agent's decision-making logic from fragile threshold-based rules to robust capacity-oriented heuristics.

The chunk examined here represents the foundational layer of what would become an increasingly sophisticated autonomous system — one that would later grow to include diagnostic sub-agents, persistent conversational context, event-driven triggering, and hardened state management. But in this chunk, the core architecture was born, deployed, and immediately tested against the unforgiving realities of production GPU proving infrastructure.

The Catalyst: A Silent Killer in the GPU Fleet

The session did not begin with grand plans for automation. It began with a mystery: multiple GPU nodes running the cuzk (CUDA ZK prover) daemon were crashing silently, with no panics, no CUDA errors, no OOM kills in the logs. The daemon simply vanished, leaving only the curio process running. Across the fleet, four of six running nodes were effectively dead — still registered as instances but producing no proofs.

The user had a theory: vast.ai enforces a separate mem_limit via a host-side watchdog, distinct from cgroups, that terminates processes exceeding memory limits without leaving standard OOM signatures. The assistant confirmed this theory through deep investigation, establishing that vast.ai's infrastructure imposes a hard memory cap enforced by a mechanism outside the container's visibility. This explained the silent terminations — the processes were being killed by the host, not by anything the software could detect or handle.

This discovery was the turning point. The user recognized that reactive debugging — fixing one crash at a time, manually restarting daemons, adjusting memory limits — was a losing battle against an environment where instances could be terminated at any moment by forces outside their control. The only sustainable solution was proactive automation: a system that could continuously monitor the fleet, detect failures, acquire new instances, and scale capacity to meet demand without waiting for a human to notice something was wrong.

The Vision: An Autonomous Fleet Manager

The user's directive was clear and ambitious: build a fully autonomous agent to manage the GPU fleet, scale it based on Curio SNARK demand, and alert humans only when necessary. The agent would need to:

1. Continuously monitor proving demand across different proof types (WindowPoSt, WinningPoSt, SnapDeals)
2. Maintain a fleet of GPU instances on vast.ai sufficient to handle the workload
3. Scale up when demand increased and scale down when it subsided
4. Respect hard constraints on budget and instance count
5. Alert human operators when situations exceeded its capacity to handle
6. Run unattended on a timer, making decisions without human intervention This was not a hypothetical design exercise. The agent would manage real GPU infrastructure for Filecoin SNARK proving, where time-sensitive proof types like WindowPoSt and WinningPoSt have blockchain-imposed deadlines. A missed proof could mean financial penalties. The agent had to be reliable, cost-effective, and safe.

Research and Foundation: Choosing the Tools

Before writing any code, the assistant researched state-of-the-art agent API patterns and assessed the available LLM models. The qwen3.5-122b model was selected as the agent's reasoning engine — a 122-billion-parameter model that passed all tool-calling tests and was available via an OpenAI-compatible endpoint. This choice was critical: the agent's decision-making quality would depend directly on the LLM's ability to understand the operational situation, follow policy rules, and call the right tools at the right times.

The assistant then built a comprehensive Go API layer for the vast-manager, creating agent_api.go with 12 endpoints covering the full lifecycle of fleet management:

• Demand monitoring: Endpoints to query current proving demand, queue depths, and pipeline status
• Fleet status: Endpoints to report running instances, loading instances, and per-machine health
• Instance lifecycle: Endpoints to launch new instances (with safety guards) and stop running ones
• Alerting: Endpoints to send structured alerts with severity levels (info, warning, critical)
• Performance tracking: Endpoints to record and query per-machine proving performance This API layer served as the interface between the agent's decision-making (which would happen in Python, calling an LLM) and the operational reality of vast.ai instance management. It abstracted away the complexity of the vast.ai API, rate limiting, and error handling into a clean, agent-friendly surface.

The Agent Script: From Specification to Implementation

With the API layer in place, the assistant turned to the agent itself. The user provided an extraordinarily detailed specification for a Python script called vast_agent.py — a pure Python 3 implementation with no external dependencies beyond the requests library. The specification covered every aspect of the agent's architecture:

• Configuration loading from environment variables and a config file
• Observation phase: Three API calls to gather demand, fleet status, and configuration
• Fast-path optimization: Skip the LLM call entirely when there is no pending demand and the fleet is healthy
• LLM decision loop: Up to five iterations of tool-calling, with each tool result fed back into the conversation
• Seven tool definitions in OpenAI function-calling format: get_demand, get_fleet, get_offers, launch_instance, stop_instance, check_health, and send_alert
• Error handling: Three-tier strategy for LLM failures, API failures, and tool execution errors
• Comprehensive logging of every API call, LLM interaction, and tool execution The assistant approached the implementation methodically. First, it explored the existing directory structure ([msg 1]), confirming the presence of main.go and ui.html — the Go backend and web UI of the vast-manager project. This reconnaissance was essential: the agent script needed to integrate into an existing codebase with established API endpoints and a running Go server. Next, it created the agent/ subdirectory ([msg 2]), preparing the filesystem for the new component. The choice of mkdir -p was a small but telling detail — a defensive, production-oriented habit that ensures the operation succeeds regardless of the current state of the filesystem. Then came the moment of creation ([msg 3]): a single write tool call that produced a 697-line Python file implementing the complete specification. The file was structured into eight sections, each with a clear purpose: | Section | Lines | Purpose | |---|---|---| | Config loading | ~40 | Reads env file + env vars with defaults | | Logging | ~25 | Dual output: stdout + file, timestamped | | API helpers | ~35 | call_api() for vast-manager, send_alert_direct() for fallback | | Tool definitions | ~130 | 7 tools in OpenAI function-calling format | | Tool execution | ~65 | execute_tool() dispatches by name | | LLM interaction | ~50 | call_llm_chat() and build_system_prompt() | | Agent core loop | ~120 | run_agent() — observe, quick-check, LLM loop | | Entry point | ~30 | main() with crash protection and alert-on-failure | The assistant immediately verified the file's syntactic correctness ([msg 4]) using Python's ast.parse — a surgical check that validates structure without executing any code. The output: "Syntax OK." A subsequent wc -l ([msg 5]) confirmed the file was 697 lines — a substantial artifact by any measure.

The Architecture: How the Agent Makes Decisions

The agent's decision-making architecture is worth examining in detail, as it embodies several design principles that would prove critical in production.

The fast path is the most important optimization. Before ever calling the LLM, the agent checks whether there is any pending demand and whether the fleet meets minimum health criteria. If everything is stable, it logs a brief message and exits — no LLM call, no token consumption, no latency. This is a cost-saving measure that recognizes the most common state of a well-provisioned fleet is "nothing needs to happen." In a system that runs every five minutes, this fast path saves significant API costs over time.

The LLM loop allows up to five iterations of tool-calling. Each iteration feeds the results of previous tool calls back into the conversation, enabling multi-step reasoning. The agent can observe demand, decide to launch an instance, check its health, and then decide whether to launch another — all within a single invocation. The five-iteration limit prevents runaway loops while allowing sufficient depth for complex scenarios.

The error handling strategy is layered and resilient. LLM API failures trigger a direct alert via the vast-manager API, bypassing the LLM entirely. Vast-manager API failures cause a graceful abort — the agent logs the error and exits, relying on the next cron cycle to retry. Tool execution errors are returned to the LLM for it to handle, giving the model a chance to recover from individual failures without aborting the entire cycle.

The tool definitions are the agent's interface to the world. Each tool has an explicit, verbose description optimized for the 122B model — telling it not just what the tool does, but when to use it and what to expect. The get_demand tool, for example, instructs the model to "call this to see queue depths, pipeline status, and throughput" and to "use this to decide if scaling is needed." This level of detail is essential for guiding the LLM toward correct tool selection.

Deployment and First Flight

The agent was deployed to the management host and configured to run via a systemd timer every five minutes. The first autonomous launch was a milestone: the agent, observing demand and finding insufficient capacity, called get_offers, selected a suitable GPU instance, and called launch_instance — all without human intervention. The system worked.

But the user's immediate feedback was critical and reshaped the agent's entire approach. The problem was the demand signal: pending task counts, which the agent used as its primary scaling trigger, are highly volatile in a proving system. A spike in pending tasks could vanish within seconds as workers pick them up, or it could persist for hours if all workers were busy. Moreover, launching a GPU instance on vast.ai takes time — sometimes hours for certain GPU types to become available. Using a volatile short-term signal to drive a decision with hours-long consequences was fundamentally wrong.

The user's insight was elegant: instead of scaling based on pending task counts, scale based on capacity targets. The agent should aim to maintain a certain number of proofs per hour of proving capacity. When demand is active (any pending tasks exist), the agent should scale up to meet the target. When demand has been absent for an extended period (an hour of inactivity), scale down. And when choosing which instances to launch, prefer machines with a proven track record — stored in a performance markdown file that records historical throughput.

This redesign transformed the agent from a reactive threshold-follower into a proactive capacity manager. The final iteration successfully demonstrated the agent observing fleet state, understanding loading instances (instances that are booting but not yet proving), respecting vast.ai rate limits, and making sound scaling decisions. It established a solid foundation for autonomous, cost-effective cluster management.

The Themes: What This Chunk Reveals

Several themes emerge from this chunk that are worth highlighting.

The pivot from reactive to proactive operations is the dominant narrative arc. The session began with the team fighting fires — crashed daemons, silent terminations, manual restarts. It ended with an autonomous system designed to prevent those fires from mattering. This is a classic pattern in infrastructure engineering: the moment when a team decides that fixing individual failures is less important than building a system that handles failures automatically.

The importance of domain-specific signal design is another critical theme. The initial agent used pending task counts as its scaling signal — a natural choice for anyone familiar with queue-based systems. But the user's operational experience revealed that this signal was misleading in the context of GPU proving, where task lifetimes are measured in seconds and instance startup times are measured in hours. The redesign to capacity-based scaling (proofs per hour) was a direct response to this domain knowledge. It is a reminder that autonomous agents are only as good as the signals they observe.

The value of methodical implementation is visible in the assistant's approach. Before writing code, it explored the directory structure. Before creating the file, it created the directory. After writing, it verified syntax. Each step built on the previous one, creating a chain of confidence. This discipline — reconnaissance, preparation, creation, verification — is the same pattern that underlies reliable software engineering in any context.

The human-in-the-loop feedback cycle was essential to the agent's success. The agent was deployed, observed, critiqued, and redesigned within a single session. The user's operational expertise — understanding that pending task counts are volatile, that instance startup takes hours, that proven machines are preferable — was injected directly into the agent's logic. This tight feedback loop between human operator and autonomous system is a model for how LLM-driven agents should be developed: deploy early, observe behavior, and iterate based on real-world performance.

Conclusion

This chunk captures a pivotal moment in the evolution of a GPU proving infrastructure: the transition from manual fleet management to autonomous operation. What began as a forensic investigation into silent daemon crashes became the foundation for an LLM-driven agent that could observe, decide, and act without human intervention. The agent's architecture — fast-path optimization, multi-iteration LLM loop, layered error handling, explicit tool definitions — was designed for reliability and cost-effectiveness from the start.

But the most important lesson from this chunk is that the agent's initial design, while technically sound, was immediately improved by operational feedback. The shift from threshold-based scaling to capacity-based scaling was not a bug fix — it was a fundamental redesign driven by domain expertise. The agent that emerged from this feedback loop was not just a script that called an LLM; it was a system that embodied the user's operational knowledge about how GPU proving actually works.

The foundation laid in this chunk would support increasingly sophisticated capabilities in subsequent sessions: diagnostic sub-agents, persistent conversational context, event-driven triggering, and hardened state management. But the core architecture — observe, decide, act, learn — was established here, in the crucible of production failure and rapid iteration.