From Production Crashes to Autonomous Fleet Management: The Evolution of an LLM-Driven GPU Proving Infrastructure

Introduction

In the span of a single intense coding session, a GPU proving infrastructure for Filecoin's storage verification network underwent a transformation that illustrates the deepest challenges in operating distributed systems at the frontier of autonomous AI assistance. What began as a methodical deployment of a budget-integrated pinned memory pool—the culmination of weeks of careful engineering—spiraled into a production crash that revealed a silent supervisor bug, and then pivoted dramatically into the design, construction, and iterative hardening of a fully autonomous LLM-driven fleet management agent.

This segment, spanning roughly 700 messages across the opencode conversation, captures one of the most remarkable arcs in the entire session: from reactive debugging to proactive automation, from ephemeral cron scripts to a persistent conversational agent with genuine cross-run awareness, and from a catastrophic failure where the agent destroyed its own fleet to a hardened production system grounded in diagnostic evidence, structured verdicts, and event-driven reliability. The agent, powered by the qwen3.5-122b language model, was iteratively refined through multiple production incidents—over-provisioning, accidentally stopping all instances due to a flawed demand signal, context overflow bugs, session resets that paralyzed the system, and race conditions that spawned duplicate runs. Each incident forged a deeper understanding of what it takes to build autonomous systems that can be trusted with real-world consequences.

Part I: The Crash That Changed Everything

The segment opened with the assistant at a natural stopping point. The budget-integrated pinned memory pool—a redesign that made CUDA pinned memory allocations visible to the system's memory budget manager—had been designed, implemented, unit-tested, deployed to a high-memory RTX 5090 test node, and validated with five successful proofs. The natural next step was to commit the six modified files to version control and move on.

But the user intervened with a critical redirection: the budget-integrated pool must be tested on a constrained memory machine before any code was committed. This was not a minor procedural suggestion—it was a fundamental reordering of priorities grounded in hard-won operational experience. The entire purpose of the budget-integrated redesign was to prevent out-of-memory (OOM) crashes on memory-constrained nodes. Testing only on a 755 GiB RTX 5090 machine would never exercise the budget-limiting path that was the core of the fix.

The assistant pivoted immediately, building a new Docker image, pushing it to Docker Hub, and provisioning a test instance on vast.ai. The provisioning process itself was a microcosm of the infrastructure challenges that defined the session: navigating API permission boundaries, reading source code to understand deployment contracts, and interpreting system telemetry to distinguish between "slow" and "stuck." The test instance, an RTX 5060 Ti with approximately 342 GiB of cgroup-limited memory, was finally coming online.

Then the production crash hit.

Multiple nodes across the fleet reported that the cuzk daemon had crashed with no automatic recovery. The diagnostic investigation revealed a fundamental reliability bug in the supervisor loop within entrypoint.sh. The script used wait -n to monitor the two daemon processes and restart them if they exited. But wait -n was blocking indefinitely in a do_wait syscall even after the cuzk process had fully exited. The supervisor thought the process was still running when it was actually dead, leaving nodes running only the curio component while cuzk was gone. Across the fleet, 4 of 6 running nodes were effectively dead.

The assistant implemented a permanent fix by replacing the unreliable wait -n with a robust polling loop using kill -0 to check process existence at regular intervals. A new Docker image was built and pushed. As a tactical measure, the cuzk daemon was manually restarted on the three crashed nodes to restore proving capacity immediately.

But the deeper question remained: why were the processes dying in the first place? The answer came from the user, who hypothesized that vast.ai might be enforcing a separate memory limit via a host-side watchdog, distinct from the cgroup limits visible inside the container. The assistant confirmed this by querying the vast.ai API and discovering the mem_limit field—a value that differed significantly from both physical RAM and cgroup limits on every machine. The daemon's detect_system_memory() function, which read the cgroup to determine the memory budget, was using an inflated value. The daemon would allocate memory up to ~95% of the cgroup limit, inadvertently exceeding vast.ai's real cap, and the host-side watchdog would deliver a silent SIGKILL.

This discovery was the catalyst for everything that followed. The user recognized that reactive debugging was insufficient—the infrastructure needed intelligent, automated management that could anticipate constraints rather than merely respond to failures. The directive was clear: build an autonomous agent to manage the fleet, scale it based on Curio SNARK demand, and alert humans only when necessary.

Part II: Architecting an Autonomous Agent from Scratch

The assistant responded to this directive with a remarkable burst of engineering. Before writing a single line of new code, the assistant conducted systematic reconnaissance: reading the existing vast-manager codebase, exploring the Curio PostgreSQL database schema, and evaluating the qwen3.5-122b model's tool-calling capabilities through a synthetic subagent assessment that passed all six tests.

The architecture that emerged was a clean separation of concerns. A comprehensive Go API (agent_api.go, 1,357 lines) was built to expose 12 endpoints covering demand monitoring, fleet status, instance lifecycle management (with safety guards against excessive concurrent launches), alerting, and per-machine performance tracking. A Python autonomous agent (vast_agent.py, 697 lines) was created to run on a 5-minute systemd timer, using the LLM to make scaling decisions by calling the Go API.

The assistant dispatched two parallel subagent tasks to construct these components simultaneously—a deliberate architectural choice that reflected a clear understanding of the dependency graph. The Go file defined endpoints that the Python agent would later call, but both could be written concurrently against a shared interface contract. The Go/Python split was pragmatic: Go excelled at robust HTTP servers with connection pooling and type-safe API definitions, while Python offered superior ergonomics for LLM interaction and rapid prompt iteration without recompilation.

The system was deployed to the management host, and the first end-to-end test revealed a routing bug—the agent hit a 404 when calling /api/agent/offers because the Go API had registered the endpoint at /api/offers. The assistant fixed the missing route, rebuilt, redeployed, and re-ran the test. This time, the agent successfully observed 8 pending PSProve tasks, identified an RTX 5090 offer at $0.361/hr with 0.985 reliability, and launched instance 33007738. The first autonomous instance launch was a milestone.

Part III: Learning to Think in Hours, Not Minutes

The celebration was short-lived. The user's very next message delivered a devastating piece of operational feedback: "Note on rates, be careful—starting an instance can take hours. Pending tasks can be really volatile and are NOT a useful metric, PSProve takes 4 minutes and each machine churns out one every 30s pipelined."

This single message shattered the assumption that the agent's logic was sound. The assistant's reasoning reveals the dawning realization: those 8 pending tasks would be cleared in approximately 4 minutes by the existing worker, while the newly launched instance would take 1-2 hours to start contributing. The agent had just wasted money launching capacity that would arrive long after the demand had evaporated.

The user clarified the correct approach: "The agent should ideally: scale down the cluster when there is no activity for 1h+, scale up to 500proofs/h as target." When the assistant began designing complex auto-targeting logic involving PI controllers and exponential moving averages, the user immediately corrected course: "There should be no complex auto targets." The assistant internalized this with refreshing clarity: "Right—keep it simple. The agent is a 122B model, not a control system."

The redesigned decision framework was distilled to three simple rules: scale up to target 500 proofs/hour fleet capacity when there is active demand; scale down when there are no completions for 1+ hour; and treat pending count as noise, ignoring it entirely for scaling decisions. This was a fundamental shift from reactive to proactive control. Instead of asking "is the queue too long?", the agent would ask "does the fleet have enough sustained capacity to meet the target?" The demand endpoint was enhanced to expose completion rates over 15-minute and 1-hour windows, plus an active flag. The fleet endpoint was enhanced with a totals section that summed capacity across all running instances.

The user's next refinement introduced the concept of provenance into the agent's decision-making. The directive was precise: "Agent should prefer instances which we saw working, especially ones that we've seen perform in curio." This was a shift from predictive trust (what a machine might do based on benchmarks) to empirical trust (what a machine has actually done in production). The assistant built a /api/agent/perf endpoint that queried Curio's harmony_task_history for per-machine completion and error counts, and updated the Python agent to maintain a markdown file summarizing instance performance. The LLM would read this file before making launch decisions, preferring machines with proven track records over unknown gambles.

The assistant also recognized a critical blind spot: the agent's fleet summary showed current capacity but ignored instances that were still loading. With 1 running instance producing 40 proofs/hour and 2 instances loading, the agent saw a gap of 460 proofs/hour against the 500 target and might launch more instances unnecessarily. Rather than adding complex rules, the assistant improved the data—the fleet summary was updated to explicitly state: "1 running, 2 loading (will add capacity in 1-2h)." This gave the LLM the information it needed to make forward-looking decisions without imposing rigid logic.

Part IV: From Ephemeral to Conversational—The Architecture of Memory

The most consequential architectural decision in the segment came from a single user question: "Is the agent in one compactable 'conversation' with feedback (Pi agent runtime style) or ephemeral per cron? How are actions linked together?" This question cut to the heart of the agent's architecture, and the assistant's honest answer revealed a fundamental limitation.

The agent was entirely ephemeral. Each five-minute cron invocation started a completely fresh Python process, fetched current state from APIs, read a flat fleet-performance.md file for context, built a new system prompt, made 1-5 LLM calls, and exited. There was zero conversational continuity between runs. The LLM had no memory of what it said or decided five minutes ago. Actions were linked only by temporal proximity in database tables, not by any coherent thread of decision-making.

The user's response was decisive: implement a persistent conversational runtime with up to 30k tokens of context. The assistant executed this vision rapidly, building a SQLite-backed conversation log, Go API endpoints for storing and retrieving messages, and a completely rewritten Python agent that appended observations, decisions, and tool results to an ongoing thread rather than starting fresh each time.

The new architecture worked as follows. A conversation log table in SQLite stored the rolling message thread, providing persistent, crash-safe storage that survived agent restarts and server reboots. Each run appended its observation, decision, and tool results to the thread, so the LLM could see its own past reasoning, past decisions, and the outcomes of those decisions. Context window management kept the thread within a 30k token budget—when the conversation approached ~25k tokens, older messages were compressed into a summary via LLM-based summarization, preserving the essential narrative while discarding redundant detail. Tool results were truncated before storage: the get_offers tool returned ~50,000 characters (~12,000 tokens) of raw JSON; storing this verbatim would consume nearly half the context budget in a single run. Truncation to 1,000 characters preserved the structure and key information while dramatically reducing token consumption. Human feedback was injected as user messages in the thread, giving the agent genuine memory of human intent.

The first real run of this new architecture produced a landmark result. The agent observed projected capacity of 485 proofs per hour against a target of 500—a 97% fill rate. Three instances were still loading. Rather than reflexively launching another machine, the agent reasoned: "Launching now would overshoot target and waste budget. Monitor next run for loading instances to complete." This was the first demonstration of genuine temporal awareness—the agent was planning across cycles, not reacting in isolation.

The conversational architecture introduced a new problem: context management. Every run appended new messages, and without careful pruning, the context window would grow without bound. The user identified the core insight: "on hold-only observe responses ideally we'd strip them from context so that we save context and don't have redundant hold.. hold.. hold.. hold.." This led to the verdict system, where the agent's output includes a structured <verdict> block indicating whether an action was taken or the state changed. Runs where no state change occurred were excluded from the LLM prompt context while remaining visible in the UI. This decoupling of monitoring and reasoning—storing the full history for human oversight while presenting only the information-dense subset to the LLM—became a core architectural principle.

Part V: The Catastrophe—When a Boolean Signal Failed

Despite the conversational architecture and the enriched demand signals, the agent suffered its most catastrophic failure. The user's report was stark: "agent just stopped all instances even though there are 50 tasks pending, tools gave it wrong info?"

The root cause was a single boolean flag. The demand endpoint computed active as totalRecent > 0—whether any proofs had completed in the last 15 minutes. Under normal operation, this was a reasonable heuristic. But the heuristic contained a fatal blind spot. When all workers crashed simultaneously—every vast.ai instance showed exited status—completions dropped to zero. The active flag dutifully reported False. The agent, following its cost-optimization mandate, concluded that demand had evaporated and began systematically stopping instances. The agent's own log entry revealed the chilling reasoning: "Demand inactive (active=False, 0 throughput 15m). Stopping expensive machine ($0.01308/proof) to reduce costs during low demand."

The agent was not broken. It was faithfully executing its programming. The flaw was in the signal itself. The active flag conflated two fundamentally different situations: "no one wants work" (genuine low demand) and "all workers are dead but tasks are piling up" (a critical emergency). The system had no way to distinguish them.

The fix spanned four layers. The Go API endpoint was enhanced with two new boolean fields: DemandQueued (true when any task queue has pending items) and WorkersDead (true when there are pending tasks but zero alive workers). The fast-path logic was updated to never skip the LLM call when demand_queued was set. The system prompt was modified to include explicit emergency guidance: "If workers_dead=True, this is an EMERGENCY—do NOT stop any instances. Scale UP immediately." And the observation string was updated to include the new fields, ensuring the LLM would actually see them.

The investigation also revealed a second critical gap. The agent's monitor loop tracked instances by their database state, but instances that exited on the vast.ai side (showing exited or error status) were invisible to the agent unless they also transitioned through the database state machine. The assistant fixed this by augmenting the fleet monitor to detect instances with exited/error status on vast.ai and reflect them in the agent's observation. Building on this, the assistant implemented a hard policy: instances stuck in loading or scheduling state for over three hours are automatically destroyed, preventing storage charges from accumulating on instances that will never become operational.

Part VI: Grounding the Agent—From Speculation to Evidence

The diagnostic grounding system was born from another painful failure. The agent observed twelve instances in various stages of startup—some as young as 0.2 hours old—and saw cuzk_alive=false with 0% GPU utilization across the board. Without any deeper investigation, it concluded the instances were "stuck" and proceeded to destroy all of them, including three that had already progressed to the params_done state, a clear signal that startup was proceeding normally.

The agent's reasoning showed a pattern of speculation: "All 3 'params_done' instances show cuzk_alive=false and 0% GPU utilization, indicating the worker process is not running. They've been stuck for 0.8-1.1 hours, which is abnormally long for benchmarking/SRS loading." This was not a malicious decision—it was an ignorant one. The agent lacked the diagnostic tools to distinguish between a genuinely stuck instance and one proceeding through a normal but lengthy startup sequence.

The assistant's first instinct was defensive: add a hard-coded startup guard to the stop_instance and destroy_vast_instance API endpoints that would reject any operation on instances less than two hours old. This is the classic engineering reflex—when an autonomous system makes a bad decision, constrain its decision space with rules.

But the user immediately intervened with a correction that would reshape the entire architecture: "Noo it is fine that the agent may destroy shorter running instances, it just needs to ground itself in truth first and never speculate whether instance state is bad or not. Maybe make it possible to call Stop only after the agent has queried the 'state discovery agent' and grounded itself that the instance is indeed in a bad state."

This was the pivotal insight. The problem was not that the agent was too aggressive—it was that the agent was making decisions without evidence. A hard age-based guard would prevent the agent from destroying genuinely broken young instances. The correct solution was not to restrict the agent's options but to give it the tools to make informed decisions.

The resulting architecture was a two-layer diagnostic pipeline. The Go endpoint (GET /api/agent/diagnose/{vast_id}) SSHes into the target instance and collects raw telemetry: entrypoint logs, daemon logs, process listings, memory statistics, GPU state, and any error messages. It includes a fallback for instances where SSH is broken, returning vast API data and manager database state instead. A sub-agent LLM receives the raw diagnostic data along with context from the codebase about what normal behavior looks like, and returns a structured verdict: "healthy/progressing," "warning (e.g., OOM but recovering)," or "error (crashed, needs intervention)." This is what the main agent sees—not raw log lines, but a distilled, evidence-based assessment.

The stop_instance tool was then gated with an HTTP 428 "Precondition Required" response: before allowing a stop, the Go handler checks whether a recent diagnose action exists in the agent_actions table for that specific vast_id. If not, it returns 428 with a clear error message telling the agent to call diagnose_instance first. This design pattern is profound: the system does not second-guess the agent's conclusions; it only ensures those conclusions are based on data.

Part VII: The Research Pivot—From Reactive Debugging to Principled Design

Despite the demand signal fix and the diagnostic grounding system, the agent continued to struggle. When the workers_dead emergency was detected, the agent tried to launch replacement instances but was blocked by a rate limiter. The assistant added an emergency boolean flag to bypass the rate limit—but the LLM simply ignored the optional parameter. The agent was stuck in a cycle of reactive patching that was not converging on a solution.

The user's response was a strategic intervention: "Start 4 agents to research more SOTA prompting for an autonomous agent like this, SOTA tool descriptions/definitions, SOTA context management, and events/triggering."

This was not a request for another bug fix. It was a directive to step back, learn from the broader community, and rebuild the agent on a foundation of established best practices rather than ad-hoc experimentation. The assistant dispatched four parallel sub-agents, each tasked with deep investigation of a specific domain: prompting strategies (Anthropic's "Right Altitude" prompting, recency bias, structured reasoning formats), tool definitions (BFCL v4 benchmarks, required enums over optional booleans, description length optimization), context management (hybrid sliding-window + summarization, session state anchors, graduated compression zones), and event-driven triggering (systemd.path units, priority-based event classification, hybrid polling+event architectures).

The research findings were synthesized into a concrete, actionable plan with six proposed changes: rename emergency bool to launch_priority required enum on launch_instance; add systemd.path trigger for immediate event response; add session state anchor table with loading/saving; lower tool output masking threshold to 5 messages with smart placeholders; reorder system prompt—critical rules at the end; and add remember and schedule_next_check tools.

The launch_priority change was particularly significant. The research had confirmed that smaller models (70B-122B parameters) systematically skip optional boolean parameters. By making launch_priority a required enum, the assistant forced the model to explicitly choose a priority level on every launch, eliminating the ambiguity that caused the rate-limiter bypass to fail.

The deployment completed the architectural transformation. The assistant created a vast-agent-trigger.path unit—a systemd path unit that uses Linux's inotify mechanism to watch for modifications to a trigger file. Whenever the Go backend detects a P0 event (workers_dead, human message) or P1 event (instance state change, config change), it touches the trigger file, and systemd immediately starts the agent service. The design is deliberately hybrid: the 5-minute timer remains as a backstop, ensuring that even if the path unit misses an event, the agent will still run. This deployment transformed the agent from a scheduled inspector to an on-call responder, dropping event response latency from minutes to milliseconds.

Part VIII: Hardening the Loop—Duplicate Runs, Context Pollution, and Event Thrashing

In the final stretch of the segment, the assistant confronted a class of failure that threatens every autonomous system: the silent, gradual degradation caused by duplicate work, context pollution, and event thrashing. The user reported that the agent seemed to be running "double/parallel" and that context was ballooning uncontrollably.

The assistant decomposed this vague symptom into three distinct root causes, each with a different fix.

Stale scheduled wakes. The agent's scheduling system allowed it to request future wake-ups, but once a wake was due and processed, its status was never updated to processed. Twenty stale pending wakes had accumulated, and every agent run would log "Processing 20 scheduled wakes..." and re-process them all, generating redundant tool calls and observation entries that inflated the conversation history. The fix was straightforward: after detecting and handling due wakes, mark them as processed in the database.

Duplicate assistant messages. The agent's conversation history stored every message the LLM produced during a run, but the LLM often produced multiple messages: an initial reasoning message, followed by tool calls, followed by a final answer. Both messages carried the same run_id, and the context management system was including both in the prompt, doubling the token cost of each run. The fix was to filter the conversation history so that only the final assistant message per run was included in the prompt context.

Burst duplicate triggering. The systemd.path unit watched a trigger file for modifications, but state changes could happen in rapid succession—multiple instances transitioning states within seconds—causing the path unit to fire repeatedly. The assistant implemented a debounce mechanism in the Go triggerAgent() function: P0 triggers (human messages) are debounced to at most one per 2 seconds; P1 triggers (state changes) are debounced to at most one per 10 seconds.

Additional hardening included making run IDs monotonic via persistent session state (so that even after a session reset, the agent correctly continues the sequence from where it left off), implementing verdict-based filtering for no-op runs (kept in the UI with a "skipped in prompt" badge but excluded from the LLM prompt), and discouraging redundant schedule requests by treating 240–360 second wake requests as no-ops since the 5-minute heartbeat timer already covered that window.

Conclusion: The Architecture of Autonomous Reliability

The segment spanning this session documents a remarkable transformation. What began as a methodical deployment of a memory pool fix spiraled into a production crash, then pivoted into the design, construction, and iterative hardening of a fully autonomous LLM-driven fleet management agent. The agent survived multiple production incidents—over-provisioning, accidentally stopping all instances due to a flawed demand signal, context overflow that ballooned to 38,000 tokens, session resets that left it completely paralyzed, and race conditions that spawned duplicate runs.

Several principles emerge from this arc. First, the quality of an autonomous agent's decisions is bounded by the quality of its observations. The active flag was not wrong—it was incomplete. The fix was not to make the agent smarter but to give it better eyes. The workers_dead flag, the diagnostic sub-agent, and the enriched fleet summary all follow this principle: improve the data, not the rules.

Second, simple deterministic rules often outperform complex AI-driven heuristics for infrastructure tasks. The 300-character compaction threshold, the three-hour hard policy for stuck instances, the required enum over optional boolean, and the priority-based debounce all follow this principle. The agent is a 122B model, not a control system—keep the rules simple and the perception rich.

Third, research is not a luxury but a necessity for building reliable autonomous systems. The four parallel sub-agents produced findings that directly shaped the agent's architecture in ways that incremental debugging never could have. The shift from optional booleans to required enums, from polling to event-driven triggering, and from ephemeral to conversational memory all came from systematic research, not reactive patching.

Fourth, grounding replaces speculation. The diagnostic sub-agent system ensures that destructive actions are based on evidence, not inference. By gating stop_instance behind a diagnosis precondition, the architecture forces the agent to ground its decisions in facts before acting. This is a general pattern for trustworthy autonomy: do not constrain what the agent can do; constrain what it can do without evidence.

The final system that emerged from this segment was not a brittle script with hard-coded thresholds, but a genuinely autonomous agent capable of observing fleet state, reasoning about demand, making scaling decisions, learning from production performance data, and responding to emergencies within seconds. It was a system that had earned the trust to spend real money on GPU instances, and had been hardened through real production incidents to handle the edge cases that only operational experience reveals. In the volatile world of GPU proving infrastructure, where instances crash without warning and tasks pile up in queues, that trust is everything.

References

[1] "From Memory Pools to Production Crashes: The Arc of a GPU Infrastructure Crisis" — Analysis of the budget-integrated pinned pool deployment and the wait -n supervisor bug.

[2] "From Silent Crashes to Autonomous Operations: Building an LLM-Driven Fleet Management Agent for GPU Proving Infrastructure" — The initial architecture and deployment of the autonomous agent.

[3] "From Ephemeral to Conversational: Building an Autonomous LLM Fleet Agent with Persistent Memory" — The transition to a persistent conversational runtime with SQLite-backed context management.

[4] "From Catastrophe to Event-Driven Autonomy: The Hardening of an Autonomous GPU Fleet Agent" — The active=False catastrophe, the multi-layer demand signal fix, and the research-driven architectural overhaul.

[5] "Grounding, Verdict, and Recovery: Engineering Reliability into an Autonomous LLM Fleet Agent" — The diagnostic grounding system, verdict-based context management, and the context overflow crisis.

[6] "Hardening the Autonomous Agent: Eliminating Duplicate Runs, Context Pollution, and Event Thrashing" — The final hardening of the agent loop against stale wakes, duplicate messages, and burst triggers.