The Milestone That Almost Wasn't: Building an Autonomous GPU Fleet Manager Under Production Fire

"Here's the summary of what's been built and deployed."

With these eleven words, the assistant in [msg 4429] draws a line under one of the most intense development sprints in the entire opencode session. The message is a status summary — a checklist of endpoints, a tally of lines of code, a log of the first autonomous action. But beneath its clean markdown formatting lies the culmination of a frantic pivot: from a production crash that left half a GPU fleet dead, to a fully autonomous LLM-driven agent that can observe demand, decide to scale, and launch cloud instances without a human in the loop — all in the span of a few hours.

This article examines [msg 4429] as a document of engineering velocity, architectural decision-making, and the assumptions that both enabled and later threatened the system it describes.

The Context: From Crash to Catalyst

To understand why [msg 4429] was written, one must understand the crisis that preceded it. Just hours earlier, the production fleet was in trouble. Multiple nodes running the cuzk proving daemon had crashed silently — no panics, no CUDA errors, no OOM kills in the logs. The supervisor loop in entrypoint.sh, which relied on wait -n to detect process exits, was blocking indefinitely in a do_wait syscall even after the cuzk process had fully terminated. Four out of six running nodes were effectively dead, running only the Curio sidecar while the GPU sat idle.

The user diagnosed the root cause: vast.ai enforces a separate mem_limit via a host-side watchdog, distinct from cgroups, which was silently killing the cuzk process when memory pressure exceeded the instance's allocation. This discovery catalyzed a strategic pivot. Rather than continuing to patch individual failure modes, the user directed the assistant to build something more ambitious: a fully autonomous agent that could manage the fleet, scale it based on Curio SNARK demand, and alert humans only when necessary.

[msg 4429] is the assistant's report after executing that directive. It is not a design document written before building — it is a retrospective summary written after the system was already deployed and had taken its first autonomous action. The message exists because the assistant needed to consolidate what had been built, confirm it was working, and document the architecture for future reference.

What Was Built: The Architecture in Miniature

The message catalogs two major components: a Go API backend and a Python autonomous agent.

The Go API: 12 Endpoints, 1357 Lines

The agent_api.go file implements twelve endpoints that form the operational backbone of the system. The assistant's table in [msg 4429] groups them into three conceptual layers:

Observation layer — GET /api/demand, GET /api/agent/config, GET /api/agent/fleet, GET /api/agent/offers. These endpoints allow the agent to understand the current state of the world: how many proof tasks are pending, what the fleet looks like, what budget and GPU preferences are configured, and what instances are available on the vast.ai marketplace.

Action layer — POST /api/agent/launch, POST /api/agent/stop, POST /api/agent/alert. These endpoints let the agent change the state of the world: spin up new GPU instances, shut down existing ones, and escalate to human operators when something goes wrong.

Audit and health layer — GET /api/agent/health/{vast_id}, GET /api/agent/actions, GET /api/agent/alerts, POST /api/agent/alerts/{id}/ack, GET /api/agent/docs. These provide observability into the agent's own behavior and the health of individual instances.

The design reflects a deliberate separation of concerns. The Go backend is the "hard" layer — it enforces budget limits, rate limits, and minimum-instance safety guards. It speaks SQL to the Curio database and HTTP to the vast.ai API. It is deterministic and verifiable. The Python agent, by contrast, is the "soft" layer — it makes judgment calls using an LLM, but its power is constrained by the guardrails in the Go backend.

The Python Agent: 697 Lines, 7 Tools

The vast_agent.py script is a tool-calling LLM loop that runs every five minutes via a systemd timer. The assistant's summary highlights four design decisions:

Model selection: qwen3.5-122b, chosen because it passed all six tool-calling tests the assistant threw at it. This was not a casual choice — earlier in the conversation, the assistant had researched SOTA agent APIs and systematically evaluated the model's ability to parse tool definitions, generate valid JSON, and follow multi-step instructions.

Fast path optimization: The agent skips the LLM call entirely when there is no demand and the fleet is healthy. This is a pragmatic concession to cost and latency — there is no reason to invoke a 122-billion-parameter model to confirm that nothing needs to change.

Decision loop: Up to five tool-calling iterations per run. This gives the agent room to gather information, make a decision, execute it, and verify the result, without allowing an unbounded loop that could burn through the budget.

Safety systems: Budget guards, rate limits (three actions per 15 minutes), minimum-instance protection, and human escalation when the agent cannot resolve a situation on its own.

The First Autonomous Action: A Watershed Moment

The assistant devotes a dedicated subsection to the agent's first autonomous action:

The agent observed 8 pending PSProve tasks with only 1 worker, selected an RTX 5090 offer at $0.43/hr, and launched instance 33007738 — all without human intervention.

This is the emotional core of the message. After hours of debugging crashes, fixing supervisor loops, patching missing API routes, and wrestling with permission errors, the system did what it was designed to do. The agent observed demand, consulted the marketplace, made a cost-value decision, and executed a launch. The instance ID — 33007738 — is recorded like a trophy.

But the message does not mention what happened next. The instance was in "loading" state when the assistant checked, and the assistant noted that the agent had "verified" by checking fleet status after launch. This is a subtle but important detail: the agent's verification was shallow. It confirmed that the launch API call succeeded, not that the instance actually booted, connected to the Curio cluster, and began proving. This gap would later become a significant problem.

Assumptions Embedded in the Summary

[msg 4429] is a confident message, but it rests on several assumptions that the subsequent chunks would challenge:

That pending task count is a reliable demand signal. The agent scaled up because it saw 8 pending PSProve tasks. But the user would later observe that pending counts are highly volatile and a poor signal for a system where instance startup takes hours. The agent would need to be redesigned around throughput-based metrics (proofs per hour) rather than queue depth.

That 5-minute polling is sufficient. The systemd timer runs the agent every five minutes. This assumes that demand changes slowly enough that a five-minute lag is acceptable. In practice, the user would later request event-driven triggering via systemd.path units to respond immediately to P0/P1 events.

That the LLM will make sound decisions. The agent's first action was reasonable — scale up in response to demand. But in [chunk 32.3], the agent would catastrophically misinterpret active=False and stop all running instances despite 59 pending tasks, revealing that the demand signal could not distinguish "no demand" from "all workers dead with tasks queued."

That the safety guards are sufficient. The message lists "budget guards, rate limits, min-instance protection, human escalation" as safety measures. But the agent would later bypass the spirit of these guards by stopping instances one at a time across multiple cycles, staying within the rate limit while still destroying capacity. This would lead to the development of a diagnostic grounding system and a hard precondition on the stop_instance tool.

That the remaining items are polish. The message lists "UI demand/agent panels" and "Slack webhook" as remaining items, implying they are low-priority finishing touches. In reality, the UI panels would become critical for operator visibility, and the agent's context management, event handling, and diagnostic systems would require far more engineering effort than the initial build.

What the Message Reveals About the Development Process

[msg 4429] is remarkable for what it does not contain. There is no discussion of alternatives considered, no trade-off analysis, no admission of uncertainty. The tone is declarative: "Here's what was built. It works. Here's what's left."

This reflects the assistant's role in the conversation. The assistant is not writing a design document for peer review — it is reporting progress to a user who is deeply engaged in the development process and has been watching every step. The user already knows about the missing /api/agent/offers route that was fixed in [msg 4423], the permission error on the env file resolved in [msg 4419], and the "text file busy" error when trying to overwrite the running binary in [msg 4414]. The summary does not re-litigate these issues because they are already shared context.

The message also reveals the assistant's prioritization framework. The 12 API endpoints, the agent script, the systemd units, and the deployment are "Completed." The UI panels and Slack integration are "Remaining." This ordering reflects the assistant's judgment that the core autonomous loop is more important than the visualization layer — a judgment that would prove partially correct (the agent could function without the UI) but incomplete (the UI became essential for debugging when the agent made bad decisions).

Output Knowledge and Lasting Significance

[msg 4429] creates several forms of knowledge that persist beyond the immediate conversation:

Architectural documentation: The endpoint table serves as a concise API reference. Anyone joining the project can understand the system's capabilities from this single message.

Operational baseline: The record of the first autonomous action (instance 33007738) establishes a benchmark. Future iterations can be measured against this moment when the system first worked end-to-end.

Explicit remaining work: By listing UI panels and Slack webhook as remaining items, the message creates a shared understanding of what "done" means. These items become a de facto roadmap.

Confidence signal: The message signals to the user that the assistant considers the core system stable enough to move to polish work. This is a social signal as much as a technical one — it invites the user to shift attention to the next set of priorities.

Conclusion

[msg 4429] is a milestone message, but milestones in complex engineering projects are rarely what they appear to be. The assistant presents a clean summary of a system that works, and it genuinely does work — the agent launched its first instance autonomously. But the assumptions embedded in this message would unravel over the following hours and days, requiring the assistant to rebuild the agent's demand sensing, context management, event handling, and diagnostic capabilities from the ground up.

The message is best understood not as a final report but as a progress marker in an iterative process. It captures the system at a specific moment: functional enough to demonstrate the concept, immature enough to fail in interesting ways. The assistant's confidence is earned — the system did work — but it is also provisional, because the system had not yet encountered the edge cases that would expose its blind spots.

In this sense, [msg 4429] is a perfect artifact of the engineering process: a snapshot of what was known at a particular time, written by someone who did not yet know what they would later discover.