The Blueprint for an Autonomous Fleet: How One Message Transformed GPU Cluster Management

Introduction

In the lifecycle of any complex software project, there are moments when the trajectory shifts fundamentally—when a team stops fighting fires and starts building systems that prevent them. For the CuZK proving infrastructure project, that moment arrived in message index 4374, a comprehensive design plan for an autonomous fleet management agent. This message, written by an AI assistant in response to a production crisis and a bold user directive, represents a remarkable synthesis of systems thinking, API design philosophy, and pragmatic engineering. It is the architectural blueprint that would guide the development of a fully autonomous LLM-driven agent capable of managing a distributed GPU proving cluster across vast.ai's marketplace, scaling it up and down based on real-time SNARK proving demand, and alerting humans only when truly necessary.

To understand why this message matters, one must appreciate the context that produced it. The conversation leading up to this point had been a whirlwind of production debugging. The assistant had just diagnosed a critical failure mode: all GPU instances on vast.ai had been silently terminated, not by kernel OOM or application crashes, but by vast.ai's own host-side watchdog enforcing a mem_limit parameter that was invisible from inside the containers. The cgroup limits that the system carefully calculated were being overridden by a separate enforcement layer, causing abrupt SIGKILLs with no trace in logs. Every instance was dead. The user's response was not to tweak a parameter—it was to demand a fundamentally new approach.

"From vast-manager plan Agent-friendly (Rest+md doc? MCP?) API," the user wrote in [msg 4369], "to check on instance states, list and start instances, check on instance business and performance. Idea is that we have a cron tool-calling/mcp agent which autonomously manages the vast instances, checks health, escalates issues, scales the cluster up and down based on Curio snark demand."

This was not a request for a quick fix. It was a strategic pivot from reactive operations to autonomous infrastructure management. And the assistant's response—message 4374—would become the foundational document for that transformation.

The Crisis That Demanded a New Approach

The production crash that precipitated this pivot was instructive in its brutality. The assistant had spent chunks of development building a sophisticated budget-integrated pinned memory pool, deploying it to RTX 5090 test machines, and validating SnapDeals proof completion. But when the system was put under real load, vast.ai's hidden memory enforcement layer struck without warning. The mem_limit field in vast.ai's API revealed the truth: the platform enforces its own memory ceiling, independent of and potentially conflicting with the cgroup limits the containers see internally. When the CuZK daemon's memory usage approached the cgroup limit—which was set higher than mem_limit—vast.ai's host watchdog sent a SIGKILL. No error logs. No panic messages. No CUDA errors. Just dead processes and a supervisor loop that was itself broken, blocking indefinitely on a wait -n syscall that never noticed the child had already exited.

The assistant's diagnosis in the preceding messages ([msg 4365] through [msg 4368]) was masterful, tracing through raw vast.ai API output, comparing mem_limit against cgroup values across five instances, and identifying the discrepancy. But the user's response in [msg 4369] was not to ask for a fix to the memory accounting. It was to ask for something far more ambitious: an autonomous agent that could manage the entire fleet, sense demand, and act without human intervention.

This is the critical context for understanding message 4374. It was written at the intersection of deep operational knowledge (the assistant had just spent hours debugging the exact failure modes of these instances) and strategic vision (the user wanted to leapfrog from manual operations to autonomous management). The assistant did not start coding immediately. Instead, it did something that distinguishes great engineering from merely competent work: it researched, explored, and designed before building.

The Research Phase: Understanding the Landscape

Before message 4374 could be written, the assistant conducted two parallel deep-dive investigations. The first, launched via the task tool in [msg 4370], explored the existing vast-manager codebase at /tmp/czk/cmd/vast-manager/main.go. The second, also in [msg 4370], explored the Curio database schema on the PostgreSQL instance accessible through the portavailc tunnel.

The vast-manager exploration revealed a service running two HTTP listeners: an API port for worker-facing operations and a UI port for the dashboard. It had existing endpoints for instance management, log pushing, and dashboard data. But critically, it had no concept of demand sensing, no integration with the Curio database, and no agent-facing API. The assistant needed to understand every existing route, every SQLite table, and every instance lifecycle operation before it could design extensions.

The Curio DB exploration was equally essential. The assistant needed to understand how SNARK proving demand manifests in the database. The HarmonyTask framework emerged as the central scheduling mechanism, with tables like harmony_machines for registered workers, harmony_task for active tasks, and harmony_task_history for completed work. The proof types—PoRep, WindowPoSt, WinningPoSt, SnapDeals—each flow through this framework, and the assistant needed to know which queries would reveal pending work, throughput rates, and worker availability.

But the most consequential research was the third task, launched in [msg 4372]: a research agent to investigate the state of the art for building APIs that autonomous LLM agents interact with. The user had specifically requested this in response to the assistant's question about API format—"Delegate a research agent to learn about current SOTA (march 2026) and give you more grounded input on what's best, especially for a non-big-model agent."

The research findings were decisive. MCP (Model Context Protocol) had become the de facto industry standard by March 2026, with 97M+ monthly SDK downloads and governance under the Linux Foundation. But the research also revealed that MCP's complexity was better suited for rich tool ecosystems, while REST with well-structured documentation remained the most reliable approach for smaller models. The assistant synthesized this into a clear recommendation: REST endpoints returning JSON, with a /api/agent/docs endpoint returning markdown that the agent reads at startup to learn the API. This was the "system prompt for the tool layer"—a self-describing API that even weaker models could navigate.

The Architecture Blueprint: A Deep Dive

Message 4374 is, at its core, a complete architectural specification. It covers five major areas: the REST API surface, UI additions, agent design, database schema, and implementation order. Each section reflects careful consideration of the constraints imposed by the problem domain and the tools available.

The REST API Surface

The assistant designed nine new endpoints, each with a specific purpose in the agent's observe-decide-act loop:

GET /api/agent/docs — The meta-endpoint. Returns a markdown document describing all agent-callable endpoints. This is the agent's onboarding manual, designed to be read once at startup and cached. It reflects a deep understanding of how LLMs interact with APIs: they need structured, predictable interfaces with clear semantics, not the sprawling, hyperlinked documentation that human developers prefer.

GET /api/demand — The eyes of the agent. This single endpoint queries the Curio database and returns a complete picture of SNARK proving demand: queue depths per proof type, pipeline stages (sectors awaiting SDR, encoding, proving), proofshare demand, throughput over the last hour, and worker availability. The inclusion of a summary field—a pre-computed natural language line—is a brilliant concession to the limitations of weaker models. The assistant explicitly noted this was "for weaker models that struggle to interpret nested JSON."

GET /api/agent/fleet — The state of the cluster. Returns a simplified, agent-optimized fleet view with per-instance metrics (GPU utilization, memory usage, last proof age, cost per hour) and aggregate totals. The budget section provides headroom calculations so the agent can immediately see how much scaling capacity remains.

GET /api/agent/offers — The marketplace interface. Returns filtered and ranked vast.ai offers, pre-sorted by value (proofs per dollar based on historical host_perf data). This endpoint encapsulates the complex economics of GPU rental: not all RTX 5090s are equal, and the agent needs to pick winners.

POST /api/agent/launch and POST /api/agent/stop — The action endpoints. These are the agent's hands. The launch endpoint accepts an offer_id and a reason string, returning the new vast_id or an error. The stop endpoint accepts a vast_id and a reason. Both endpoints validate against budget constraints and safety guards. The reason field is not decorative—it is logged to the agent_actions table and displayed in the UI, providing an audit trail for every decision the agent makes.

GET /api/agent/health/{vast_id} — The diagnostic endpoint. Returns a quick health check with structured fields (cuzk_alive, curio_alive, last_proof_completed_secs_ago) and an issues array with human-readable problem descriptions and suggested actions.

POST /api/agent/alert — The escalation endpoint. Allows the agent to send alerts with severity levels (info, warning, critical), a title, detail, action taken, and a needs_human flag. The alert is delivered via Slack webhook, stored in an in-memory alert log visible in the UI, and made available via a GET endpoint for the agent to review its own history.

GET /api/agent/config — The constraint endpoint. Returns the agent's operating parameters (max spend, max instances, preferred GPUs, scale thresholds). Configurable without redeploying the agent, allowing humans to adjust boundaries in real time.

GET /api/agent/actions — The history endpoint. Returns recent agent actions with timestamps, action types, details, and reasons. Stored in a new agent_actions SQLite table.

Safety by Design

Perhaps the most important aspect of the API design is the safety architecture. The assistant embedded multiple layers of protection:

Server-side budget enforcement: The launch endpoint checks current_dph + offer_dph <= max_dph before provisioning. The stop endpoint refuses if only min_instances would remain. These guards operate independently of the agent's reasoning, providing a hard safety net.
Rate limiting: Maximum 3 launches and 3 stops per 15-minute window. This prevents the agent from thrashing—launching and stopping instances in rapid succession due to noisy demand signals.
Audit trail: Every action is logged with the agent's stated reason. This creates accountability and enables post-hoc analysis of the agent's decision-making.
Human override: The config endpoint allows humans to set max_instances=0 to freeze all agent activity. This is the nuclear option—a single API call that overrides every agent decision.
Alerting: The agent can escalate when it encounters situations it cannot handle, such as >3 crashes per hour or budget exceeding 80%. This creates a human-in-the-loop for exceptional circumstances.

The Agent Design

The assistant's agent design in Part 3 of the message is notably pragmatic. It proposes a simple Python script run by a systemd timer every 5 minutes, following an observe-decide-act loop:

systemd timer (every 5min)
  → agent.py
    → GET /api/demand         (observe demand)
    → GET /api/agent/fleet    (observe capacity)
    → GET /api/agent/config   (read limits)
    → [LLM decision]          (tool-calling)
    → POST /api/agent/launch  (scale up)
    → POST /api/agent/stop    (scale down)
    → POST /api/agent/alert   (escalate)

The decision prompt is carefully structured with the current state, demand, and rules. The rules are simple and deterministic: never exceed budget, keep minimum instances, scale up when pending tasks exceed threshold for 2+ cycles, scale down when idle beyond timeout. The LLM's role is to choose which offer to pick (price/performance tradeoff), write good alert messages, and handle novel situations.

The assistant also considered a purely rule-based alternative for maximum cheapness, with LLM escalation only for ambiguous situations. This reflects a sophisticated understanding of where LLMs add genuine value versus where they introduce unnecessary cost and latency.

The LLM Choice

The assistant recommended Qwen 2.5 72B via vLLM or an API provider, noting that DeepSeek V3 at $0.28/M tokens was the cheapest option. The reasoning was that the decision logic was simple enough for 30B models, but 70B+ provided reliable tool-calling at 10-15 tools. This recommendation would prove prescient—as the agent system evolved across subsequent chunks, the LLM's ability to interpret complex fleet states and make nuanced decisions became increasingly important.

Design Decisions and Their Rationale

Every element of message 4374 reflects deliberate tradeoffs. The assistant could have chosen MCP over REST, but the research showed that REST with markdown documentation was more reliable for smaller models. The assistant could have embedded the agent logic in the vast-manager Go code, but chose a separate Python script for isolation and independent iteration. The assistant could have designed a single monolithic "do everything" endpoint, but chose granular endpoints that map to distinct agent capabilities.

The most subtle design decision is the summary field in the demand and fleet endpoints. This is a recognition that LLMs, particularly smaller ones, struggle with deeply nested JSON structures. By providing a pre-computed natural language summary, the assistant effectively gives the model a "cheat sheet" that it can use to understand the situation even if it fails to parse the structured data correctly. This is the kind of design insight that comes from deep experience with LLM limitations.

The decision to put safety guards on the server side (in the launch and stop endpoints) rather than trusting the agent to follow its prompt rules is another critical design choice. The assistant understood that LLMs are not reliably obedient—they can misinterpret instructions, hallucinate constraints, or simply ignore rules under certain conditions. By making the API itself refuse dangerous actions, the assistant created a hard safety boundary that no amount of LLM creativity could bypass.

Assumptions and Their Implications

Message 4374 makes several assumptions that would prove significant in the subsequent development:

The agent would run on a 5-minute cron timer. This assumption shaped the entire architecture—the observe-decide-act loop, the polling-based demand sensing, the stateless design. In later chunks, this would prove insufficient, leading to event-driven triggering via systemd.path units and a persistent conversation context.

The LLM would be a relatively small model (72B parameters). This drove the design of simplified endpoints with summary fields, structured prompts with clear rules, and limited tool sets. In practice, the agent would eventually need more sophisticated reasoning capabilities, but the initial design constraint ensured the system was accessible to cheaper models.

The agent would be stateless. The design shows no mechanism for maintaining context across runs. Each 5-minute invocation would start fresh, reading the current state and making decisions independently. This assumption would be challenged by the need for persistent memory, conversation history, and learning from human feedback—all of which would be added in later chunks.

Slack would be the alert channel. The assistant asked for confirmation about Slack webhook configuration, but the design assumes Slack as the primary escalation path. This proved correct, though the alert system would evolve significantly.

The Curio DB would be accessible without authentication. The assistant asked about credentials, but the design assumes the portavailc tunnel on port 5433 provides unauthenticated access. This assumption held true.

What This Blueprint Enabled

The design in message 4374 is remarkable not just for its completeness, but for how accurately it anticipated the challenges that would emerge during implementation. The safety guards (budget enforcement, minimum instances, rate limiting) would prove essential when the agent later misinterpreted demand signals and tried to stop all running instances. The audit trail (action logging with reasons) would enable debugging when the agent made unexpected decisions. The alert endpoint would become the foundation for the escalation system.

But the blueprint was also a starting point, not a final specification. In the chunks that followed ([msg 4375] onwards), the assistant would build the API endpoints, create the agent script, and deploy the system. The user's feedback would reshape the agent's logic—pending task counts proved too volatile as a demand signal, instance startup times of hours made rapid scaling impractical, and the simple cron-based architecture needed event-driven triggering and persistent context.

The design evolved significantly from this blueprint. The agent gained a rolling conversation log in SQLite, LLM-based summarization, a diagnostic sub-agent system for grounding decisions in facts, and a verdict system for structured feedback. The simple Python script became a sophisticated autonomous system with file locks, debounce mechanisms, and session state management. But the core architecture—the REST endpoints, the safety guards, the observe-decide-act loop, the Slack integration—remained exactly as designed in message 4374.

The Thinking Process Behind the Design

The assistant's reasoning in the messages leading up to 4374 reveals a methodical approach to design. In [msg 4370], the assistant explicitly breaks down the user's requirements into three categories: API capabilities, agent responsibilities, and model constraints. It then identifies the need to explore the existing codebase before designing anything new. This is not a designer who jumps to solutions—it is one who seeks to understand the terrain before drawing the map.

The research phase in [msg 4372] is particularly instructive. The assistant does not simply ask "should we use MCP or REST?" and take the answer. Instead, it launches a full research task to investigate the current state of the art, the adoption rates of different protocols, the performance characteristics with smaller models, and the ecosystem maturity. This is design informed by evidence, not opinion.

The synthesis in message 4374 itself shows a designer working through multiple layers of abstraction. The architecture diagram at the top establishes the physical and logical topology. The endpoint descriptions drill into specific API contracts with example JSON payloads. The agent design section addresses the runtime behavior and decision-making process. The safety section enumerates failure modes and their mitigations. The implementation order section provides a concrete roadmap. This is design at four levels of abstraction simultaneously—system architecture, API contract, agent behavior, and project planning.

Conclusion

Message 4374 stands as a masterclass in architectural design for autonomous systems. It was written at a moment of crisis—all GPU instances dead, the supervisor loop broken, the team facing the prospect of another night of firefighting. But instead of proposing another tactical fix, the assistant proposed a strategic transformation. It synthesized deep operational knowledge of the existing system, research into state-of-the-art agent API design, and a pragmatic understanding of LLM capabilities into a coherent blueprint for autonomous fleet management.

The design decisions in this message—the choice of REST over MCP, the server-side safety guards, the summary fields for weaker models, the granular endpoint design, the audit trail—reflect a sophisticated understanding of both the technical domain and the limitations of the tools being used. The assumptions, while not all proven correct, were reasonable starting points that enabled rapid iteration.

What makes this message truly remarkable is not just its completeness as a design document, but its role as the foundation for a system that would grow far beyond its initial specification. The agent that eventually emerged from this blueprint was more complex, more robust, and more capable than anything envisioned in message 4374. But it was built on the solid architectural foundation laid in those carefully designed endpoints, those thoughtfully structured JSON payloads, and those hard safety boundaries. The blueprint did not constrain the implementation—it enabled it.