The Fleet Management Specification: Designing a Self-Orchestrating GPU Cluster for Zero-Knowledge Proving

Introduction

In the course of building a production-ready Docker image for the Curio/CuZK zero-knowledge proving stack, a pivotal moment arrives when the conversation shifts from "how do we package the software" to "how do we operate it at scale." Message 758 in this opencode session is that moment. It is a dense, multi-layered specification delivered by the user that outlines an entire management system for a fleet of GPU instances running on vast.ai, a marketplace for cloud GPU compute. This message is not a simple request or a bug report—it is a complete system architecture, expressed in natural language, that the assistant must implement.

The message arrives after more than 60 prior exchanges in segment 5 alone, covering Docker build fixes, benchmark script creation, port tunneling with portavailc, and a spurious StorageMetaGC error fix. The Docker image (theuser/curio-cuzk:latest) is now functional, containing curio, sptool, cuzk-daemon, and cuzk-bench binaries with CUDA 13 supraseal support. What remains is to turn this single-container solution into a self-managing fleet that can provision itself, benchmark its own performance, and gracefully integrate into a larger proving infrastructure. Message 758 is the blueprint for that transformation.

The Message

The user writes:

We need to expand the built-in bg entrypoint - after param fetching is done, benchmark should auto-run (note both bench and run should have --partition-workers 10 if ram is <400GB), then if the benchmark result is > 50 proofs per hour instance should move to running cuzk deamon (run.sh with same --partition-workers flag, so 16 default, 10 if lowmem), then once cuzk endpoint is ready start curio run --listen 0.0.0.0:[2000+instanceID]; 10.1.2.104 (with the portavaild.service making local ports available on the vast contaiers) should run an additional service which monitors and manages the vast workers. on start the vast container sends 'register' with $VAST_CONTAINERLABEL, service returns matched uuid (makes it safer because other containers won't be able to guess the uuid for other operations), then the service will -1. list running vast instances in the account, if any is noticed that doesn't register for 15 mins kill it, 0. kill instance if vast host was previously marked as bad (instances selected manually so errors can be made) 1. wait for a signal that param fetch is done (up to 90min), kill instance on timeout; 2. wait for benchmark finish max 20mins, kill instances below benchmark threshold (passed to instance as MIN_RATE env var, default 50 per hour; Passed to instance as env because instance price decides min rate). 3. before starting curio the instance queries 'runner ID' to use with curio bind. Once curio and cuzk are running create a 'monitor.sh' script which displays instance logs from the two services. Install official vast CLI on 10.1.2.104, copy api key from this machine there (~/.config/vastai/vast_api_key, DO NOT READ IT DIRECTLY, ls/scp but don't view)

This single paragraph contains approximately 15 distinct requirements, each with its own rationale, edge cases, and failure modes. It is a remarkable example of how an experienced operator thinks about distributed system management: not in terms of happy-path functionality, but in terms of lifecycle, failure detection, timeout enforcement, and security boundaries.

Why This Message Was Written: The Motivation

The motivation behind this message is the gap between "the Docker image works" and "the fleet is self-sustaining." The user has been iterating on the Docker build for hours, pushing new versions to Docker Hub, fixing build blockers, and testing on remote instances. Each iteration revealed new problems: the daemon startup was too slow, the default config was too aggressive for smaller GPUs, the StorageMetaGC task spammed errors on snark-only clusters. These were all single-instance concerns.

But the real operational challenge is managing dozens or hundreds of GPU instances simultaneously. On vast.ai, instances are rented on-demand, priced by the hour, and must be monitored continuously. An instance that fails to register, takes too long to download parameters, or achieves poor proving throughput is burning money without producing value. The user needs a system that automates the entire lifecycle: provisioning, parameter fetching, benchmarking, production proving, and teardown.

The message also reveals a specific topology. The IP address 10.1.2.104 appears repeatedly—this is the controller host, likely a dedicated machine within the same network as the vast.ai instances (connected via the portavaild tunnel service). The controller runs the management service, the vast CLI, and coordinates the fleet. The instances themselves are ephemeral workers that register, prove their worth through benchmarking, and then enter production.

Design Decisions Embedded in the Specification

The message is dense with design decisions, each reflecting operational experience:

RAM-based configuration partitioning. The user specifies --partition-workers 10 if RAM is below 400GB, and 16 otherwise. This is a heuristic based on the observation that the synthesis phase of proof generation is memory-intensive. On a 256GB machine (as seen in earlier messages where a "broken pipe" error indicated an OOM kill), 16 partition workers would exhaust memory. By tying the partition worker count to available RAM, the system self-tunes to the hardware without manual configuration per instance.

The 50 proofs/hour benchmark threshold. This is the gate that separates viable instances from underperformers. The user passes this as the MIN_RATE environment variable rather than hardcoding it, because "instance price decides min rate." A $0.50/hour GPU might be profitable at 30 proofs/hour, while a $2.00/hour GPU needs 100+. By making this an env var, the management service can set different thresholds per instance based on cost.

The numbered management service operations (-1, 0, 1, 2, 3). The user lists operations with specific indices that are notable: -1 (kill unregistered instances after 15 minutes) and 0 (kill instances on bad hosts) are listed before 1, 2, and 3. This ordering suggests priority: the management service should first clean up orphaned or blacklisted instances before processing new registrations. The negative index for "kill unregistered" and zero for "bad host" is an intentional design choice to communicate execution order.

Timeout values. Each lifecycle phase has a specific timeout: 15 minutes for registration, 90 minutes for parameter fetching, 20 minutes for benchmarking. These are not arbitrary. The 44GB SRS file and 25.7GB PCE file take significant time to download (as seen in earlier messages where the daemon startup timeout had to be increased to 600 seconds). The 90-minute param fetch timeout accounts for slow network conditions. The 20-minute benchmark timeout ensures that even with 12 warmup proofs at ~2 minutes each, there's headroom.

UUID-based registration security. The user specifies that the container sends $VAST_CONTAINERLABEL on registration and receives a UUID back. This prevents other containers from guessing UUIDs for operations like status updates or runner ID queries. It's a simple but effective security measure in a multi-tenant environment.

Port allocation scheme. Curio listens on port 2000+instanceID. This is a deterministic port allocation that avoids conflicts when multiple instances connect through the portavaild tunnel. The instance ID is likely a monotonically increasing integer assigned by the management service's SQLite database.

Assumptions Made

The message makes several implicit assumptions that are worth examining:

Network connectivity. The specification assumes that vast.ai instances can reach 10.1.2.104 reliably, and that the portavaild tunnel service makes local ports available on the instances. This is a non-trivial networking assumption—vast.ai instances run on diverse cloud providers with varying network configurations.

Instance identity. The user assumes that $VAST_CONTAINERLABEL is a reliable, unique identifier set by vast.ai on container startup. If this label is not guaranteed unique or can be spoofed, the security model weakens.

Benchmark as a proxy for production performance. The assumption that 12 benchmark proofs predict sustained production throughput is reasonable but not proven. GPU thermal throttling, memory fragmentation, and network latency to the controller could all degrade performance over time.

Single controller, no failover. The management service runs on a single host (10.1.2.104). If this host goes down, the entire fleet operates without oversight—instances continue running but no new instances can be managed, and orphan detection stops.

The assistant's ability to implement all of this. The user is asking the assistant to build a Go-based management service, modify the entrypoint, create monitor scripts, install the vast CLI, and set up the entire system. This is a substantial engineering effort spanning multiple programming languages and systems.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message creates a complete specification document for a distributed system. The output knowledge includes:

  1. A lifecycle model for GPU proving instances: register → fetch params → benchmark → prove → (implicitly) terminate.
  2. A failure model with specific timeouts and thresholds for each phase.
  3. A security model using UUID-based registration to prevent unauthorized access.
  4. A resource allocation model that adapts partition workers to available RAM.
  5. A monitoring model with monitor.sh for log aggregation.
  6. A cost model where the minimum acceptable performance is a function of instance price. This specification is the bridge between the Docker image (a deployable artifact) and the operational system (a running fleet). It defines the "what" that the assistant must implement in subsequent messages.

The Thinking Process Visible in the Message

The message reveals a mind that thinks in terms of systems, not just features. The numbered list (-1, 0, 1, 2, 3) shows the user mentally ordering operations by priority. The specific timeout values (15min, 90min, 20min) show someone who has internalized the expected duration of each phase. The security concern about UUID guessing shows awareness of adversarial environments.

The parenthetical "(instances selected manually so errors can be made)" is particularly revealing. It acknowledges that human operators make mistakes when selecting instances, and the system must be resilient to those errors. The "bad host" list is a safety net for human error.

The instruction "DO NOT READ IT DIRECTLY, ls/scp but don't view" about the vast API key shows operational security awareness. The assistant should copy the key file without ever displaying its contents, preventing accidental exposure in logs or conversation history.

Conclusion

Message 758 is a turning point in this opencode session. It transforms the project from "build a Docker image" to "build a self-managing GPU fleet." The specification it contains is remarkable for its completeness, its operational realism, and its attention to failure modes. Every timeout, every threshold, every security measure reflects real experience running GPU workloads at scale.

For the assistant, this message is a complex implementation challenge. For the reader, it is a window into how experienced infrastructure engineers think about distributed systems: not as a collection of features, but as a lifecycle with well-defined phases, failure detectors, and safety nets. The message is simultaneously a requirements document, an architecture proposal, and an operations manual—all in a single, dense paragraph.