From Build to Fleet: The Complete Engineering of a CUDA Proving Stack

Introduction

In the span of a single opencode coding session spanning messages 589 through 771, the assistant and user accomplished a remarkable breadth of engineering work: they completed a production-grade Docker container for a GPU-accelerated zero-knowledge proving stack, pushed it to Docker Hub, created benchmarking and deployment scripts, integrated a port tunneling system for distributed communication, patched a spurious error in the proving daemon, and designed a comprehensive self-managing fleet orchestration system for vast.ai GPU instances. This article traces the full arc of that work, from the last build blockers to the architectural specification of a distributed GPU proving fleet.

The Docker image at the center of this work — theuser/curio-cuzk:latest — bundles the Curio Filecoin proving node, the CuZK GPU proving engine, the sptool utility, and the cuzk-bench benchmarking tool, all compiled against CUDA 13.0.2 on Ubuntu 24.04. But the image is merely the foundation; the real story is in the operational infrastructure built around it: scripts that handle the full lifecycle of proving instances, a tunnel integration that connects distributed nodes to a central controller, a bug fix that eliminated spurious error logs on snark-only clusters, and a complete management system design that turns a collection of GPU instances into a self-regulating fleet.

Part I: The Docker Build — Overcoming Every Blocker

The journey to a working Docker image was a catalog of everything that can go wrong in a multi-stage CUDA Docker build. Each blocker required a different diagnostic approach, and each fix taught something about the system's architecture. By the end, the assistant had resolved four distinct build failures and one production bug, each with its own root cause and remediation strategy.

The Pip Conflict That Nearly Derailed Everything

The first major blocker emerged from a seemingly innocuous interaction between Python packaging systems. The supraseal build script — part of the SPDK (Storage Performance Development Kit) that provides high-performance storage operations for the proving pipeline — creates a Python virtual environment and runs pip install --upgrade pip inside it. On Ubuntu 24.04, the Debian-managed python3-pip package installs pip as a system-level package. When the venv's pip attempts to upgrade itself, it detects the system pip package in sys.path and tries to uninstall it — but the uninstallation fails because the Debian package manager owns the RECORD file, not pip. The result is a cryptic error: ERROR: Cannot uninstall pip 24.0, RECORD file not found. Hint: The package was installed by debian.

The assistant's diagnostic process for this issue was methodical. It traced the error through the build chain — build.sh → pkgdep.sh → pip install --upgrade pip — and identified the root cause as the conflict between the Debian-managed pip and the venv's pip. The fix was elegantly simple: remove python3-pip from the Dockerfile's apt-get install list entirely. The python3-venv package bootstraps pip via ensurepip, which uses the wheel file from python3-pip-whl (a dependency of python3-venv). This gives the venv its own pip without any system-level pip package to conflict with. The assistant verified this approach by checking package dependencies in a clean container, confirming that python3-venv does not depend on python3-pip and that removing python3-pip would not break venv creation ([msg 597], [msg 600]).

The Phantom Library: Tracing libcudart_static.a

The second blocker appeared during the Go link step. The Go linker could not find libcudart_static.a, producing the error cannot find -lcudart_static. This static library is required for CUDA runtime support in the Go binaries — curio and sptool are Go programs with CUDA bindings via cgo. The assistant traced the file to /usr/local/cuda-13.0/targets/x86_64-linux/lib/libcudart_static.a, which is accessible via the symlink chain /usr/local/cuda → /etc/alternatives/cuda → /usr/local/cuda-13.0 and thus available at /usr/local/cuda/lib64/libcudart_static.a ([msg 608]).

The root cause was that the Dockerfile's LIBRARY_PATH environment variable only included the CUDA stubs directory (where stub versions of CUDA libraries live for compile-time linking). The main lib64 directory was missing. The fix was to append /usr/local/cuda/lib64 to LIBRARY_PATH, giving the linker access to libcudart_static.a and any other static libraries needed during the Go build ([msg 611]).

The Missing Runtime Libraries

The third category of issues emerged during smoke testing. When the assistant ran ldd on the binaries inside the container, it found four unresolved dependencies: libconfig++.so.9, libaio.so.1t64, libfuse3.so.3, and libarchive.so.13 ([msg 616]). These are dynamically linked by the curio binary as dependencies of the supraseal/SPDK build. The runtime stage of the Dockerfile, which starts from the minimal nvidia/cuda:13.0.2-runtime-ubuntu24.04 image, did not include these libraries. The fix was to add libconfig++9v5, libaio1t64, libfuse3-3, and libarchive13t64 to the runtime stage's apt-get install command ([msg 618]).

The Successful Build and Push

After all blockers were resolved, the assistant rebuilt the Docker image and pushed it to Docker Hub ([msg 721]). The final image, at approximately 3GB, contains all four binaries — curio, sptool, cuzk-daemon, and cuzk-bench — compiled with CUDA 13 supraseal support and all necessary runtime libraries. The push succeeded with the digest sha256:4a12325fa3f312c375435b5a2589d5cb7074cb4ae6490bfe1c569c4bfe187966, marking the culmination of the Docker build effort that had been ongoing across multiple segments.## Part II: The Benchmark and Run Scripts

With the Docker build stable, the user requested a benchmarking script in [msg 634]. The assistant responded by spawning subagent tasks to research the cuzk codebase, discovering that cuzk-bench is a separate binary with a batch subcommand that can run multiple proofs and report timing. A second subagent task investigated how C1 output test data is obtained, discovering a downloadable JSON file (~51 MB) from a public R2 bucket that serves as test data for PoRep proofs.

The Benchmark Script

The resulting benchmark.sh script ([msg 638]) was designed with several thoughtful features:

The Run Script

The run.sh script ([msg 695]) followed a similar pattern but was designed for production deployment rather than benchmarking. It starts the cuzk-daemon with configurable GPU and partition parameters, handling the full lifecycle of daemon startup, parameter preloading, and graceful shutdown. Key features include:

Part III: The Portavailc Tunnel Integration

The Docker image was functionally complete for local proving, but a standalone proving node is only half the story. The user requested integration of portavailc — a Go-based port forwarding client that connects to a central daemon at 10.1.2.104:22222 and establishes tunnels for ports 1234, 5433, 9042, and 4701 ([msg 737]). This is the mechanism by which instances in the distributed fleet communicate with the controller host.

The assistant's integration was architecturally sound: it added the portavailc build in the builder stage of the multi-stage Dockerfile (where Go is already installed), then copied the compiled binary to the runtime stage with a COPY --from=builder directive (<msg id=740-741>). The entrypoint script was updated to start portavailc in the background when the $PAVAIL environment variable is set ([msg 743]), keeping the image flexible — the same image can run with or without port forwarding depending on the deployment context.

This design decision — embedding the tunnel client directly in the image rather than requiring a separate setup step — dramatically simplifies deployment. A vast.ai instance can be launched with a single command that sets the PAVAIL secret, and the tunnel is established automatically before any proving work begins. The tunnel provides access to the management service on port 1234, the Curio node on port 5433, and other internal services, all through a single encrypted connection to the controller host.

Part IV: The StorageMetaGC Bug Fix

Just as the Docker image was nearing completion, the user reported a spurious error: the StorageMetaGC task in Curio was logging ERROR-level messages every 21 minutes on snark-only clusters (instances with no local storage paths). The error read: sector_path_url_liveness update: transaction didn&#39;t commit. While the error was non-fatal — the GC task simply skipped its work — it was concerning because ERROR-level logs trigger alerting systems and create noise in monitoring dashboards.

The assistant traced the issue to a subtle behavior in the database transaction layer ([msg 718]). When BeginTransaction with OptionRetry() is called with no executed statements (because there are no storage paths to check), the transaction refuses to commit and returns committed=false. The OptionRetry() flag is designed for concurrent access scenarios where transactions may need to retry on conflict — but when there are no statements to execute, the commit is a no-op that returns false, which the error handler interprets as a failure.

The fix was a one-line guard clause: if there are no storage paths, return early with success before entering the transaction at all. This eliminated the spurious error entirely and was immediately built into the Docker image and pushed to Docker Hub ([msg 721]).

This bug is a classic example of an edge case that only manifests in specific deployment configurations. On a full Curio node with storage paths, the transaction always has work to do and commits successfully. On a snark-only proving node — a configuration that was only recently made possible by the CUDA proving stack — the empty transaction path was never exercised during development. The fix is minimal but essential for production reliability.## Part V: The Vast.ai Management System Design

The capstone of this segment is the design of a comprehensive vast.ai management system, documented in vast-cuzk-plan.md ([msg 763]). This is not a simple script or configuration — it is a complete distributed system architecture for managing a fleet of GPU proving instances, designed to operate autonomously with minimal human intervention.

The User's Vision

The user's specification in [msg 758] was remarkably dense, containing approximately 15 distinct requirements in a single paragraph. It described a feedback loop where a Docker container running on a vast.ai GPU instance would autonomously progress through stages: register with a management service, fetch proving parameters (with a 90-minute timeout), run a benchmark (with a 20-minute timeout), and only if the benchmark rate exceeded a configurable threshold (default 50 proofs per hour), proceed to start the cuzk daemon and curio service. The management service on the controller host would track every instance through its lifecycle, kill orphaned or underperforming instances, assign monotonically increasing runner IDs, and maintain a bad-host list to prevent re-deployment on problematic machines.

The Assistant's Architectural Response

The assistant's response was a masterclass in architectural decomposition. It identified five components that together form the complete system:

1. The Go-based Management Service (cmd/vast-manager/main.go): Running on the controller host at 10.1.2.104, this service listens on port 1234 (forwarded by the portavailc tunnel) and uses SQLite for persistence. Its API surface is minimal and carefully designed:

| Endpoint | Method | Description | |---|---|---| | /register | POST | {label} → {uuid, runner_id}. Creates entry, assigns sequential runner_id | | /param-done | POST | {uuid} → signals param fetch complete | | /bench-done | POST | {uuid, rate} → signals benchmark result (proofs/hour) | | /runner-id | GET | ?uuid=X → {runner_id}. Allocates persistent monotonic runner ID | | /status | GET | List all instances and states | | /bad-host | POST | {host_id, reason} → add to bad hosts list | | /bad-host/{host_id} | DELETE | Remove from bad hosts list |

The background goroutine runs every 60 seconds, executing vastai show instances --raw and applying a cascade of kill conditions:

Key Design Decisions

Several design decisions in this plan merit detailed attention.

The unification of runner ID and instance ID was a critical insight. The assistant initially treated them as separate entities, but after re-reading the user's answers, realized they are the same monotonically increasing counter that maps directly to the curio listen port via 127.0.0.1:$((2000 + runner_id)) ([msg 761]). This simplification reduces the API surface and eliminates a potential source of confusion. The runner ID counter is stored in SQLite and persists across management service restarts, ensuring that port numbers never collide — a requirement for Curio's HarmonyTask internal task manager, which uses the bind address to track which node is working on which tasks.

The UUID-based registration scheme provides lightweight security. The management service generates a UUID at registration time, which serves as a bearer token for subsequent API calls. This prevents other containers from impersonating registered instances — a container would need to know both the instance's label and its UUID to make authenticated requests. Since the UUID is generated server-side and never exposed to other instances, the attack surface is minimal.

The RAM-based partition worker tuning (10 workers for <400GB, 16 otherwise) is a direct response to operational experience. The user had previously encountered OOM errors on the RTX 4000 Ada (20GB) system when too many partition workers were spawned. Encoding this heuristic into the entrypoint means that every instance automatically tunes itself to its hardware capabilities, without requiring manual configuration per instance type.

The background monitor's cascade of kill conditions reveals a deep understanding of operational reality. The assistant doesn't assume that instances will behave correctly. Instead, it designs for failure: timeouts for every step, explicit kill actions, and a clear separation between "still working" and "stuck." The 15-minute grace period for registration acknowledges that Docker image pulls and cold starts take time. The 90-minute param fetch timeout acknowledges that downloading multi-gigabyte parameter files over the internet is inherently unreliable. The 20-minute benchmark timeout acknowledges that GPU proving can fail in ways that don't produce clean error messages.

The Self-Cleanup Philosophy

The most striking aspect of the management system design is its emphasis on self-cleanup. Every instance in the fleet is expected to either progress through the lifecycle to become a productive proving node, or be terminated. There is no manual intervention path for cleaning up stuck instances — the system handles it automatically.

This philosophy extends to the bad-host list. If a particular vast.ai host consistently produces underperforming or unreliable instances, it can be added to the bad-host list with a single API call. From that point forward, any instance that appears on that host is killed immediately, regardless of its state. This provides a mechanism for the operator to encode operational knowledge into the system without writing code.

The design also anticipates the need for rate limiting and cost management. The MIN_RATE threshold is passed as an environment variable to each instance, allowing the operator to set different thresholds for different GPU types or pricing tiers. An expensive A100 instance might require a higher threshold than a budget RTX 4090, ensuring that the fleet's cost-per-proof remains within acceptable bounds.

The Thinking Process: From Build to Architecture

What makes this segment remarkable is not just the breadth of work accomplished, but the quality of the thinking at every stage. The assistant's diagnostic process for each build blocker was methodical: trace the error through the build chain, identify the root cause, verify the fix, and confirm with a rebuild. The architectural design for the vast.ai management system was equally rigorous: decompose the problem into components, identify dependencies, design interfaces, anticipate failure modes, and present the plan for review before implementing.

The assistant's willingness to ask clarifying questions — about the management service language, the source of instance IDs, the semantics of the runner ID — demonstrates a crucial engineering discipline: resolve ambiguity before writing code. The user's answers shaped the entire architecture. Without understanding that the runner ID is implicitly derived from the listen port, the assistant might have designed an unnecessarily complex configuration system. Without understanding that the management service should be written in Go (the same language as Curio), the assistant might have proposed a Python or Node.js service that would add an unnecessary language dependency to the deployment.

The assistant also demonstrated a strong sense of operational awareness. The StorageMetaGC bug fix, the RAM-based worker tuning, the UUID-based registration security — these are not features that were explicitly requested. They are responses to edge cases and failure modes that the assistant anticipated based on its understanding of the system. This proactive approach to reliability engineering is what separates a competent implementation from a truly robust one.

Conclusion

This segment of the opencode session is a microcosm of the entire engineering lifecycle: from debugging a broken build, through creating operational scripts, integrating network infrastructure, fixing edge-case bugs discovered in production, and designing a complete distributed system architecture. The Docker image that emerged — theuser/curio-cuzk:latest — is not just a container; it is the foundation for a self-managing fleet of GPU proving instances that can provision themselves, benchmark their own performance, and gracefully integrate into a larger proving infrastructure.

The journey from "the build fails" to "here is the complete fleet management specification" spans approximately 180 messages and countless tool calls. It demonstrates that effective AI-assisted engineering is not about generating code in isolation — it is about maintaining coherent context across debugging cycles, asking the right questions at the right time, and building shared understanding through structured documentation. The vast-cuzk-plan.md file, written at the end of this segment, is the crystallization of that understanding — a document that transforms a conversational design into an implementable specification for a self-managing GPU fleet.

The themes that emerge from this work — complete Docker build and push to Docker Hub, fixing the SPDK pip upgrade conflict, resolving the libcudart_static.a linker error, installing missing runtime libraries, creating benchmark.sh and run.sh scripts, adding portavailc tunnel support to the entrypoint, fixing the StorageMetaGC error in curio, and designing the vast.ai management system — are not isolated accomplishments. They are interconnected pieces of a larger puzzle: building a reliable, scalable, and self-managing infrastructure for GPU-accelerated zero-knowledge proving on the Filecoin network.