From Deployment to Dashboard: The Consolidation of a GPU Fleet Management System

Introduction

In the sprawling development of a distributed GPU proving infrastructure for the Filecoin network, there comes a phase where raw construction gives way to consolidation. The code compiles, the services start, the APIs respond—but the system must prove itself against the unforgiving reality of production infrastructure. This article examines a pivotal chunk of an opencode coding session where an AI assistant and a human operator took a freshly built management service called vast-manager and subjected it to the crucible of real-world deployment: port conflicts, routing bugs, automated monitors that killed pre-existing instances, and the eventual pivot from a headless API to a comprehensive web dashboard.

The chunk spans messages 772 through 847, covering the deployment of the vast-manager service to a controller host, the verification of its background monitor, the discovery and fix of a subtle Go routing bug, a Docker image rebuild, and the user's request for a full-featured operations dashboard. Each of these episodes reveals something fundamental about how infrastructure software is built, tested, and operated in practice.

The Deployment: When Assumptions Meet Reality

The vast-manager service was designed as a Go-based HTTP API backed by SQLite, tasked with orchestrating GPU instances rented through Vast.ai—a decentralized marketplace for cloud GPU compute. The service tracks instances through a state machine (registered → param-done → bench-done → running), runs a background monitor that periodically enumerates all Vast.ai instances and destroys unregistered ones older than 15 minutes, and maintains a "bad hosts" blacklist for avoiding unreliable GPU providers.

Deploying this service to the controller host at 10.1.2.104 was anything but straightforward. The assistant built the Go binary with cross-compilation flags (CGO_ENABLED=1 GOOS=linux GOARCH=amd64), set up a systemd unit, and configured the vast CLI with the API key. But the first assumption to break was the port: port 1234, the originally intended port for the vast-manager, was already occupied by a Lotus node—the Filecoin consensus software. This forced a move to port 1235 and a corresponding update to the portavaild tunnel configuration, a change that rippled through the entrypoint scripts and the Docker image ([msg 793], [msg 797], [msg 802]).

The assistant's methodical approach to this port conflict is instructive. Rather than simply picking a new port and moving on, the assistant verified the conflict with ss -tlnp, confirmed that lotus was the occupant, updated the service file, the portavaild configuration, and the entrypoint script—all before restarting the service. This is the hallmark of disciplined infrastructure engineering: when an assumption fails, trace its dependencies and update them all, not just the most obvious one.

The Monitor Proves Itself

With the service running on port 1235, the assistant turned to verifying the background monitor—the component that periodically queries the Vast.ai API and destroys unregistered instances. The monitor's first cycle was dramatic: it immediately identified two pre-existing instances that had user-set labels but were not registered in the manager database, and destroyed them ([msg 827], [msg 830]). A third unlabeled instance was also targeted after the user explicitly instructed the assistant to kill it ([msg 833]).

This was a double-edged sword. On one hand, it proved the monitor worked exactly as designed—unregistered instances were identified and terminated within the 15-minute grace period. On the other hand, it destroyed instances that the user had been actively using before the manager was deployed. The assistant correctly identified this as an operational issue, not a bug: "The monitor worked exactly as designed—it killed unregistered instances older than 15 minutes. The problem is that we deployed the monitor with real instances already running that weren't registered" ([msg 830]).

Only one instance survived: C.32705217, a single RTX 4090 machine that had been properly registered in the manager database. The assistant verified its health via SSH, confirming that cuzk (the proving engine) and curio (the Filecoin node) were both running, with the GPU idling at 3% utilization between proofs ([msg 837]).

The Routing Bug: A 404 That Revealed a Deeper Truth

With the monitor validated, the assistant turned to testing the bad-host API—a critical feature for blacklisting unreliable GPU providers. The test sequence was straightforward: POST to add a test entry, GET to list entries, DELETE to remove the test entry, and GET to confirm removal. The POST and GET succeeded. The DELETE returned a 404 HTML page ([msg 837]).

The assistant's diagnosis was immediate and precise: "The DELETE returned a 404 HTML page—the routing for DELETE /bad-host/99999 isn't working. The issue is that Go's http.ServeMux doesn't route /bad-host/99999 to the /bad-host handler—it needs a separate pattern" ([msg 838]).

This is a classic Go HTTP routing pitfall. The standard library's http.ServeMux uses exact path matching for patterns registered without a trailing slash. A handler registered for the pattern /bad-host matches exactly that path—not /bad-host/99999. The handler code already contained logic to parse the host_id from the URL path using strings.SplitN, but the mux never routed requests to it. The fix, applied in message 840, switched the mux registration to a catch-all pattern with manual path parsing—a common pattern in Go HTTP servers that need parameterized paths without a third-party router.

The fix was deployed atomically: the Go binary was rebuilt, copied to /tmp/vast-manager on the remote host via scp, moved to /usr/local/bin/vast-manager with sudo, and the systemd service was restarted—all in a single chained command ([msg 841]). The verification test confirmed the fix: DELETE /bad-host/99999 now returned {"ok": true}, and the subsequent GET confirmed the entry was removed ([msg 842]).

This episode illustrates a fundamental principle of software engineering: the most insidious bugs are often not in the logic but in the plumbing. The DELETE handler was correct. The database query was correct. The HTTP method was correct. The only thing wrong was that the request never reached the handler. The assistant's ability to trace a 404 response back through the routing layer to the mux registration—and to articulate the fix in a single sentence—is the essence of effective debugging.

The Checkpoint: A Moment of Metacognition

With all API endpoints verified and the monitor functioning correctly, the assistant paused to update the plan document ([msg 843]). This todowrite call marked four high-priority tasks as completed: building and deploying the binary, installing the systemd unit, setting up the vast CLI, and exposing port 1235 through portavaild.

This checkpoint is far more significant than its brevity suggests. It represents a moment of metacognitive reflection—the assistant pausing the forward momentum of execution to formally acknowledge completion, update its internal model of the world, and prepare for the next phase. The plan document is a persistent artifact that survives across conversation rounds, tool invocations, and potential interruptions. By writing to it, the assistant creates a durable record of what has been accomplished, ensuring that even if the conversation were interrupted or resumed later, the state of progress would not be lost.

The checkpoint also serves as a synchronization point with the user. The user can inspect the plan document, see what has been completed, and redirect the assistant to the next priority. In this case, the user did exactly that—immediately after the checkpoint, the user requested a comprehensive web UI, fundamentally changing the direction of the work.

The Docker Image Rebuild: Baking Correctness into the Artifact

The assistant's next step was to rebuild the Docker image that would be used to launch new Vast.ai instances ([msg 844]). This was motivated by a fundamental architectural insight: the entrypoint script—the code that runs inside each Vast.ai container when it boots—needed to be updated to communicate with the vast-manager on port 1235. Without this change, any new instances launched from the existing Docker image would attempt to reach the manager on port 1234, fail silently, and operate outside the management system's oversight.

The assistant had two options: patch the entrypoint on each running instance individually (fragile, manual, non-scalable) or rebuild the Docker image with the corrected entrypoint baked in (consistent, automated, scalable). The assistant chose the latter, demonstrating an understanding that infrastructure management is fundamentally about reducing operational surface area and ensuring consistency across deployments.

The inclusion of monitor.sh in the rebuild was equally significant. The monitor script, which runs inside each container and performs health checks, was now bundled into the image from the start, ensuring that every instance would have the monitoring infrastructure available from the moment it boots.

The Dashboard Imperative: From Headless API to Operations Hub

With the deployment consolidated and the Docker image rebuilt, the user issued a request that would reshape the entire project: "Plan and build a comprehensive manager webui (go embed), served from a separate port bound to 0.0.0.0" ([msg 845]). The user enumerated a feature list that included an instance list with states and timeouts, performance metrics (price per hour, proofs per hour, price per proof), manager logs, instance logs piped from remote machines, a manual kill button, and SSH connection commands derived from the vast CLI.

This request was grounded in deep operational experience. The user had watched the assistant deploy and test the system over many messages and recognized that the next step was operational visibility. The backend worked, but operating it required SSHing into the controller host and making curl calls. A web UI would make fleet status immediately accessible to anyone with a browser.

The user's specification carried several implicit architectural assumptions that shaped the subsequent implementation. The choice of "go embed" meant the HTML, CSS, and JavaScript would be compiled into the Go binary—a single-binary deployment that eliminates deployment complexity and avoids Node.js dependencies on the controller host. The separate port bound to 0.0.0.0 (unlike the management API which was bound to localhost) meant the dashboard would be externally accessible while keeping the management API private. The request for instance logs with the parenthetical "yes need to pipe from instances" acknowledged the architectural challenge: logs cannot be fetched on demand from the UI; they must be continuously shipped from worker instances back to the manager.

The assistant's response was a masterclass in disciplined software development. Rather than immediately generating code, the assistant began by reading the existing main.go to understand the current data structures and API surface ([msg 847]). This act of reading before building ensured that the subsequent implementation would be grounded in reality—the assistant would know exactly which data structures to extend, which API patterns to follow, and which existing functionality to leverage.

The Architecture of Consolidation

What unifies these episodes—the port conflict, the monitor verification, the routing bug fix, the checkpoint, the Docker rebuild, and the dashboard request—is a consistent pattern of disciplined infrastructure engineering. Each episode follows the same cycle: observe a symptom, form a hypothesis, gather evidence, apply a fix, verify. The assistant never jumps to conclusions or applies random changes. It reads code before modifying it, tests endpoints before trusting them, and updates persistent state before moving to the next task.

This chunk also reveals the collaborative nature of the development process. The user is not a passive observer but an active participant who identifies operational gaps (the need for a dashboard), provides critical context (the port conflict, the missing VAST_CONTAINERLABEL), and redirects the assistant's priorities. The assistant, for its part, maintains transparency about its reasoning, updates shared artifacts, and accepts user corrections without defensiveness.

The transition from a headless API to a web dashboard marks a fundamental shift in the project's trajectory. The vast-manager was no longer just a background service for orchestrating instances—it was becoming an operations hub, a visible and controllable center for managing a fleet of GPU proving workers. The dashboard would eventually include summary cards for fleet-wide metrics, a sortable instance table with expandable rows, log viewers filtered by source, bad hosts management, auto-refresh, and keyboard shortcuts. But at the moment captured in this chunk, it was still just a plan—a plan grounded in a thorough understanding of the existing code, the operational requirements, and the infrastructure constraints.

Conclusion

The chunk spanning messages 772 through 847 represents a critical consolidation phase in the vast-manager project. The assistant deployed a management service, tested it against real infrastructure, fixed a routing bug that would have silently broken a critical operational feature, rebuilt the Docker image to bake in correctness, and laid the foundation for a comprehensive web dashboard. Each step was methodical, grounded in observation and verification, and responsive to the realities of production infrastructure.

The routing bug fix, in particular, exemplifies the kind of debugging that separates effective engineers from merely competent ones. The assistant recognized the 404 response as a routing issue rather than a handler logic issue, understood Go's ServeMux semantics well enough to name the exact cause without reading the code first, and applied a minimal fix that stayed within the standard library rather than introducing a third-party dependency. This is the hallmark of an engineer who understands not just what the code says, but what the framework expects.

As the project moves into the web UI implementation phase, the foundation laid in this chunk—the verified API endpoints, the tested monitor, the fixed routing, the rebuilt Docker image, and the shared understanding of what the dashboard should be—will serve as the bedrock for the next stage of construction. The system is no longer just functional; it is consolidated, verified, and ready to be made visible.