From Black Box to Dashboard: The Observability Architecture of a GPU Proving Engine

Introduction

In any complex distributed system, there is a fundamental tension between the internal complexity of the system and the operator's ability to understand what it is doing. A GPU proving engine like cuzk is a particularly extreme case: it manages multiple concurrent proof jobs, each requiring CPU-bound synthesis phases, GPU-bound proving phases, and careful memory management across SRS (Structured Reference String) tables and PCE (Prover Cache Entry) allocations. When something goes wrong — an OOM crash, a stuck partition, a GPU worker that has silently failed — the operator needs to know immediately. But the engine runs on remote, firewalled instances accessible only via SSH. Its internal state is, by default, invisible.

This segment of the opencode session represents the final step in a long journey toward observability. Over the preceding segments, the assistant had built the cuzk proving engine from scratch, implemented a budget-based memory manager, deployed it to remote vast.ai instances, and added a lightweight HTTP status API that exposes the engine's internal state as JSON snapshots. But a status API that nobody can see is a status API that might as well not exist. The segment documented here — Segment 19 — closes the loop by bridging that API into the operator's management dashboard, transforming the proving engine from a black box into a transparent, observable process.

The user's instruction in [msg 2553] was precise and concise: "commit, then extend vast-manager (has access to the ssh addrs already) ui to, when a running node is selected, show a rich timeline visualization (live, no history because we don't poll the data in background)." This kicked off a tightly sequenced implementation spanning roughly 30 messages, from codebase exploration through backend endpoint construction to frontend visualization. The result is a live monitoring bridge that tunnels GPU proof pipeline state through SSH ControlMaster and renders it as a rich dashboard — a memory gauge, per-job partition pipeline grids with color-coded phases, GPU worker state cards, SRS/PCE allocation tables, and aggregate counters — all polled every 1.5 seconds with clean lifecycle management via AbortController.

This article examines that implementation in depth, focusing on the architectural decisions, the exploration process, the incremental build discipline, and the design of the visualization itself. It also steps back to consider what this segment means in the broader context of the cuzk project: the completion of an observability architecture that spans from the engine's internals to the operator's screen.

The Commit That Made It Possible

The segment opens with the assistant confirming the commit of the status API (<msg id=2554-2555>). The commit message tells the story in detail:

"cuzk: add lightweight HTTP status API for live monitoring. Adds a StatusTracker that records pipeline, GPU worker, and memory state as proof jobs flow through the engine. A minimal raw-TCP HTTP/1.1 server on a configurable port (daemon.status_listen) serves GET /status returning JSON snapshots at 500ms polling granularity."

This was not a trivial addition. The commit spanned 10 files and 717 lines of new and modified Rust code. It included a new module (status.rs), modifications to the engine's lifecycle to wire in status tracking at every pipeline transition point, configuration for the listen port, and the HTTP server itself — all implemented as a raw TCP listener with manual HTTP response formatting to avoid adding dependencies to the daemon. The assistant had tested this end-to-end in the preceding messages, running a full PoRep proof through the daemon and verifying that every partition transitioned through the expected phases: synthesizing → synth_done → gpu → done, with completed jobs garbage-collected after 30 seconds.

The commit created a clean architectural boundary. The status API was now a committed, production-ready feature with a well-defined JSON schema. The next phase — integrating it into the operator dashboard — could proceed on solid ground, knowing that the data source was reliable and the schema was stable.

The SSH ControlMaster Architecture

The most critical architectural decision of this segment emerged during the assistant's reasoning in [msg 2558]. The cuzk daemon's status API listens on port 9821, but these are remote vast.ai instances — their ports are not exposed to the internet. The only access path is SSH, which the vast-manager already uses for instance management. But establishing a fresh SSH connection every polling cycle would be prohibitively expensive. SSH handshakes involve key exchange, authentication, and channel establishment, easily adding a second or more of latency per request.

The assistant's reasoning shows a moment of genuine reconsideration: "but we actually don't need SSH for this! The status API is on a TCP port. But wait — the user said 'SSH-based poller (ports aren't exposed)'. So we need to tunnel through SSH to reach port 9821." This self-correction is crucial. The assistant had momentarily forgotten a key constraint and caught itself before proceeding down the wrong path.

The solution was SSH ControlMaster — a feature of OpenSSH that allows multiple SSH sessions to share a single TCP connection via a Unix domain socket. The first ssh invocation creates the control socket; subsequent invocations within the timeout window piggyback on the same connection, bypassing the expensive handshake entirely. The assistant articulated this clearly: "The cleanest approach is to use SSH ControlMaster through the shell — running ssh with the right flags to reuse connections via a Unix socket. The first call sets up the connection, and subsequent calls within the timeout window reuse it automatically, which avoids the expensive handshake overhead while keeping the implementation simple."

This decision reflects a deep understanding of the operational environment. The vast-manager is a Go binary deployed on a control machine; it already runs shell commands for SSH management. Adding an SSH library would increase build complexity, binary size, and maintenance burden. By contrast, shelling out to ssh -o ControlMaster=auto -o ControlPath=... is zero-dependency, leverages an already-proven tool, and integrates naturally with the existing codebase. The ControlMaster socket path is derived from the instance's SSH address, ensuring that each instance gets its own connection pool and that connections are properly scoped.

The architecture works as follows:

  1. The vast-manager's Go backend receives a GET request for /api/cuzk-status/{uuid}.
  2. It extracts the UUID and looks up the instance's SSH details from an in-memory map.
  3. It constructs an SSH command using ControlMaster=auto and ControlPersist=300 (5-minute timeout).
  4. The command executes curl -s http://localhost:9821/status on the remote instance.
  5. The JSON response is captured from stdout and returned directly to the frontend. The ControlPersist=300 setting means that the control socket remains alive for 5 minutes after the last SSH invocation. Since the frontend polls every 1.5 seconds, the connection is constantly refreshed and never times out during active monitoring. When the operator collapses the instance panel, polling stops, and the control socket eventually expires on its own — no explicit cleanup needed.

Codebase Exploration: Reading Before Writing

Before writing a single line of code, the assistant conducted a thorough exploration of the vast-manager codebase. This exploration phase, spanning [msg 2557] through [msg 2574], was methodical and deliberate. The assistant did not guess at the code structure — it read the actual files, understood the patterns, and built a mental model of how the new feature would integrate.

The first action was to spawn a subagent task ([msg 2557]) to explore the codebase comprehensively. The subagent returned a detailed analysis: the vast-manager was a single-file Go service (main.go, 1804 lines) serving a single-file HTML/CSS/JS dashboard (ui.html, 1354 lines). It used SQLite for state, had SSH access to all provisioned instances, and already had an instance expansion mechanism that fetched logs when a running node was clicked.

The assistant then performed a series of targeted reads to understand the specific integration points:

Designing the Frontend Visualization

The frontend implementation was the most substantial piece of work in this segment. The assistant approached it in four distinct edits to ui.html, each adding a layer of the visualization in strict dependency order.

Edit 1: CSS Foundation ([msg 2579])

The assistant added styles for the cuzk status panel, integrating with the existing dark theme. The CSS defined the visual vocabulary:

Edit 2: HTML Container ([msg 2580])

The assistant added a structural placeholder inside the expanded instance detail row — a div with a specific ID that would be populated by the JavaScript rendering logic. Without this container, the polling and rendering code would have no DOM node to target. The container is positioned within the existing expansion mechanism, appearing below the instance metadata and log viewer.

Edit 3: Lifecycle Wiring ([msg 2581])

The assistant modified the toggleExpand function to start and stop cuzk polling when an instance is expanded or collapsed. This is the behavioral bridge: polling only happens for the one instance whose detail panel is currently visible. When the operator collapses the panel, polling stops immediately.

The lifecycle management uses the AbortController API — a modern browser API that allows clean cancellation of in-flight fetch requests. When the panel is expanded, a new AbortController is created and stored on the DOM element. When the panel is collapsed, controller.abort() is called, which cancels any pending fetch and prevents the callback from processing stale responses. This prevents a subtle race condition: without the AbortController, a fetch that was initiated just before collapse could complete after collapse, causing the render function to update a DOM node that is no longer visible, potentially leaking data or causing errors.

Edit 4: Polling and Rendering Logic ([msg 2582])

The assistant added the fetchCuzkStatus and renderCuzkStatus functions. The fetch function calls the Go backend endpoint at 1.5-second intervals, handling errors gracefully. The render function transforms the JSON response into the full visualization:

  1. Memory gauge: Reads memory.budget_bytes and memory.used_bytes from the status JSON, computes a percentage, and renders a horizontal bar with the appropriate color and a text label showing the absolute values (e.g., "3.2 GB / 8.0 GB").
  2. Per-job partition pipeline grid: For each active job, renders a grid where each column represents a partition and each row represents a phase transition. The current phase of each partition is shown as a color-coded block. Completed partitions are green, active partitions show their current phase color, and failed partitions are red. The grid includes timing information showing how long each partition spent in each phase.
  3. GPU worker state cards: Reads gpu_workers from the status JSON and renders a card for each worker showing its index, its current state (busy/idle), and the job it is working on (if any).
  4. SRS/PCE allocation tables: Reads srs_allocations and pce_allocations from the status JSON and renders compact tables showing the number of allocations, total memory consumed, and per-structure breakdown.
  5. Aggregate counters: Displays jobs_completed, jobs_failed, and jobs_active as numeric badges at the top of the panel. The polling interval of 1.5 seconds was a deliberate tradeoff. The cuzk status API updates at 500ms granularity internally, but polling at that rate over SSH would generate excessive traffic. At 1.5 seconds, the operator sees near-real-time updates while keeping the SSH overhead manageable. This is especially important because each poll involves an SSH exec of curl, which has non-trivial latency even with ControlMaster connection reuse.

The Visualization Design Philosophy

The assistant's design for the visualization, reasoned through in [msg 2558], was driven by operational use cases. Each element of the dashboard answers a specific question that an operator might ask:

The Engineering Discipline

Throughout this segment, the assistant demonstrated a consistent engineering discipline that is worth examining in its own right. The pattern is visible in every phase:

Explore before acting. The assistant did not start writing code until it had read the relevant sections of both main.go and ui.html. It understood the routing conventions, the SSH access model, the UI rendering lifecycle, and the data schema of the cuzk status API before making a single edit. This upfront investment in understanding paid dividends in implementation speed — the edits were correct on the first attempt, with no backtracking or rework.

Validate after every change. The Go build verification in [msg 2577] is a textbook example. The assistant did not assume the edits were correct — it ran the compiler immediately and checked the output. The separation of concerns was deliberate: verify the backend before building the frontend. If the backend had a compilation error, the frontend work would be wasted.

Sequence work by dependency. The four edits to ui.html followed a strict dependency order: CSS first (foundation), then HTML structure (scaffolding), then lifecycle wiring (behavior), then rendering logic (execution). Each edit built on the previous one without backtracking. This is the software equivalent of building a house from foundation to roof — you cannot install the windows before the walls are framed.

Document the process. The todo list updates in [msg 2568] and [msg 2578] serve as a persistent scratchpad, tracking what has been completed and what remains. The assistant's reasoning in [msg 2558] is a detailed architectural document that explains not just what was built, but why it was built that way. This documentation is not for the assistant — it is for anyone who reads the conversation later, including the user and future maintainers.

Handle edge cases proactively. The AbortController lifecycle management is a perfect example. The assistant anticipated the race condition where a fetch could complete after the panel was collapsed, and designed a solution before the bug could manifest. This is the mark of an engineer who thinks not just about the happy path, but about the failure modes.

Assumptions and Risks

The implementation rests on several assumptions that deserve examination. The SSH ControlMaster approach assumes that curl is available on the remote instances and that the SSH client configuration supports ControlMaster auto and ControlPersist. If a remote instance lacks curl, the polling will fail silently. If the ControlMaster timeout is too short, every poll would re-establish a connection, defeating the optimization. The assistant mitigated this by using a 5-minute ControlPersist timeout, which is well within the typical polling session duration.

The assistant assumes that the cuzk daemon's status endpoint will always be responsive and return the expected JSON schema. The visualization has basic error handling — if the fetch fails, it logs the error and retries on the next polling cycle — but it does not distinguish between transient failures (e.g., SSH connection hiccup) and permanent failures (e.g., daemon not running). An operator seeing a blank panel would need to investigate further.

The polling interval of 1.5 seconds assumes that this is fast enough for operators to track pipeline progress. In the test runs from the preceding messages, GPU times were as short as 3.8 seconds and 6.9 seconds per partition. At 1.5-second polling, a partition that spends only 3-4 seconds on GPU might be captured in the "on GPU" state for only 2-3 polls. This is sufficient for trend detection but not for precise timing measurements. If precise timing is needed, the operator can look at the timing fields in the status JSON, which are captured at 500ms granularity internally.

The Broader Context: Completing the Observability Architecture

This segment does not exist in isolation. It is the culmination of a multi-segment effort to make the cuzk proving engine observable. Looking back across the preceding segments, we can trace the evolution of the observability architecture:

Conclusion

This segment of the opencode session represents a complete, self-contained feature delivery: from a committed backend API through codebase exploration, architectural design, backend implementation, build verification, and frontend visualization. The result is a live monitoring bridge that connects a GPU proving daemon running on a remote, firewalled instance to an operator's management dashboard, using SSH ControlMaster for efficient polling and a rich HTML/CSS/JS visualization for real-time pipeline insight.

The assistant's methodical approach — explore before acting, validate after every change, sequence work by dependency, document the process — is a model of disciplined software engineering. The SSH ControlMaster decision, the AbortController lifecycle management, the color-coded partition pipeline grid, and the separation of fetch and render concerns all reflect careful architectural thinking.

But perhaps the most important outcome is invisible in any single message. The proving engine, which had been a black box running on remote GPUs, was now transparent. Operators could watch proofs flow through the pipeline in real time, spot bottlenecks, detect memory pressure, and verify that all GPU workers were active. The bridge between daemon and dashboard was complete, and the system was finally observable.

In the broader context of the cuzk project, this segment marks a transition. The engine is no longer just a piece of software to be built and deployed — it is a system to be operated, monitored, and maintained. The observability architecture that began with memory management and continued through status tracking has reached its natural conclusion in the operator's dashboard. The proving engine has become a managed service.