From Broken Containers to Living Dashboard: The Full Arc of Debugging and Observability in a Distributed S3 Cluster

Introduction

In the life of a complex software engineering session, there are moments that define the entire trajectory of the work. This chunk of a multi-hour coding session to build a horizontally scalable S3-compatible storage gateway captures one of the most complete arcs imaginable: starting from a test cluster where both Kuri storage nodes crash on startup and the web UI returns "connection refused," and ending with a fully instrumented, visually rich monitoring dashboard that displays real-time throughput, latency, error rates, I/O byte volumes, and cluster topology with color-coded node roles. The journey spans infrastructure debugging, database schema repair, metrics system design, cross-language serialization fixes, and React frontend enhancement—all driven by a combination of systematic engineering and responsive user feedback.

This article synthesizes the work across this chunk, tracing the reasoning, decisions, assumptions, and knowledge that shaped each phase. It is a story about how distributed systems become observable, and how the gap between "the code compiles" and "the system works" is bridged through methodical debugging, architectural thinking, and iterative refinement.

Phase 1: The Cluster That Wouldn't Start

The chunk opens with the test cluster in a state of complete failure. Two Kuri storage nodes are crashing immediately on startup with a Go runtime panic. The S3 proxy on port 8078 returns "Internal Server Error." The web UI on port 9010 returns "connection refused." Three distinct failures, each with its own root cause, combine to make the cluster entirely non-functional.

The Go 1.22 HTTP Route Conflict

The most dramatic failure is the Kuri node panic. The error message is precise and baffling:

panic: pattern "GET /" (registered at /app/server/s3/fx.go:94) conflicts with pattern "/healthz" (registered at /app/server/s3/fx.go:93):
    GET / matches fewer methods than /healthz, but has a more general path pattern

This is a behavioral change introduced in Go 1.22's enhanced ServeMux. The new router implements strict conflict detection: if one pattern matches a superset of HTTP methods but a subset of paths compared to another pattern, the router considers them ambiguous and refuses to start. The code had registered GET / as a catch-all handler alongside a methodless /healthz pattern—a combination that worked in earlier Go versions but now triggers a fatal panic. [24]

The assistant's first attempt at fixing this was a superficial patch that didn't resolve the structural issue. The breakthrough came when the assistant recognized the geometry of the conflict: making both patterns equally specific by adding the GET method prefix to /healthz would satisfy the router's conflict detector. However, even this approach proved insufficient because the S3 server's handler structure required more nuanced routing. The eventual solution was to replace the standard ServeMux entirely with a custom handler that manually dispatches based on r.URL.Path and r.Method, sidestepping the router's strictness altogether. [24][25][26][27]

This debugging episode illustrates a critical lesson: framework upgrades can silently break working code in subtle ways. The Go 1.22 routing changes were well-intentioned—they prevent ambiguous routing configurations that could lead to silent request misrouting—but they also break patterns that were perfectly valid in earlier versions. The assistant's journey from panic message to custom handler is a textbook example of reading error messages carefully, understanding the framework's semantics, and choosing the right level of abstraction for the fix.

The Missing Database Column

While the Kuri nodes were crashing from the route conflict, the S3 proxy was failing for a completely different reason: the S3Objects table in YugabyteDB lacked the node_id column. The proxy's PUT handler, when attempting to insert a new object record, received an "Undefined Column" error from the database, which it surfaced as a generic "Internal Server Error" to the client. [30][31]

The root cause was a schema migration gap. The docker-compose.yml had a db-init section that created the S3Objects table, but the initial schema omitted node_id. The assistant had updated the migration script to include the column, but when the cluster was restarted, the db-init container detected that the keyspace already existed and skipped the schema creation step entirely. The old, column-deficient table remained in place. [30][33]

The fix required manual intervention: dropping the existing tables and recreating them with the correct schema. The assistant connected to YugabyteDB via ycqlsh, ran DROP TABLE statements, and then re-created the S3Objects and MultipartUploads tables with the proper columns including node_id and expires_at. [35][36][37] This manual repair was necessary because the automated migration path lacked idempotency checks that could detect schema drift—a common infrastructure pitfall where initialization scripts check for existence rather than correctness.

The Placeholder Web UI

The third failure was almost comical in its simplicity. The web UI container on port 9010 was not actually serving a web interface. It was a placeholder that printed "Web UI runs on kuri-1" and then ran sleep infinity. The assistant replaced it with an Nginx reverse proxy configuration that forwarded traffic to kuri-1's internal web server, and added a second proxy for kuri-2 on port 9011. [11][86]

This fix was straightforward, but it reveals an important assumption: the Docker Compose configuration was a development artifact that had been set up hastily, with placeholder services where the real implementation hadn't yet been built. The assistant's earlier work had focused on the core data path (S3 proxy → Kuri nodes → database), and the web UI was treated as a secondary concern. The user's discovery that the web UI was non-functional forced the assistant to prioritize observability infrastructure alongside data path correctness.

Phase 2: The Cluster Monitoring Gap

With the cluster now running—both Kuri nodes starting successfully, the S3 proxy accepting requests, and the web UI serving HTML—the user began exploring the monitoring dashboard. What they found was disheartening. The cluster monitoring page displayed a barren message: "No cluster nodes configured. Set up FGW_BACKEND_NODES to see cluster topology." All the charts showed "No data available." The event timeline showed "Invalid Date." [80]

The Stub Implementation

The assistant traced this to rbstor/diag.go, where the ClusterTopology function was a stub. The code literally contained a TODO comment: "Implement actual cluster monitoring when running in distributed mode." It returned empty arrays for proxies and storage nodes. [80][84][85]

This is a classic pattern in software development: the interface is defined, the RPC endpoint is registered, the frontend component is built—but the backend implementation is deferred. The system appears to have a feature (there's a page for it, there's an API endpoint for it), but the feature doesn't work because the actual logic was never written. The TODO comment was a promise that was only now being fulfilled because a real user had encountered the empty dashboard.

Implementing Cluster Topology

The fix required two coordinated changes. First, the ClusterTopology function in diag.go needed to be rewritten to parse the FGW_BACKEND_NODES environment variable, perform HTTP health checks against each backend node's /healthz endpoint, and return structured topology data with proxy and storage node information. Second, the FGW_BACKEND_NODES environment variable needed to be added to the Kuri node configurations in docker-compose.yml so that each storage node knew about its peers. [86][87][88]

The implementation involved writing approximately 50 lines of Go code, adding imports for net/http, os, strings, and time, and iterating through LSP errors to match the struct definitions correctly. The assistant also had to navigate a WebSocket protocol discovery: the RPC endpoint at /rpc/v0 required WebSocket transport rather than plain HTTP POST, which meant that curl tests returned 400 Bad Request until the assistant discovered websocat as the appropriate testing tool. [107][108][109][110]

This WebSocket discovery is a microcosm of distributed systems debugging. The assistant's initial verification attempt—a well-formed JSON-RPC POST request via curl—failed not because the backend code was wrong, but because the testing methodology was mismatched to the transport protocol. The 400 Bad Request was not a dead end; it was a signpost pointing toward WebSocket, toward a deeper understanding of the system architecture, and ultimately toward a working cluster monitoring dashboard. [107]

Phase 3: Building Real-Time Metrics

With the cluster topology now returning data, the user tested the remaining monitoring endpoints and found them all returning empty structures. The RequestThroughput, LatencyDistribution, ErrorRates, ActiveRequests, and ClusterEvents RPCs all returned zeroed-out data. The user's message was succinct: "still empty." [117]

The Metrics Collector

The assistant recognized that the existing diagnostic code was never intended to collect real metrics. It was scaffolding—placeholder logic that returned empty structures to satisfy the interface contract without actually measuring anything. The solution was to build a dedicated metrics collection subsystem from scratch. [119]

The assistant created a new file, rbstor/cluster_metrics.go, implementing a ClusterMetrics struct with a rolling 10-minute window and 10-second intervals. This collector tracks:

Wiring Metrics into the Request Pipeline

Building the metrics collector was only half the work. The collector was a passive data structure—it could store metrics, but nothing was feeding data into it. The assistant had to instrument the actual S3 request handlers to call the collector's recording methods. [126]

This required modifying server/s3/server.go and fx.go to wrap the GET, PUT, and HEAD handlers with metrics recording. The assistant used a decorator pattern: the handler logic is wrapped in code that records latency, tracks active requests, and counts errors before and after the actual handler executes. The decision to instrument at the handler level rather than at a lower level (e.g., in HTTP middleware) reflected the architecture of the S3 server, which uses a custom routing handler to avoid the Go 1.22 ServeMux route conflicts. [126][145]

The instrumentation process was iterative. The assistant had to work through LSP errors—unknown struct fields, leftover code, mismatched function signatures—one by one until the build succeeded. Each error was a signal that the assistant's mental model of the codebase didn't perfectly match the actual code, and each fix brought the implementation closer to correctness.

Verification and Validation

After rebuilding the Docker image and restarting the containers, the assistant verified that every RPC endpoint now returned real data. The RequestThroughput endpoint showed timestamps and request counts. The LatencyDistribution endpoint showed P50, P95, and P99 values. The ErrorRates endpoint showed per-node error statistics. The ActiveRequests endpoint showed current in-flight operations. The ClusterEvents endpoint showed node startup events with proper timestamps. [145]

The verification was thorough: the assistant generated test traffic, queried each endpoint via WebSocket, and confirmed that the data structures were populated. The monitoring dashboard, which had been a ghost town of empty charts and zero-filled tables, now had live data flowing through every visualization.

Phase 4: The JSON Serialization Bridge

With the backend returning real metrics, the user checked the web UI and found it still empty. The topology showed "No cluster nodes configured." The charts displayed "No data available." Recent events showed "Invalid Date." The backend was returning data—the assistant had verified this with websocat RPC calls—but the frontend was refusing to display it. [149]

The PascalCase/camelCase Mismatch

The assistant investigated by reading the React component source code. In ClusterTopology.js, the component destructured:

const { proxies = [], storageNodes = [], dataFlows = [] } = topology;

The frontend expected proxies, storageNodes, and dataFlows—all in camelCase. Then the assistant checked the Go backend's struct definitions in iface/iface_ribs.go:

type ClusterTopology struct {
    Proxies      []ProxyInfo
    StorageNodes []StorageNodeInfo
    DataFlows    []DataFlowInfo
}

Go's default JSON marshaling produces field names that match the struct field names exactly: Proxies, StorageNodes, DataFlows—all in PascalCase. JavaScript's object destructuring is case-sensitive, so const { proxies } = topology would yield undefined when the actual key was Proxies. [149][150][151]

This same mismatch affected every data structure in the monitoring system. ThroughputHistory had fields Timestamps, Total, Reads, Writes that the frontend expected as timestamps, total, reads, writes. ClusterEvent had Timestamp and NodeID that the frontend expected as timestamp and nodeId. Every single chart, every single component, every single display was broken by the same root cause.

The Fix and Its Cascading Effects

The solution was to add json:"camelCaseName" struct tags to every relevant Go struct. This is a standard Go mechanism for controlling JSON serialization field names. The fix touched every struct in the cluster monitoring interface: ClusterTopology, ProxyInfo, StorageNodeInfo, DataFlowInfo, ThroughputHistory, LatencyDistribution, ErrorRates, NodeErrorStats, ActiveRequests, and ClusterEvent. [149][152][153]

But the fix didn't stop at the interface definitions. When the assistant rebuilt the Go binary, the compiler caught a type mismatch in rbstor/cluster_metrics.go. The ByOperation field in LatencyDistribution used an anonymous struct that now had JSON tags, but the code that initialized empty values used a plain anonymous struct without tags. Go's type system treats these as distinct types, so the compiler rejected the assignment. The assistant had to update the initialization code to use the tagged anonymous struct—a cascading consequence of the original change. [154][155][156][157]

This cascading effect is a fascinating property of Go's type system. Adding JSON tags to a struct literal changes its type identity, even though the runtime behavior (JSON serialization) is the only thing that changes. The assistant had to trace through every place where the anonymous struct was constructed and ensure the tags were consistent. This is the kind of subtle, type-system-level debugging that distinguishes experienced Go developers from novices.

Verification and the Phantom Proxy

After rebuilding and redeploying, the assistant verified that the JSON now used camelCase field names. The ClusterTopology RPC returned {"proxies":[...],"storageNodes":[...]}—exactly what the React frontend expected. The RequestThroughput RPC returned {"timestamps":[...],"total":[...],"reads":[...],"writes":[...]}. The frontend could finally parse and render the data. [160][161][162][163][164][165]

However, a new anomaly emerged. The topology showed kuri-2 as the proxy even when querying kuri-1's web UI. Each Kuri node should report itself as the local proxy, not its peer. The assistant investigated and discovered that the ClusterTopology handler was using a hardcoded or incorrectly resolved node identifier. The fix required updating the topology logic to correctly identify the self node based on the FGW_NODE_ID environment variable. [166][167][168][169][170][171][172][173]

This episode illustrates a fundamental truth about distributed systems debugging: fixes often cascade. The assistant fixed a JSON serialization bug, which enabled the frontend to render topology data at all, which in turn revealed a data correctness bug that had been invisible. The topology was always returning the wrong node identity, but no one could see it because the frontend wasn't rendering anything. Each layer can mask errors in the layer below. [166]

Phase 5: User-Driven Dashboard Enhancement

With the monitoring dashboard now displaying data correctly, the user reviewed the interface and provided targeted feedback. The user's message was precise and design-conscious: "Frontend Proxies and Storage Nodes tables don't show live stats or storage. Topology - add visual distinction to S3 Frontend and KuRI nodes. Add a I/O bytes chart. Implove layout." Attached was a screenshot showing the current state of the dashboard. [177]

This message represents a critical transition from "technically working" to "actually useful." The assistant had fixed the plumbing—the data was flowing, the JSON was correctly formatted, the charts were rendering—but the user was evaluating the tap. The dashboard was functional but not yet informative. [177]

The I/O Bytes Chart

The user's request for an I/O bytes chart was particularly insightful. The existing throughput chart tracked requests per second, but request counts alone don't tell you how much data is moving through the system. A node handling many small requests might show high throughput but low I/O, while a node handling large file uploads might show low request counts but high I/O bytes. Both metrics are essential for understanding cluster behavior. [177][180][181]

The assistant implemented this by extending the ClusterMetrics collector with byte-tracking fields (totalReadBytes, totalWriteBytes, intervalReadBytes, intervalWriteBytes, and their rolling history slices). The RecordRead and RecordWrite methods gained a bytes int64 parameter. A new GetIOThroughputHistory method exposed the data as an IOThroughputHistory struct with timestamps, read bytes, and write bytes arrays. [182][183][184][185][186]

The S3 server handlers were updated to pass byte counts to the metrics functions. For writes, the assistant used Content-Length from the HTTP request header, which is readily available. For reads, the response body size isn't known until the response is fully written, so the assistant used a response wrapper approach. This pragmatic trade-off—getting something working over perfect accuracy—reflects the iterative nature of development. [197][198][199][200]

Visual Distinction in the Topology

The topology view was updated to visually distinguish S3 frontend proxies from Kuri storage nodes. The assistant chose color coding: blue for S3 frontend proxies (the stateless routing layer) and green for Kuri storage nodes (the stateful data persistence layer). This is not merely cosmetic—it helps operators quickly understand which nodes are handling which responsibilities in the distributed system. [202]

The visual distinction reinforces the architectural pattern that had been hard-won earlier in the session: the critical separation between stateless routing and stateful storage. The topology component, once updated, would serve not just as a monitoring tool but as a visual affirmation of the system's correct architecture. [202]

The Capstone Edit

The final piece was updating Cluster.js—the main page component—to import and render the new IOThroughputChart, restructure the layout into a two-column grid, and wire everything together. [203]

This capstone edit represents the culmination of a carefully orchestrated sequence spanning over twenty-five messages. The assistant had worked through the changes in dependency order: backend data structures first (since the frontend consumes them), then the metrics collection logic, then the RPC endpoint, then the new chart component, then the existing component updates, and finally the layout integration. Each layer had a solid foundation before the next was built. [203]

Themes and Lessons

The Iterative Nature of Distributed Systems Debugging

The most striking theme across this chunk is the iterative, layered nature of debugging. Each fix revealed a new problem. The Go 1.22 route conflict fix enabled the Kuri nodes to start, which revealed the missing database column. Fixing the column enabled the S3 proxy to work, which revealed the stub cluster topology implementation. Implementing the topology revealed the empty metrics stubs. Building the metrics collector revealed the JSON serialization mismatch. Fixing the serialization revealed the phantom proxy identity bug. Fixing the identity bug enabled the user to evaluate the dashboard, which revealed the need for I/O bytes tracking and visual distinction.

This pattern—where fixing one bug reveals another—is characteristic of complex systems with multiple layers of abstraction. Each layer can mask errors in the layer below. The only way to reach a truly working system is to methodically work through each layer, fixing what's broken and then testing what the fix reveals.

The Importance of User-Driven Testing

The user played a critical role throughout this chunk. The user's initial report that the web UI and S3 proxy were broken set the debugging agenda. The user's discovery of the empty cluster monitoring page drove the implementation of the topology RPC. The user's raw RPC dump ("still empty") exposed the metrics collection gap. The user's screenshot and feature requests shaped the dashboard enhancement phase.

Without the user's active testing and precise reporting, many of these issues would have remained invisible. The stub topology implementation would have stayed a stub. The metrics collector would have remained unwired. The JSON serialization mismatch would have gone undetected. The user's willingness to explore the system, read error messages, and provide detailed feedback was essential to the progress made.

The Data Contract Problem

The JSON serialization mismatch between Go's PascalCase defaults and JavaScript's camelCase conventions is a classic example of the data contract problem in polyglot systems. Two services can both speak "valid JSON" yet fail to communicate because they disagree on naming conventions. The Go backend and React frontend were both correct in isolation—they each followed their ecosystem's conventions—but they were incompatible when connected.

The fix was trivial once the mismatch was identified: a few lines of struct tags. But finding that mismatch required systematic investigation, reading both sides of the wire, and connecting the dots between a blank UI and a capitalization difference. This is a timeless lesson: JSON is not a contract, it is a serialization format. The actual contract must be explicitly defined and enforced.

The Architecture of Observability

The monitoring system built in this chunk follows a clear architectural pattern: data is collected at the point where work happens (the S3 handlers), aggregated in a per-node metrics collector with a rolling time window, exposed through WebSocket JSON-RPC endpoints, and rendered by React frontend components. Each layer has a clear responsibility, and the data flows through a well-defined pipeline.

This architecture is not accidental. The assistant made deliberate design decisions at each layer: the 10-minute rolling window for temporal resolution, the per-node collector for isolation, the WebSocket transport for real-time updates, the camelCase JSON tags for frontend compatibility. These decisions reflect an understanding of what makes a monitoring system useful: timeliness, accuracy, clarity, and responsiveness to operator needs.

Conclusion

This chunk of the coding session captures a complete arc of distributed systems development: from a broken, non-functional test cluster to a fully instrumented, visually rich monitoring dashboard with real-time metrics, color-coded topology, and byte-level I/O tracking. The journey required debugging infrastructure failures (Go 1.22 route conflicts, missing database columns, placeholder containers), implementing core observability features (cluster topology, metrics collection, RPC endpoints), fixing cross-language integration issues (JSON serialization mismatches, WebSocket protocol discovery), and responding to user feedback with targeted enhancements (I/O bytes charts, visual distinction, layout improvements).

The work demonstrates that building observable distributed systems is not a single task but a continuous process of discovery, implementation, and refinement. Each layer of the system—the container infrastructure, the database schema, the Go backend, the RPC layer, the React frontend—must be individually correct and correctly integrated with every other layer. The debugging process is not linear but iterative: each fix reveals the next problem, and the system only becomes fully functional through persistent, methodical effort.

The final state of the cluster—with both Kuri nodes reporting live metrics, the S3 proxy routing requests correctly, the web UI displaying real-time charts with camelCase JSON, and the topology view distinguishing proxies from storage nodes in blue and green—represents not just a working system but a observable one. And in distributed systems, observability is not a feature. It is a necessity.