From Debugging to Observability: Building a Self-Revealing Distributed S3 Cluster

Introduction

There is a moment in every complex software project when the system stops being a collection of code that should work and becomes a system that does work. The transition is rarely dramatic. It happens through a thousand small corrections: a route handler rewritten, a database column added, a JSON tag inserted, a CSS class renamed. Each correction is individually unremarkable, but their cumulative effect is transformative. The system becomes not just functional but observable—capable of revealing its own internal state, performance characteristics, and failure modes to the humans who operate it.

This article examines a concentrated sequence of work within a larger coding session to build a horizontally scalable S3-compatible storage gateway. The segment spans the full arc from a broken test cluster—where both storage nodes crash on startup, the web UI returns "connection refused," and the S3 proxy returns "Internal Server Error"—to a fully instrumented monitoring dashboard with real-time throughput charts, color-coded topology views, and byte-level I/O tracking. The journey is a masterclass in layered debugging, cross-language integration, and the iterative construction of observability.

What makes this segment particularly instructive is the recursive nature of the debugging process. Each fix revealed a new problem at a different layer of the system. Fixing the Go 1.22 route conflict enabled the storage nodes to start, which revealed a missing database column. Fixing the column enabled the S3 proxy to work, which revealed the stub cluster topology implementation. Implementing the topology revealed the empty metrics collectors. Building the collectors revealed the JSON serialization mismatch between Go and React. Fixing the serialization revealed the phantom proxy identity bug. And fixing the identity bug enabled the user to evaluate the dashboard, which revealed the need for I/O byte tracking, visual distinction between node roles, and proper terminology in latency charts.

This pattern—where fixing one problem reveals the next—is not a sign of poor engineering. It is the natural behavior of complex layered systems. Each layer can mask errors in the layer below it. The only way to reach a truly working, observable system is to methodically work through each layer, fixing what is broken and then testing what the fix reveals. This article traces that journey layer by layer, extracting the architectural decisions, debugging methodologies, and integration patterns that transformed a broken test cluster into a living, self-revealing distributed system.

Layer 1: Infrastructure — The Foundation That Had to Hold

The Go 1.22 Route Conflict

The segment opens with the test cluster in a state of complete failure. Two Kuri storage nodes are crashing on startup with a Go runtime panic. The error message is precise and baffling:

panic: pattern "GET /" (registered at /app/server/s3/fx.go:94) conflicts with pattern "/healthz" (registered at /app/server/s3/fx.go:93):
    GET / matches fewer methods than /healthz, but has a more general path pattern

This is a behavioral change introduced in Go 1.22's enhanced ServeMux. The new router implements strict conflict detection: if one pattern matches a superset of HTTP methods but a subset of paths compared to another pattern, the router considers them ambiguous and refuses to start. The code had registered GET / as a catch-all handler alongside a methodless /healthz pattern—a combination that worked in earlier Go versions but now triggers a fatal panic. [1]

The assistant's first attempt at fixing this was a superficial patch that didn't resolve the structural issue. The breakthrough came when the assistant recognized the geometry of the conflict: making both patterns equally specific by adding the GET method prefix to /healthz would satisfy the router's conflict detector. However, even this approach proved insufficient because the S3 server's handler structure required more nuanced routing. The eventual solution was to replace the standard ServeMux entirely with a custom handler that manually dispatches based on r.URL.Path and r.Method, sidestepping the router's strictness altogether. [1]

This debugging episode illustrates a critical lesson about framework upgrades. Go 1.22's routing changes were well-intentioned—they prevent ambiguous routing configurations that could lead to silent request misrouting—but they also break patterns that were perfectly valid in earlier versions. The assistant's journey from panic message to custom handler is a textbook example of reading error messages carefully, understanding the framework's semantics, and choosing the right level of abstraction for the fix. The custom handler, while more verbose, gives the team complete control over routing behavior and eliminates the risk of future framework-level routing surprises.

The Missing Database Column

While the Kuri nodes were crashing from the route conflict, the S3 proxy was failing for a completely different reason: the S3Objects table in YugabyteDB lacked the node_id column. The proxy's PUT handler, when attempting to insert a new object record, received an "Undefined Column" error from the database, which it surfaced as a generic "Internal Server Error" to the client. [1]

The root cause was a schema migration gap. The docker-compose.yml had a db-init section that created the S3Objects table, but the initial schema omitted node_id. The assistant had updated the migration script to include the column, but when the cluster was restarted, the db-init container detected that the keyspace already existed and skipped the schema creation step entirely. The old, column-deficient table remained in place. [1]

The fix required manual intervention: dropping the existing tables and recreating them with the correct schema. The assistant connected to YugabyteDB via ycqlsh, ran DROP TABLE statements, and then re-created the S3Objects and MultipartUploads tables with the proper columns including node_id and expires_at. [1] This manual repair was necessary because the automated migration path lacked idempotency checks that could detect schema drift—a common infrastructure pitfall where initialization scripts check for existence rather than correctness.

The Placeholder Web UI

The third infrastructure failure was almost comical in its simplicity. The web UI container on port 9010 was not actually serving a web interface. It was a placeholder that printed "Web UI runs on kuri-1" and then ran sleep infinity. The assistant replaced it with an Nginx reverse proxy configuration that forwarded traffic to kuri-1's internal web server, and added a second proxy for kuri-2 on port 9011. [1]

This fix was straightforward, but it reveals an important assumption: the Docker Compose configuration was a development artifact that had been set up hastily, with placeholder services where the real implementation hadn't yet been built. The assistant's earlier work had focused on the core data path (S3 proxy → Kuri nodes → database), and the web UI was treated as a secondary concern. The user's discovery that the web UI was non-functional forced the assistant to prioritize observability infrastructure alongside data path correctness.

Layer 2: Monitoring — Building the Observability Pipeline

The Stub Implementation

With the cluster now running—both Kuri nodes starting successfully, the S3 proxy accepting requests, and the web UI serving HTML—the user began exploring the monitoring dashboard. What they found was disheartening. The cluster monitoring page displayed a barren message: "No cluster nodes configured. Set up FGW_BACKEND_NODES to see cluster topology." All the charts showed "No data available." The event timeline showed "Invalid Date." [1]

The assistant traced this to rbstor/diag.go, where the ClusterTopology function was a stub. The code literally contained a TODO comment: "Implement actual cluster monitoring when running in distributed mode." It returned empty arrays for proxies and storage nodes. [1]

This is a classic pattern in software development: the interface is defined, the RPC endpoint is registered, the frontend component is built—but the backend implementation is deferred. The system appears to have a feature (there's a page for it, there's an API endpoint for it), but the feature doesn't work because the actual logic was never written. The TODO comment was a promise that was only now being fulfilled because a real user had encountered the empty dashboard.

The Metrics Collector

The assistant recognized that the existing diagnostic code was never intended to collect real metrics. It was scaffolding—placeholder logic that returned empty structures to satisfy the interface contract without actually measuring anything. The solution was to build a dedicated metrics collection subsystem from scratch. [1]

The assistant created a new file, rbstor/cluster_metrics.go, implementing a ClusterMetrics struct with a rolling 10-minute window and 10-second intervals. This collector tracks:

The Cluster Topology RPC

The ClusterTopology function in diag.go needed to be rewritten to parse the FGW_BACKEND_NODES environment variable, perform HTTP health checks against each backend node's /healthz endpoint, and return structured topology data with proxy and storage node information. The FGW_BACKEND_NODES environment variable also needed to be added to the Kuri node configurations in docker-compose.yml so that each storage node knew about its peers. [1]

The implementation involved writing approximately 50 lines of Go code, adding imports for net/http, os, strings, and time, and iterating through LSP errors to match the struct definitions correctly. The assistant also had to navigate a WebSocket protocol discovery: the RPC endpoint at /rpc/v0 required WebSocket transport rather than plain HTTP POST, which meant that curl tests returned 400 Bad Request until the assistant discovered websocat as the appropriate testing tool. [1]

This WebSocket discovery is a microcosm of distributed systems debugging. The assistant's initial verification attempt—a well-formed JSON-RPC POST request via curl—failed not because the backend code was wrong, but because the testing methodology was mismatched to the transport protocol. The 400 Bad Request was not a dead end; it was a signpost pointing toward WebSocket, toward a deeper understanding of the system architecture, and ultimately toward a working cluster monitoring dashboard. [1]

Layer 3: Integration — The JSON Bridge Between Go and React

The PascalCase/camelCase Mismatch

With the backend returning real metrics, the user checked the web UI and found it still empty. The topology showed "No cluster nodes configured." The charts displayed "No data available." Recent events showed "Invalid Date." The backend was returning data—the assistant had verified this with websocat RPC calls—but the frontend was refusing to display it. [1]

The assistant investigated by reading the React component source code. In ClusterTopology.js, the component destructured:

const { proxies = [], storageNodes = [], dataFlows = [] } = topology;

The frontend expected proxies, storageNodes, and dataFlows—all in camelCase. Then the assistant checked the Go backend's struct definitions in iface/iface_ribs.go:

type ClusterTopology struct {
    Proxies      []ProxyInfo
    StorageNodes []StorageNodeInfo
    DataFlows    []DataFlowInfo
}

Go's default JSON marshaling produces field names that match the struct field names exactly: Proxies, StorageNodes, DataFlows—all in PascalCase. JavaScript's object destructuring is case-sensitive, so const { proxies } = topology would yield undefined when the actual key was Proxies. [1]

This same mismatch affected every data structure in the monitoring system. ThroughputHistory had fields Timestamps, Total, Reads, Writes that the frontend expected as timestamps, total, reads, writes. ClusterEvent had Timestamp and NodeID that the frontend expected as timestamp and nodeId. Every single chart, every single component, every single display was broken by the same root cause.

The Fix and Its Cascading Effects

The solution was to add json:"camelCaseName" struct tags to every relevant Go struct. This is a standard Go mechanism for controlling JSON serialization field names. The fix touched every struct in the cluster monitoring interface: ClusterTopology, ProxyInfo, StorageNodeInfo, DataFlowInfo, ThroughputHistory, LatencyDistribution, ErrorRates, NodeErrorStats, ActiveRequests, and ClusterEvent. [1]

But the fix didn't stop at the interface definitions. When the assistant rebuilt the Go binary, the compiler caught a type mismatch in rbstor/cluster_metrics.go. The ByOperation field in LatencyDistribution used an anonymous struct that now had JSON tags, but the code that initialized empty values used a plain anonymous struct without tags. Go's type system treats these as distinct types, so the compiler rejected the assignment. The assistant had to update the initialization code to use the tagged anonymous struct—a cascading consequence of the original change. [1]

This cascading effect is a fascinating property of Go's type system. Adding JSON tags to a struct literal changes its type identity, even though the runtime behavior (JSON serialization) is the only thing that changes. The assistant had to trace through every place where the anonymous struct was constructed and ensure the tags were consistent. This is the kind of subtle, type-system-level debugging that distinguishes experienced Go developers from novices.

The Phantom Proxy

After rebuilding and redeploying, the assistant verified that the JSON now used camelCase field names. The ClusterTopology RPC returned {"proxies":[...],"storageNodes":[...]}—exactly what the React frontend expected. The RequestThroughput RPC returned {"timestamps":[...],"total":[...],"reads":[...],"writes":[...]}. The frontend could finally parse and render the data. [1]

However, a new anomaly emerged. The topology showed kuri-2 as the proxy even when querying kuri-1's web UI. Each Kuri node should report itself as the local proxy, not its peer. The assistant investigated and discovered that the ClusterTopology handler was using a hardcoded or incorrectly resolved node identifier. The fix required updating the topology logic to correctly identify the self node based on the FGW_NODE_ID environment variable. [1]

This episode illustrates a fundamental truth about distributed systems debugging: fixes often cascade. The assistant fixed a JSON serialization bug, which enabled the frontend to render topology data at all, which in turn revealed a data correctness bug that had been invisible. The topology was always returning the wrong node identity, but no one could see it because the frontend wasn't rendering anything. Each layer can mask errors in the layer below. [1]

Layer 4: Verification — Proving the System Works

The Verification Loop

With the monitoring dashboard now displaying data correctly, the user reviewed the interface and provided targeted feedback. The user's message was precise and design-conscious: "Frontend Proxies and Storage Nodes tables don't show live stats or storage. Topology - add visual distinction to S3 Frontend and KuRI nodes. Add a I/O bytes chart. Implove layout." Attached was a screenshot showing the current state of the dashboard. [2]

This message represents a critical transition from "technically working" to "actually useful." The assistant had fixed the plumbing—the data was flowing, the JSON was correctly formatted, the charts were rendering—but the user was evaluating the tap. The dashboard was functional but not yet informative. [2]

The verification loop that followed is documented in detail in the companion article "The Verification Loop: How a Distributed S3 Monitoring System Proved Itself Through Iteration" [2]. The assistant generated real traffic—ten PUT operations of 10KB random files followed by ten GET operations—to test whether the I/O byte tracking mechanism actually captured data. The choice of 10KB payloads was intentional: large enough to produce clearly measurable byte counts (100KB total in each direction) yet small enough to execute quickly. The use of /dev/urandom ensured the data was incompressible and uncacheable, genuinely exercising the write and read paths. [2]

The assistant's approach to verification reveals a systematic methodology. Rather than loading the web UI in a browser and visually inspecting charts—which would be the end-user perspective—the assistant went directly to the data layer, using websocat to make raw WebSocket JSON-RPC calls and piping results through jq. This preference for direct data inspection over visual inspection is faster, more precise, and eliminates the possibility of frontend rendering bugs masking backend issues. [2]

The Build Pipeline Discovery

The most consequential verification moment came when the user, having observed the running web UI, asked a pointed question: "Don't see layout fixes nor IO chart, not deployed?" This single observation exposed a critical gap in the assistant's mental model of the deployment pipeline. [2]

The assistant's response reveals the moment of realization: "You're right, let me check if the frontend was actually rebuilt and deployed. The Docker build copies the pre-built React app, so I need to rebuild the React app first." The assistant had edited React source files—new components, updated layouts, CSS changes—and then rebuilt the Docker image. But the Dockerfile's COPY instruction consumed the pre-built build/ directory, which contained stale compiled assets from a previous npm run build. The Docker build succeeded, the containers restarted, but the frontend was serving old code. [2]

This is a classic CI/CD trap. The Go backend used a Docker multi-stage build that compiled binaries from source, creating a self-contained build process. The React frontend, however, used an external build step that produced static files outside the Docker build context. The assistant treated both as equivalent, but they were not. The user's sharp observation—coupled with the willingness to question rather than assume—caught the gap and forced the correction. [2]

The SLO Edit

With the deployment pipeline corrected, the assistant addressed the user's second request: renaming "SLA" to "SLO" in the latency distribution chart and setting the threshold to 350ms. [2]

The rename from "SLA" to "SLO" is not cosmetic. In cloud infrastructure, a Service Level Agreement (SLA) is a contractual commitment with legal and financial consequences. A Service Level Objective (SLO) is an internal target used for operational decision-making. Displaying "SLA" on an internal monitoring dashboard for a test cluster was technically incorrect—there is no customer-facing contract for a development testbed. The user's correction reflects attention to conceptual accuracy and proper operational terminology. [2]

The 350ms threshold is also significant. It represents a deliberate engineering decision about what constitutes acceptable latency for this system. The assistant had likely used a default value (perhaps 200ms or 500ms) without specific justification. The user's choice of 350ms suggests a considered judgment based on the system's performance characteristics and operational requirements. [2]

Layer 5: Refinement — The User-Driven Polish

The I/O Bytes Chart

The user's request for an I/O bytes chart was particularly insightful. The existing throughput chart tracked requests per second, but request counts alone don't tell you how much data is moving through the system. A node handling many small requests might show high throughput but low I/O, while a node handling large file uploads might show low request counts but high I/O bytes. Both metrics are essential for understanding cluster behavior. [1]

The assistant implemented this by extending the ClusterMetrics collector with byte-tracking fields (totalReadBytes, totalWriteBytes, intervalReadBytes, intervalWriteBytes, and their rolling history slices). The RecordRead and RecordWrite methods gained a bytes int64 parameter. A new GetIOThroughputHistory method exposed the data as an IOThroughputHistory struct with timestamps, read bytes, and write bytes arrays. [1]

The S3 server handlers were updated to pass byte counts to the metrics functions. For writes, the assistant used Content-Length from the HTTP request header, which is readily available. For reads, the response body size isn't known until the response is fully written, so the assistant used a response wrapper approach. This pragmatic trade-off—getting something working over perfect accuracy—reflects the iterative nature of development. [1]

Visual Distinction in the Topology

The topology view was updated to visually distinguish S3 frontend proxies from Kuri storage nodes. The assistant chose color coding: blue for S3 frontend proxies (the stateless routing layer) and green for Kuri storage nodes (the stateful data persistence layer). This is not merely cosmetic—it helps operators quickly understand which nodes are handling which responsibilities in the distributed system. [1]

The visual distinction reinforces the architectural pattern that had been hard-won earlier in the session: the critical separation between stateless routing and stateful storage. The topology component, once updated, serves not just as a monitoring tool but as a visual affirmation of the system's correct architecture. [1]

The Capstone Edit

The final piece was updating Cluster.js—the main page component—to import and render the new IOThroughputChart, restructure the layout into a two-column grid, and wire everything together. [1]

This capstone edit represents the culmination of a carefully orchestrated sequence spanning dozens of messages. The assistant had worked through the changes in dependency order: backend data structures first (since the frontend consumes them), then the metrics collection logic, then the RPC endpoint, then the new chart component, then the existing component updates, and finally the layout integration. Each layer had a solid foundation before the next was built. [1]

Themes and Lessons

The Recursive Nature of Debugging

The most striking theme across this segment is the recursive, layered nature of debugging. Each fix revealed a new problem at a different layer of the system. The Go 1.22 route conflict fix enabled the Kuri nodes to start, which revealed the missing database column. Fixing the column enabled the S3 proxy to work, which revealed the stub cluster topology implementation. Implementing the topology revealed the empty metrics stubs. Building the metrics collector revealed the JSON serialization mismatch. Fixing the serialization revealed the phantom proxy identity bug. Fixing the identity bug enabled the user to evaluate the dashboard, which revealed the need for I/O bytes tracking and visual distinction.

This pattern—where fixing one bug reveals another—is characteristic of complex systems with multiple layers of abstraction. Each layer can mask errors in the layer below. The only way to reach a truly working system is to methodically work through each layer, fixing what's broken and then testing what the fix reveals.

The Data Contract Problem

The JSON serialization mismatch between Go's PascalCase defaults and JavaScript's camelCase conventions is a classic example of the data contract problem in polyglot systems. Two services can both speak "valid JSON" yet fail to communicate because they disagree on naming conventions. The Go backend and React frontend were both correct in isolation—they each followed their ecosystem's conventions—but they were incompatible when connected.

The fix was trivial once the mismatch was identified: a few lines of struct tags. But finding that mismatch required systematic investigation, reading both sides of the wire, and connecting the dots between a blank UI and a capitalization difference. This is a timeless lesson: JSON is not a contract, it is a serialization format. The actual contract must be explicitly defined and enforced.

The Importance of User-Driven Testing

The user played a critical role throughout this segment. The user's initial report that the web UI and S3 proxy were broken set the debugging agenda. The user's discovery of the empty cluster monitoring page drove the implementation of the topology RPC. The user's raw RPC dump ("still empty") exposed the metrics collection gap. The user's screenshot and feature requests shaped the dashboard enhancement phase. The user's question "not deployed?" caught the build pipeline gap.

Without the user's active testing and precise reporting, many of these issues would have remained invisible. The stub topology implementation would have stayed a stub. The metrics collector would have remained unwired. The JSON serialization mismatch would have gone undetected. The build pipeline gap would have persisted. The user's willingness to explore the system, read error messages, and provide detailed feedback was essential to the progress made.

The Architecture of Observability

The monitoring system built in this segment follows a clear architectural pattern: data is collected at the point where work happens (the S3 handlers), aggregated in a per-node metrics collector with a rolling time window, exposed through WebSocket JSON-RPC endpoints, and rendered by React frontend components. Each layer has a clear responsibility, and the data flows through a well-defined pipeline.

This architecture is not accidental. The assistant made deliberate design decisions at each layer: the 10-minute rolling window for temporal resolution, the per-node collector for isolation, the WebSocket transport for real-time updates, the camelCase JSON tags for frontend compatibility. These decisions reflect an understanding of what makes a monitoring system useful: timeliness, accuracy, clarity, and responsiveness to operator needs.

Conclusion

This segment of the coding session captures a complete arc of distributed systems development: from a broken, non-functional test cluster to a fully instrumented, visually rich monitoring dashboard with real-time metrics, color-coded topology, and byte-level I/O tracking. The journey required debugging infrastructure failures (Go 1.22 route conflicts, missing database columns, placeholder containers), implementing core observability features (cluster topology, metrics collection, RPC endpoints), fixing cross-language integration issues (JSON serialization mismatches, WebSocket protocol discovery), and responding to user feedback with targeted enhancements (I/O bytes charts, visual distinction, layout improvements).

The work demonstrates that building observable distributed systems is not a single task but a continuous process of discovery, implementation, and refinement. Each layer of the system—the container infrastructure, the database schema, the Go backend, the RPC layer, the React frontend—must be individually correct and correctly integrated with every other layer. The debugging process is not linear but recursive: each fix reveals the next problem, and the system only becomes fully functional through persistent, methodical effort.

The final state of the cluster—with both Kuri nodes reporting live metrics, the S3 proxy routing requests correctly, the web UI displaying real-time charts with camelCase JSON, and the topology view distinguishing proxies from storage nodes in blue and green—represents not just a working system but an observable one. And in distributed systems, observability is not a feature. It is a necessity.

References

[1] "From Broken Containers to Living Dashboard: The Full Arc of Debugging and Observability in a Distributed S3 Cluster" — Chunk 0 article covering the full debugging arc from infrastructure fixes through metrics implementation and frontend enhancement.

[2] "The Verification Loop: How a Distributed S3 Monitoring System Proved Itself Through Iteration" — Chunk 1 article covering the iterative verification process, build pipeline discovery, and SLO terminology correction.