The Moment the Metrics Came Alive: Debugging Distributed I/O Tracking in a Multi-Node S3 Cluster

In the middle of a lengthy coding session to build a horizontally scalable S3 architecture with separate stateless frontend proxies and Kuri storage nodes, a single message captures the precise moment when a developer's assumption collides with the reality of distributed systems. Message 787 in this conversation is deceptively simple: it contains two RPC queries to two different web UI endpoints, followed by their JSON responses. But behind this brief exchange lies a rich story about debugging, distributed architecture, and the subtle ways that data flows through a multi-node system.

The Context: Building Real-Time Cluster Monitoring

The assistant had just completed a significant implementation phase. The test cluster consisted of three layers: an S3 frontend proxy on port 8078 that routes requests to backend Kuri storage nodes, which in turn store data in a shared YugabyteDB database. A React-based monitoring dashboard had been built to display cluster topology, request throughput, latency distributions, error rates, and—most recently—I/O byte throughput. The user had specifically requested an "I/O bytes chart" in message 751, and the assistant had spent messages 754 through 773 implementing the backend tracking infrastructure: adding IOThroughputHistory types to the interface definitions, extending the ClusterMetrics struct to accumulate read and write byte counters, updating the S3 server handlers to pass byte counts to RecordRead and RecordWrite, and wiring the new RPC method through the web layer.

After rebuilding the Docker image and restarting the containers, the assistant ran a quick test: uploading ten 10KB files via curl to the S3 proxy, then reading them back. The initial result, visible in message 785, was puzzling:

{
  "timestamps": [1769868303],
  "readBytes": [0],
  "writeBytes": [0],
  "totalBytes": [0]
}

All zeros. The I/O tracking appeared to be completely broken.

The Assumption and Its Breakdown

Message 786 shows the assistant's immediate hypothesis: "The I/O isn't being tracked. That's because the traffic went through the S3 proxy, not directly to the kuri nodes." This is a reasonable assumption. The architecture has a clear separation: the S3 frontend proxy on port 8078 receives client requests and proxies them to Kuri storage nodes on their internal ports. The metrics collection lives inside the Kuri nodes themselves, in the rbstor/cluster_metrics.go file. If the proxy is handling the request without forwarding it properly, or if the byte tracking code in the Kuri node's S3 server handler isn't being triggered, the metrics would indeed show zeros.

The assistant attempts to verify this by trying to upload directly to a Kuri node from inside the Docker network, using docker exec to run wget inside the proxy container. This fails—BusyBox's wget doesn't support PUT requests. The assistant then tries to inspect the X-Node-Id response header to see which backend node handled each request, but the output is truncated.

The Insight: Querying Both Nodes

Message 787 is where the story turns. Instead of continuing down the path of assuming the tracking is broken, the assistant makes a critical decision: query both web UI endpoints. The test cluster has two web UI containers: one on port 9010 (connected to kuri-1) and another on port 9011 (connected to kuri-2). The assistant runs the same RIBS.IOThroughput RPC call against both endpoints.

The results reveal the truth:

Port 9010 (kuri-1):

{
  "timestamps": [1769868303, 1769868314, 1769868325],
  "readBytes": [0, 5120, 0],
  "writeBytes": [0, 5120, 2048],
  "totalBytes": [0, 10240, 2048]
}

Port 9011 (kuri-2):

{
  "timestamps": [1769868325],
  "readBytes": [5120],
  "writeBytes": [8192],
  "totalBytes": [13312]
}

The metrics are being tracked. They're just distributed across both nodes. The S3 proxy, as designed, routes requests to different backend nodes based on its load-balancing or routing logic. Some of the test uploads went to kuri-1, some to kuri-2. When the assistant queried only port 9010 (kuri-1) in message 785, the metrics from requests handled by kuri-2 were invisible. The system was working correctly all along—the assistant was just looking at the wrong node.

The Distributed Systems Lesson

This message embodies a fundamental principle of distributed systems debugging: in a multi-node architecture, you cannot assume that any single node's view represents the whole system. The assistant's initial assumption—that querying one web UI endpoint would show all metrics—was natural but incorrect. The S3 proxy distributes requests across storage nodes, and each node independently tracks its own metrics. The monitoring dashboard, which connects to a single Kuri node's web UI, only shows that node's local view.

The architecture here is worth examining. The web UI containers are reverse proxies that connect to individual Kuri nodes. Port 9010 proxies to kuri-1's RPC endpoint, port 9011 to kuri-2's. This means the monitoring dashboard is inherently per-node, not aggregated. To see the full cluster picture, you'd need either an aggregated endpoint that collects metrics from all nodes, or you'd need to query each node separately and combine the results. The assistant's decision to query both ports was the correct debugging approach.

Input Knowledge Required

To fully understand this message, several pieces of context are necessary. First, the architecture of the test cluster: three layers with S3 proxies, Kuri storage nodes, and YugabyteDB. Second, the RPC mechanism: the RIBS.IOThroughput method returns historical I/O byte data with timestamps. Third, the web UI setup: port 9010 connects to kuri-1, port 9011 to kuri-2. Fourth, the recent implementation work: the IOThroughputHistory type, the byte tracking in RecordRead/RecordWrite, and the GetIOThroughputHistory method in cluster_metrics.go. Fifth, the test procedure: uploading and downloading files via curl to generate traffic.

Output Knowledge Created

This message produces several important pieces of knowledge. First and most concretely, it confirms that the I/O byte tracking implementation is functional—the RecordRead and RecordWrite calls in the S3 server handlers are correctly accumulating byte counts, the interval rotation logic in ClusterMetrics is working, and the RPC endpoint is returning properly formatted JSON. Second, it reveals the routing behavior of the S3 proxy: requests are being distributed across both Kuri nodes, confirming that the load-balancing layer is operational. Third, it identifies a gap in the monitoring architecture: the per-node web UI cannot provide a unified cluster view. Fourth, it demonstrates a debugging methodology: when a distributed system appears broken, check each node individually before concluding the feature is defective.

The Thinking Process

The assistant's reasoning in this message is a model of systematic debugging. The sequence is:

1. Observe symptom: I/O metrics show zeros on port 9010.
2. Form hypothesis: The traffic went through the S3 proxy, not directly to Kuri nodes, so the metrics aren't being recorded.
3. Test hypothesis: Try to upload directly to a Kuri node (fails due to tool limitations).
4. Refine approach: Instead of trying to bypass the proxy, check if metrics exist on the other node.
5. Execute: Query port 9011 (kuri-2) with the same RPC call.
6. Analyze results: Both nodes have metrics. The system is working. The problem was the single-node view. This is a classic debugging pattern: when a system appears broken, verify that your observation method is correct before concluding the system is faulty. The assistant could have spent significant time digging into the S3 proxy code, adding logging, or rebuilding containers. Instead, a simple check—querying the other node—solved the mystery in seconds.

The Broader Significance

This message, while brief, marks a transition point in the session. Before it, the assistant was in implementation mode: writing code, fixing compilation errors, and deploying changes. After it, the focus shifts to verification and refinement: confirming that the monitoring dashboard displays real data, improving the frontend layout, and eventually renaming "SLA" to "SLO" with a 350ms threshold. The I/O tracking was the last major backend feature to be implemented and verified. Once confirmed working, the assistant could move on to polishing the user experience.

The message also reveals something about the nature of the system being built. This is not a monolithic application where a single process handles all requests and a single metrics endpoint tells the whole story. It is a genuinely distributed system, with independent nodes that each maintain their own state and metrics. The fact that the assistant initially forgot to check both nodes is a reminder that distributed thinking requires constant vigilance. Even experienced developers, when deep in implementation mode, can default to single-system assumptions.

Conclusion

Message 787 is a small but pivotal moment in a larger coding session. It is the point where a potentially confusing bug—"I/O tracking is broken"—resolves into a simple misunderstanding about distributed observation. The metrics were never broken; they were just spread across two nodes. The assistant's decision to query both web UI endpoints, rather than continuing to debug a non-existent problem, demonstrates efficient troubleshooting. The message stands as a testament to the importance of understanding your system's architecture before diagnosing its behavior, and a reminder that in distributed systems, the first place you look is not always the right place.