The Moment the Metrics Came Alive: Verifying Real-Time Latency Distribution in a Distributed S3 Cluster

Introduction

In the course of building a horizontally scalable S3 architecture with a three-layer hierarchy of stateless frontend proxies, Kuri storage nodes, and a shared YugabyteDB backend, there comes a pivotal moment when the abstractions prove themselves. Message 717 in this coding session captures exactly that moment: a single command sent via websocat to a WebSocket RPC endpoint, and the response that confirms the cluster's monitoring infrastructure is finally producing real, meaningful data. This message is not merely a test—it is the culmination of a debugging and implementation effort that transformed empty stub responses into a live metrics pipeline.

Why This Message Was Written

The immediate trigger for message 717 was a series of earlier verifications. In message 691, the user had pasted the raw output of multiple RPC calls made from the React frontend's monitoring dashboard. Every single metrics endpoint—RIBS.RequestThroughput, RIBS.LatencyDistribution, RIBS.ErrorRates, RIBS.ActiveRequests, and RIBS.ClusterEvents—returned empty arrays, zero values, or empty objects. The ClusterTopology RPC correctly listed both storage nodes as healthy, but the dashboard was essentially a skeleton with no vital signs. The user's comment "still empty" at the end of that message made the problem explicit: the cluster monitoring UI existed, the node list populated, but the actual operational metrics that make monitoring useful were nowhere to be found.

The assistant recognized that the metrics endpoints were still stub implementations in rbstor/diag.go. Rather than patching individual return values, the assistant undertook a systematic effort to build a proper metrics collection subsystem. This involved creating a new file rbstor/cluster_metrics.go with a ClusterMetrics collector that tracks throughput, latency, error rates, active requests, and I/O bytes with a rolling time window. It required wiring metrics recording into the S3 server handlers so that every PUT, GET, DELETE, and LIST operation would be instrumented. It meant adding event logging for node lifecycle events. And it demanded careful attention to the data structures so that the JSON serialization matched what the React frontend expected.

Message 717 is the final verification step in that chain. After rebuilding the Docker image, restarting the containers, uploading ten test objects, and performing ten read operations, the assistant methodically checked each metrics endpoint one by one. Message 713 confirmed RequestThroughput returned data. Message 714 confirmed ClusterEvents recorded a node start event. Message 715 confirmed ActiveRequests showed zero (correct, since no requests were in flight). Message 716 confirmed ErrorRates showed zero errors with a stable trend. Message 717—the subject of this article—confirms LatencyDistribution returns real percentile data with timestamps. Every endpoint now produces live metrics.

The Technical Decisions Embedded in This Message

Though the message itself is a simple command and response, it encodes several architectural decisions that were made during the implementation. The choice to use a rolling 10-minute window for metrics aggregation, with per-second resolution, reflects a design trade-off between memory usage and granularity. The decision to track P50, P95, and P99 latency percentiles rather than just averages shows an understanding that tail latency matters more than mean latency in distributed systems. The fact that ByOperation is empty in the response reveals that per-operation-type breakdowns were planned but not yet populated—a conscious scope decision to get the basic pipeline working first.

The use of websocat as the testing tool is itself a decision. Earlier in the session, the assistant attempted to use curl to test the RPC endpoint and received "Bad Request" responses. The assistant then discovered that the RPC endpoint required WebSocket transport, not plain HTTP POST. This led to installing websocat and using it for all subsequent RPC testing. Message 717 continues that pattern, demonstrating that the testing methodology had stabilized around a reliable tool.

What the Response Data Reveals

The response in message 717 contains two data points at timestamps 1769867189 and 1769867201—approximately 12 seconds apart. The first data point shows P50 latency of 13 milliseconds, P95 of 15ms, and P99 of 15ms. The second shows dramatically lower latencies: P50 of 2ms, P95 of 3ms, and P99 of 3ms. This pattern is characteristic of a system experiencing cold-start effects. The first requests after a restart encounter initialization overhead—connection pool warming, cache misses, database connection establishment—while subsequent requests benefit from warmed caches and established connections. The fact that the second data point shows latencies dropping by roughly 80% is strong evidence that the metrics collection is working correctly and capturing real system behavior.

The empty ByOperation field indicates that while overall latency percentiles are being tracked, the breakdown by S3 operation type (GET, PUT, DELETE, LIST) has not yet been implemented. This is consistent with the assistant's incremental approach: get the aggregate metrics working first, then add per-operation granularity later.

Assumptions and Potential Blind Spots

The implementation makes several assumptions worth examining. It assumes that the S3 proxy's latency measurements are representative of the full request path, including the backend Kuri storage nodes. If the proxy itself introduces significant overhead, the reported latencies would not reflect storage node performance. It assumes that a rolling 10-minute window is sufficient for the monitoring dashboard—if the dashboard needs to show historical trends over hours or days, the in-memory metrics would need to be persisted. It assumes that the FGW_BACKEND_NODES environment variable correctly describes the cluster topology, which requires manual configuration and could become out of sync with the actual running nodes.

There is also an assumption that the WebSocket RPC transport is the right interface for the monitoring dashboard. The earlier confusion between HTTP POST and WebSocket suggests that the RPC layer might benefit from supporting both transports, but the current implementation commits to WebSocket only.

Input and Output Knowledge

To understand message 717, one needs to know that the cluster consists of two Kuri storage nodes (kuri-1, kuri-2) each exposing an S3-compatible API on port 8078, with web UI dashboards on ports 9010 and 9011. One needs to know that the RPC layer uses JSON-RPC 2.0 over WebSocket, that RIBS.LatencyDistribution accepts a time window parameter (here "5m" for five minutes), and that the response format includes percentile arrays indexed by time. One needs to know that the metrics collector was just implemented and that the test objects uploaded in messages 711-712 provided the request traffic that populated these data points.

The output knowledge created by this message is the confirmation that the latency distribution metrics pipeline is fully functional. This unblocks the frontend monitoring dashboard, which can now display real-time latency charts instead of empty graphs. It validates the architectural decision to instrument the S3 handlers at the proxy layer rather than at the storage node layer. And it provides a baseline for future performance tuning—the cold-start latency pattern visible in the data gives the team a reference point for optimization.

The Thinking Process Visible in the Session

The broader session reveals a systematic debugging methodology. When the user reported empty metrics, the assistant did not guess at the cause—it read the source code of diag.go to confirm that the endpoints were stubs. It then designed a metrics collector, implemented it, wired it into the request handling path, rebuilt the Docker image, restarted the containers, generated test traffic, and verified each endpoint in sequence. Message 717 is the last in that verification chain, and the fact that it returns plausible data (with the cold-start latency pattern) is the strongest possible evidence that the implementation is correct.

The assistant also demonstrated awareness of the frontend's expectations. The earlier work in the session included adding JSON tags to ensure camelCase serialization matching the React frontend's property naming conventions. The metrics data structures were designed to produce exactly the format the frontend expected, avoiding a common pitfall where backend and frontend disagree on field names.

Conclusion

Message 717 is a small message with outsized significance. It represents the transition from a cluster that merely exists to a cluster that can be observed. The empty arrays and zero values that plagued the monitoring dashboard in message 691 have been replaced with real latency percentiles, real timestamps, and a pattern that tells a story about the system's behavior under load. For the developer watching this session, message 717 is the moment when the cluster monitoring infrastructure proves itself—when the abstractions of metrics collectors, rolling windows, and percentile calculations produce output that matches reality. It is the difference between a system that is theoretically observable and one that actually is.