From Empty Stubs to Live Metrics: Instrumenting a Distributed S3 Cluster for Real-Time Monitoring

Introduction

In the course of building a horizontally scalable S3-compatible storage architecture, a critical gap emerged between the infrastructure and its observability. The cluster monitoring dashboard, designed to provide real-time visibility into the health and performance of distributed storage nodes, was displaying nothing but zeros. The ClusterTopology RPC correctly identified the nodes in the cluster, but every metrics endpoint—RequestThroughput, LatencyDistribution, ErrorRates, ActiveRequests, and ClusterEvents—returned empty arrays, empty maps, and zero values. The user demonstrated this starkly in message 691, firing off a volley of RPC calls and receiving back a wall of empty JSON structures. The monitoring UI was a ghost town.

Message 719 is the assistant's summary of the work that bridged this gap. It is a concise, almost terse report: "Both nodes are now collecting and returning metrics." But beneath that simple statement lies a significant engineering effort—the design and implementation of a real-time metrics collection subsystem, the instrumentation of HTTP handlers, the wiring of RPC methods to live data sources, and the verification that every endpoint now returns meaningful measurements. This article examines that message in depth, exploring the reasoning, decisions, assumptions, and knowledge that shaped the work.

The Subject Message

The message itself is structured as a summary, listing the files changed and the capabilities now available:

Both nodes are now collecting and returning metrics.

>

## Summary

>

Implemented real metrics collection for the cluster monitoring UI:

>

New file: rbstor/cluster_metrics.go - Time-series storage for throughput, latency, errors - Active request tracking - Cluster events log - 10-minute rolling window with 10-second intervals

>

Updated: rbstor/diag.go - Connected RPC methods to real metrics collector

>

Updated: server/s3/server.go and fx.go - Added metrics instrumentation to GET, PUT, HEAD handlers - Records latency, errors, and active requests - Logs node startup event

>

Now returning real data: - RequestThroughput - reads/writes per second over time - LatencyDistribution - P50/P95/P99 latencies over time - ErrorRates - error percentage with trend - ActiveRequests - current in-flight requests - ClusterEvents - node startup and other events - ClusterTopology - node list with health status

This message is not a transcript of the coding session—it is a retrospective summary, written after the implementation was complete and verified. It serves as a status report, a documentation of what was done, and a declaration that the monitoring system is now functional.## Context and Motivation: Why This Message Was Written

To understand message 719, we must first understand the situation that preceded it. The assistant had been building a test cluster for a horizontally scalable S3 architecture, following a roadmap that called for three distinct layers: stateless S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB backend. The cluster was operational—both Kuri nodes were running, the S3 proxy on port 8078 was accepting requests, and the web UI on ports 9010 and 9011 was accessible. The ClusterTopology RPC, implemented in the previous segment, correctly enumerated the proxies and storage nodes with their health status.

But the user's test in message 691 revealed a fundamental problem. Every metrics RPC returned empty data:

{"id":10,"jsonrpc":"2.0","result":{"Timestamps":[],"Total":[],"Reads":[],"Writes":[],"ByProxy":{}}}
{"id":11,"jsonrpc":"2.0","result":{"Timestamps":[],"P50":[],"P95":[],"P99":[],"ByOperation":{}}}
{"id":12,"jsonrpc":"2.0","result":{"Nodes":{}}}

The user appended the damning observation: "still empty." The monitoring dashboard, which was supposed to provide real-time visibility into the cluster's performance, was showing nothing. The RPC methods existed—they were defined in rbstor/diag.go—but their implementations were stubs that returned hardcoded empty structures. The assistant acknowledged this immediately in message 692: "The UI is probably showing 'empty' because while it gets the node list, all the actual metrics are zeros."

This is the core motivation behind message 719. The assistant was not writing a summary for its own sake; it was documenting the resolution of a critical observability gap. The message exists because the monitoring system was broken, and the assistant fixed it. The summary serves as a clear, structured record of what was done, enabling the user to verify the work and understand the architecture of the new metrics subsystem.

The Reasoning Process: From Diagnosis to Implementation

The assistant's reasoning, visible in the messages leading up to 719, reveals a methodical approach to solving the problem. The first step was understanding the scope of the issue. The user's RPC dump showed that six different endpoints were returning empty data. The assistant could have taken a narrow approach—fixing just one endpoint, or patching the stubs to return dummy data. Instead, it recognized that the problem required a comprehensive solution: a real metrics collection infrastructure.

The assistant's thinking is visible in message 692: "I need to implement actual metrics collection. This requires: 1. A metrics collector that tracks requests over time 2. Storage for historical data points." This is a clear architectural decision. Rather than retrofitting each RPC handler with ad-hoc data gathering, the assistant chose to create a centralized metrics collector that would serve as the single source of truth for all monitoring data.

The next decision was where to place this collector. The assistant created a new file, rbstor/cluster_metrics.go, rather than adding to the existing rbstor/diag.go. This is a deliberate separation of concerns: diag.go handles the RPC layer (receiving requests and formatting responses), while cluster_metrics.go handles the data layer (collecting, storing, and aggregating metrics). This separation makes the system more maintainable and testable.

The design choices within the metrics collector reveal further reasoning. The assistant chose a "10-minute rolling window with 10-second intervals." This is a pragmatic trade-off between memory usage and temporal resolution. A 10-minute window provides enough data to show meaningful trends without consuming excessive memory. The 10-second interval granularity is sufficient for monitoring purposes without creating an overwhelming number of data points. These choices reflect an understanding of what a cluster monitoring dashboard needs: enough history to identify trends, but not so much that the UI becomes sluggish or the memory footprint becomes problematic.## Decisions Made During Implementation

Several key decisions are embedded in the work that message 719 summarizes. The first is the choice of data structures for time-series storage. The assistant implemented a rolling window using slices of timestamps and corresponding metric values, rather than using a circular buffer or a more complex data structure. This is a straightforward approach that works well for the scale of a test cluster—a few nodes generating metrics at 10-second intervals over a 10-minute window results in only 60 data points per metric. The simplicity of the implementation reduces the risk of bugs and makes the code easy to understand.

The second decision is how to instrument the HTTP handlers. The assistant modified server/s3/server.go and fx.go to wrap the GET, PUT, and HEAD handlers with metrics recording. This is a decorator pattern: the handler logic is wrapped in code that records latency, tracks active requests, and counts errors before and after the actual handler executes. The decision to instrument at the handler level rather than at a lower level (e.g., in the HTTP middleware) reflects the architecture of the S3 server, which uses a custom routing handler to avoid Go 1.22's ServeMux route conflicts. By instrumenting at the handler level, the assistant ensures that metrics are recorded for all S3 operations without needing to modify the routing layer.

The third decision is how to expose the metrics through RPC. Rather than creating new RPC methods, the assistant modified the existing stub implementations in rbstor/diag.go to call through to the metrics collector. This preserves the existing RPC API—the frontend continues to call the same methods it was already calling—while replacing the empty stubs with live data. This is a backward-compatible change that requires no modifications to the React frontend.

Assumptions Made

The implementation rests on several assumptions. The most fundamental is that the metrics collector should be per-node rather than cluster-wide. Each Kuri node runs its own instance of the metrics collector, tracking only the requests that pass through that node. The ClusterTopology RPC aggregates data from all nodes, but the per-node metrics (like RequestThroughput and LatencyDistribution) only reflect the local node's perspective. This assumption is reasonable for a test cluster where each node handles its own traffic, but it could become problematic in a production deployment where load is unevenly distributed.

Another assumption is that the 10-minute rolling window is sufficient for monitoring purposes. The assistant implicitly assumes that operators will be watching the dashboard continuously or that the dashboard auto-refreshes frequently enough that older data is not needed. If someone wanted to view historical trends over hours or days, the current implementation would not support it—the data simply disappears after 10 minutes.

The assistant also assumes that the metrics collector's in-memory storage is adequate. There is no persistence layer, no write-ahead log, and no mechanism for recovering metrics after a node restart. If a Kuri node crashes and restarts, all accumulated metrics are lost. For a test cluster, this is acceptable; for production, it would be a significant limitation.

Mistakes and Incorrect Assumptions

The implementation process was not without its stumbles. The LSP errors reported during the coding session reveal that the assistant initially made mistakes in the metrics collector code. In message 693, the LSP reported errors about unknown fields TotalErrors and LastError in a struct literal. These errors occurred because the assistant was trying to populate fields that didn't exist in the NodeErrorStats struct defined in iface/iface_ribs.go. The assistant had to read the interface file to discover the correct field names (ErrorRate, ByType, Trend) and then fix the code.

Similarly, in message 697, the LSP reported "expected declaration, found 'return'" at line 314 of cluster_metrics.go. This was caused by leftover code—a duplicate return statement that had been left behind after an earlier edit. The assistant had to read the file, identify the problematic code, and remove it.

These mistakes highlight the challenges of implementing code across multiple files while maintaining consistency with interface definitions. The assistant's process of reading the interface file (iface/iface_ribs.go) to verify the correct struct fields is a good example of how to resolve such issues. The mistakes were caught by the LSP before the code was built, preventing runtime errors.

Input Knowledge Required

To understand message 719, a reader needs knowledge of several domains. First, an understanding of the cluster architecture: the three-layer design with S3 proxies, Kuri storage nodes, and YugabyteDB. The metrics collector sits in the Kuri storage node layer, recording the operations that each node processes.

Second, familiarity with Go's HTTP handler patterns is necessary. The assistant's decision to wrap handlers with metrics recording relies on the http.HandlerFunc pattern and the ability to measure latency using time.Now() before and after handler execution.

Third, knowledge of the project's existing RPC framework is required. The RPC methods (RIBS.RequestThroughput, RIBS.LatencyDistribution, etc.) were already defined and registered. The assistant's work was to connect these RPC endpoints to live data sources rather than stub implementations.

Fourth, understanding of the monitoring dashboard's data requirements is implicit in the implementation. The React frontend expects specific JSON structures for each RPC method—for example, Timestamps, Total, Reads, and Writes arrays for RequestThroughput. The assistant had to ensure that the metrics collector produced data in the exact format that the frontend expected.

Output Knowledge Created

Message 719 creates several kinds of knowledge. At the most concrete level, it documents the files changed and the capabilities now available. This is valuable for anyone who needs to understand or modify the monitoring system in the future. The summary format—listing new files, updated files, and the data now returned—provides a quick reference for the system's architecture.

At a deeper level, the message establishes a pattern for metrics collection that can be extended to other parts of the system. The cluster_metrics.go file provides a template for adding new metrics: define a data structure, implement a recording function, and expose the data through an RPC method. Future developers can follow this pattern to add, for example, per-operation latency breakdowns or storage utilization trends.

The message also creates operational knowledge: the monitoring system is now functional. The user can verify this by running the same RPC calls from message 691 and seeing non-empty results. The assistant's verification in messages 713-718 provides concrete examples of the data now being returned, serving as both a test and a demonstration.

The Thinking Process Visible in the Message

While message 719 is a summary rather than a transcript of reasoning, the thinking process is still visible in its structure and content. The assistant chose to organize the summary by file, listing what was created and what was updated. This reflects a developer's mental model of the codebase: changes are understood in terms of their location in the file system.

The bullet-point format under "Now returning real data" reveals the assistant's verification-driven approach. Each RPC method is listed with a description of what it returns. This is not just documentation—it's a checklist. The assistant verified each endpoint individually (as shown in messages 713-718) and then summarized the results. The thinking is: "I have confirmed that each of these endpoints returns real data, and here is the evidence."

The inclusion of implementation details like "10-minute rolling window with 10-second intervals" shows that the assistant considers these design parameters important enough to document. They represent deliberate choices that affect system behavior, and documenting them helps others understand the system's capabilities and limitations.

Conclusion

Message 719 is deceptively simple. On its surface, it is a brief summary of code changes. But when examined in the context of the surrounding conversation, it reveals a significant engineering effort: the design and implementation of a real-time metrics collection subsystem for a distributed S3 cluster. The assistant diagnosed the problem (empty stubs), designed a solution (centralized metrics collector with time-series storage), implemented it across multiple files, instrumented the HTTP handlers, wired the RPC methods, and verified that every endpoint returned real data.

The message captures the transition from a non-functional monitoring system to a working one. It documents not just what was done, but the architectural decisions, the design parameters, and the verification results. For anyone studying this coding session, message 719 serves as a milestone—the point at which the cluster monitoring dashboard came to life.