The Moment the Dashboard Came Alive: Verifying Real-Time Cluster Metrics in a Horizontally Scalable S3 Architecture

Message in Context

[bash] echo '{"jsonrpc":"2.0","method":"RIBS.RequestThroughput","params":["5m"],"id":1}' | websocat ws://localhost:9010/rpc/v0 2>&1 {"id":1,"jsonrpc":"2.0","result":{"timestamps":[1769867790,1769867800],"total":[192.9,2862.6],"reads":[132.9,1865.5],"writes":[60,997.1],"byProxy":{}}}

At first glance, this message appears to be a routine verification step—a developer firing a curl-like command against a WebSocket RPC endpoint and inspecting the JSON response. But within the arc of a complex distributed systems debugging session, this single command represents the culmination of a multi-hour effort to make a cluster monitoring dashboard actually work. The message is the moment of proof: the frontend will now render real data because the backend finally speaks the right language.

Why This Message Was Written: The Debugging Arc

This message was written as a verification probe—the final check in a chain of fixes that began when the user reported that the cluster monitoring frontend was "still empty" and the topology didn't render beyond a top "load balancer" label. The user had pasted raw RPC responses showing that the backend was returning perfectly valid data: throughput numbers, latency distributions, error rates, active request counts, and cluster events. Yet the React frontend stubbornly showed "No data available" and "No cluster nodes configured."

The root cause was a classic serialization mismatch. The Go backend, using the standard encoding/json package, was serializing struct fields with their Go-defined names—PascalCase (Timestamps, Total, Reads, Writes, Proxies, StorageNodes, NodeID). The React frontend, written by a different developer (or at a different time), expected camelCase (timestamps, total, reads, writes, proxies, storageNodes, nodeId). The data was flowing, but the field names didn't match, so the JavaScript destructuring assignments (const { proxies = [], storageNodes = [] } = topology) produced empty arrays, and the chart components found no timestamps or total keys to plot.

The assistant's response to the user's bug report was methodical: read the frontend source code to understand what field names it expected, read the Go interface definitions to see what the backend emitted, identify every struct that crossed the RPC boundary, add json struct tags to force camelCase serialization, fix a cascading type mismatch in an anonymous map literal that the JSON tags broke, rebuild the Go binary, rebuild the Docker image, restart the cluster containers, generate test traffic, and finally—in this message—query the RPC endpoint to confirm the fix worked.

What the Output Reveals

The JSON response contains two data points. The first timestamp (1769867790) shows 192.9 total requests per second (132.9 reads, 60 writes). The second timestamp (1769867800) shows 2862.6 total requests per second (1865.5 reads, 997.1 writes). The ten-second interval between measurements corresponds to the metrics collector's sampling window. The dramatic spike—from ~193 to ~2863 requests per second—is not organic traffic; it is the assistant's own test loop, executed immediately before this command, which uploaded and downloaded five test objects to generate visible data.

The byProxy field is empty ({}), which is expected because the metrics collector tracks per-proxy breakdowns but the test cluster's S3 proxy layer was not yet fully instrumented for that dimension. This empty field is itself informative: it tells the developer that the per-proxy breakdown feature exists in the data model but hasn't been populated, which is consistent with the architecture's current state where the S3 frontend proxy was only recently separated from the Kuri storage nodes.

Crucially, every field name is now camelCase: timestamps, total, reads, writes, byProxy. This matches exactly what the React frontend's RequestThroughputChart component destructures. The fix is confirmed working at the protocol level.

Assumptions and Their Validity

The assistant made several assumptions in this workflow. First, it assumed that the JSON field name mismatch was the sole cause of the frontend rendering failure. This was a reasonable hypothesis given the evidence: the backend returned data, the frontend showed nothing, and the field names clearly differed. The assumption proved correct—after the JSON tags were added, the topology and charts rendered properly (as confirmed in subsequent messages).

Second, the assistant assumed that adding JSON struct tags to the Go interface types would not break other consumers of those types. This was a risk: if other code paths depended on the default PascalCase serialization (e.g., external API clients, log formatters, or persisted data), changing the field names could cause subtle breakage. The assistant mitigated this by rebuilding and testing immediately, catching the anonymous struct type mismatch in cluster_metrics.go that the JSON tags exposed.

Third, the assistant assumed that the RPC endpoint on port 9010 (the web UI container for kuri-1) would return the same data format as the backend's internal representation. This was correct because the RPC layer serializes the same Go structs regardless of which container serves them.

Input Knowledge Required

To understand this message, a reader needs several pieces of contextual knowledge:

  1. The architecture: The test cluster has a three-layer design—S3 frontend proxies (port 8078) that route requests to Kuri storage nodes, which store data in YugabyteDB. The web UI containers (ports 9010, 9011) serve a React dashboard that polls RPC endpoints for metrics.
  2. The RPC protocol: The cluster uses a JSON-RPC 2.0 protocol over WebSocket. The method RIBS.RequestThroughput is one of several monitoring RPCs (alongside RIBS.LatencyDistribution, RIBS.ErrorRates, RIBS.ActiveRequests, RIBS.ClusterEvents, and RIBS.ClusterTopology).
  3. The metrics collector: A ClusterMetrics component in rbstor/cluster_metrics.go tracks throughput, latency, error rates, active requests, and I/O bytes with a rolling 10-minute window and 10-second sampling intervals.
  4. The serialization fix: The Go structs originally lacked json tags, so the default Go serialization produced PascalCase field names. The assistant added tags like json:"timestamps" to force camelCase output matching the frontend's expectations.
  5. The debugging history: The user had reported the frontend was empty despite the backend returning data, and the assistant had just finished adding JSON tags to all relevant structs across iface/iface_ribs.go and fixing a type mismatch in rbstor/cluster_metrics.go.

Output Knowledge Created

This message creates several pieces of knowledge:

  1. Verification that the JSON casing fix works: The response shows camelCase field names, confirming the Go struct tags are being respected by the serializer.
  2. Confirmation that the metrics collector is accumulating data: Two data points with different values prove that the rolling window is capturing new samples as time passes and traffic flows.
  3. Evidence of the test traffic's impact: The tenfold increase in throughput between the two samples validates that the test loop (five PUT and five GET requests) is being registered by the metrics instrumentation in the S3 handlers.
  4. A baseline for frontend rendering: With this RPC now returning correctly-named fields, the RequestThroughputChart component should be able to render the line chart, and the other components (LatencyDistributionChart, ErrorRateChart, ClusterTopology) should follow once their respective RPCs are similarly verified.

The Thinking Process Visible in This Message

The message itself is terse—a single bash command piped through websocat—but the thinking behind it is revealed by its placement in the sequence of actions. The assistant had just:

  1. Identified the JSON field name mismatch by reading the frontend source code
  2. Added JSON tags to ClusterTopology, ThroughputHistory, LatencyDistribution, ErrorRateData, ActiveRequests, ClusterEvent, and their nested structs
  3. Fixed a compile error caused by an anonymous struct type mismatch in cluster_metrics.go
  4. Rebuilt the Go binary and Docker image
  5. Restarted the cluster containers
  6. Run a test loop to generate fresh metrics data The choice to query RIBS.RequestThroughput first, rather than RIBS.ClusterTopology (which was the component the user specifically reported as broken), is telling. Throughput data is the most dynamic metric—it changes with every request and provides immediate feedback that the instrumentation pipeline is working end-to-end. The topology data, by contrast, is relatively static (nodes don't appear or disappear every second). By checking throughput first, the assistant gets a high-signal verification: if the timestamps and throughput numbers are updating and using the correct field names, the serialization fix is confirmed, and the other RPCs will almost certainly work too. The assistant also chose to query port 9010 (kuri-1's web UI) rather than port 9011 (kuri-2's) or port 8078 (the S3 proxy). This is deliberate: port 9010 is the web UI that the user was looking at when they reported the empty dashboard. Verifying that the RPC returns correct data on that specific endpoint ensures that the fix will be visible in the user's browser when they reload the page.

Mistakes and Incorrect Assumptions

The most notable mistake in this overall debugging episode was the initial omission of JSON struct tags. The Go structs in iface/iface_ribs.go were defined without explicit serialization annotations, relying on Go's default behavior of using the struct field name as the JSON key. This is a common pitfall in projects where the backend and frontend are developed independently or at different times. The assistant had written the Go types and the React components but had not ensured they agreed on the serialization format.

A secondary issue was the cascading compile error when JSON tags were added to the LatencyDistribution struct's ByOperation field, which used an anonymous struct type. The anonymous struct struct{ P50, P95, P99 []float64 } had to be updated to struct{ P50 []float64 "json:\"p50\""; P95 []float64 "json:\"p95\""; P99 []float64 "json:\"p99\"" } to match the tagged version. This required multiple edit cycles to resolve, as the initial fix attempt left a type mismatch that the Go compiler caught.

These mistakes are characteristic of rapid prototyping in a distributed system: the developer focuses on getting the logic right first, and the serialization layer gets attention only when the frontend fails to render. The debugging process itself—reading the frontend code to discover the expected format, then updating the backend to match—is a textbook example of "make the data fit the consumer" rather than the reverse.

Conclusion

This single message, for all its brevity, encapsulates the essence of distributed systems debugging: a chain of reasoning that stretches from a user's screenshot of an empty dashboard, through source code analysis of both frontend and backend, through serialization fixes and container rebuilds, to a final verification command that proves the pipeline works. The JSON response with its two data points and camelCase field names is not just data—it is the resolution of a communication failure between two systems that speak different dialects of the same language. The dashboard would now render, the charts would now plot, and the user would finally see the cluster they had built.