Closing the Visibility Gap: How Cross-Node Stats Aggregation Fixed a Distributed Monitoring Blind Spot

Introduction

In distributed systems, observability is not a luxury—it is a necessity. When a horizontally scalable architecture spreads workloads across multiple independent nodes, the monitoring layer must reflect that distribution faithfully. A dashboard that only shows one node's perspective while traffic flows evenly across the cluster is worse than incomplete: it is actively misleading. Message 906 in this coding session captures the moment when the assistant diagnosed and resolved exactly such a gap, transforming a single-node monitoring view into a cluster-wide aggregated picture. This article examines that message in depth, exploring the reasoning, assumptions, decisions, and technical context that shaped the fix.

The Message: A Summary of Debugging and Resolution

The subject message is a concise status summary written by the assistant after an extended debugging session. It reads:

Now both storage nodes show requestsPerSecond: 0.5 — traffic is being distributed equally between kuri-1 and kuri-2, and the stats are being aggregated correctly from both nodes.

Summary of fixes: 1. Added /api/stats endpoint to integrations/web/ribsweb.go — returns JSON with node's local stats (nodeId, groupsCount, storageUsed, requestsPerSecond) 2. Updated ClusterTopology in rbstor/diag.go to fetch stats from remote nodes via the new /api/stats endpoint instead of showing zeros 3. Restarted nginx webui container — it had stale DNS entries after kuri containers were recreated

The round-robin was already working (logs showed alternating kuri-1/kuri-2), but the UI wasn't showing it because ClusterTopology only reported local node stats. Now it fetches stats from all nodes in the cluster.

This message appears at the tail end of a lengthy debugging sequence spanning messages 870 through 905. It is not a plan, a question, or a request for input—it is a retrospective summary, written after the fixes were implemented and verified. The assistant is declaring the problem resolved and documenting what was changed and why.

Why This Message Was Written: The Reasoning and Motivation

The message exists because of a fundamental tension between the architecture's operational reality and its monitoring layer. The S3 frontend proxy had been verified to distribute write requests across both Kuri storage nodes using round-robin load balancing—the logs at message 873 showed alternating backend selections. Yet the web UI, which consumed the RIBS.ClusterTopology RPC endpoint, displayed only kuri-1's metrics. Storage node kuri-2 appeared in the topology list but showed zero values for requests per second, storage used, and other live statistics.

This discrepancy was not a cosmetic bug. It meant that an operator watching the dashboard would see a flatline on kuri-2 and conclude that either the load balancer was broken, the node was unhealthy, or traffic was not reaching it. None of those conclusions would be correct. The monitoring layer was actively undermining trust in the system by presenting an incomplete picture.

The assistant's motivation for writing message 906 was to close this loop: to confirm that the fix worked, to document the three specific changes that were made, and to explain why the round-robin had appeared broken in the UI when it was actually functioning correctly all along. The message serves as a bridge between the debugging activity (the edits, builds, container restarts, and curl commands) and the architectural understanding of what went wrong.## How Decisions Were Made: The Debugging Trail

The decisions in this message were not made in a vacuum—they emerged from a chain of observations and experiments visible in the preceding context messages. The assistant's reasoning process can be reconstructed step by step.

Step 1: Identifying the symptom. At message 874, the assistant observed that "the round-robin is working — traffic is being distributed evenly between kuri-1 and kuri-2 (alternating). The issue is that the UI is not reflecting this — it's only showing traffic on kuri-1." This was the critical insight: the problem was not in the data path but in the monitoring path.

Step 2: Tracing the root cause. The assistant read the implementation of ClusterTopology in rbstor/diag.go (message 875) and immediately spotted the flaw. On lines 244–248, the method only filled in statistics for the local node—the node on which the RPC call was executed. For remote nodes discovered via the FGW_BACKEND_NODES environment variable, it populated the topology structure with zero values. This was not a bug in the sense of incorrect code; it was a design limitation. The method had no mechanism to fetch live metrics from peer nodes.

Step 3: Choosing the approach. The assistant considered making WebSocket RPC calls to remote nodes but deemed this "a more complex change" (message 877). Instead, the chosen strategy was to add a lightweight HTTP endpoint (/api/stats) that returns each node's local statistics as JSON, and then have ClusterTopology call this endpoint for every remote node in the cluster. This decision reflects a pragmatic trade-off: HTTP calls are simpler to implement, debug, and test than WebSocket RPCs, and they introduce no additional protocol dependencies.

Step 4: Implementing and testing. The assistant added the /api/stats handler to integrations/web/ribsweb.go, updated ClusterTopology in rbstor/diag.go to iterate over remote nodes and fetch their stats, rebuilt the Docker image, restarted the containers, and verified the fix by generating traffic and querying the RPC endpoint. The verification at message 896 confirmed that both storage nodes now showed requestsPerSecond: 0.5.

Step 5: Discovering the nginx DNS issue. After the fix was deployed, the assistant noticed that port 9010 (which should serve kuri-1's web UI) was returning kuri-2's data, and vice versa (message 902). This turned out to be a stale DNS cache in the nginx webui container, which had not been restarted after the kuri containers were recreated. Restarting the webui container resolved the swap. This was a secondary, independent bug that would have caused confusion even if the stats aggregation had been working perfectly.

Assumptions Made by the Assistant

Several assumptions underpinned the assistant's work, some explicit and some implicit:

The round-robin was already correct. This assumption was validated by examining the S3 proxy logs, which showed alternating backend selections. The assistant did not modify the load-balancing logic—only the monitoring layer.
Cross-node HTTP communication works within the Docker network. The assistant assumed that kuri-1 could reach kuri-2's web UI port (9010) via the internal Docker network. This was tested explicitly at message 894 with docker compose exec kuri-1 wget -q -O- http://kuri-2:9010/api/stats, which succeeded.
The /api/stats endpoint would not conflict with existing routes. The assistant registered the new endpoint after the catch-all / handler in the HTTP mux. In Go's http.ServeMux, the most specific pattern wins regardless of registration order, so this was safe—but the assistant double-checked by reading the routing code at message 890.
The environment variables (FGW_NODE_ID, FGW_BACKEND_NODES) are correctly set. The assistant verified these at messages 899–901 by inspecting container environments and config files.
The nginx DNS cache was the cause of the port swap. This was an inference drawn after verifying that the config files and environment variables were correct. The assistant tested the hypothesis by restarting the webui container, which resolved the issue.## Mistakes and Incorrect Assumptions While the assistant's debugging was largely effective, there were moments where assumptions proved incomplete or required correction. The nginx DNS cache blind spot. The assistant did not initially anticipate that restarting the kuri containers would cause the nginx webui proxy to cache stale DNS resolutions. After the fix was deployed and tested, the port swap at message 902 revealed that port 9010 was returning kuri-2's data. The assistant's first instinct was to suspect a configuration error, but the config files and environment variables were correct. The actual culprit was Docker's DNS resolution combined with nginx's connection persistence—the webui container had resolved the hostnames kuri-1 and kuri-2 to their previous container IPs before the recreate, and those IPs had changed. This is a classic Docker Compose pitfall: containers that resolve peer hostnames at startup and cache the results will break after --force-recreate changes the IP assignments. The assistant correctly identified the fix (restart the webui container) but did not add a permanent solution such as a health check dependency or a restart policy. The stats endpoint route ordering assumption. At message 891, the assistant expressed concern that the /api/stats route might be shadowed by the catch-all / handler. The comment "the order of registration doesn't matter — the most specific pattern wins" is correct for Go 1.22+ http.ServeMux, but the assistant's earlier code at message 880 had registered /api/stats after /, which would have been a problem in older Go versions where ServeMux used first-match semantics. The assistant was working with Go 1.22, so the assumption held, but this detail is worth noting for readers maintaining compatibility with older Go toolchains. The scope of the fix. The assistant assumed that adding HTTP-based stats fetching to ClusterTopology would be sufficient for full cluster observability. However, this approach introduces a dependency on the web UI port being reachable from every storage node. In a production deployment, the web UI port might not be exposed to the internal cluster network, or it might be behind authentication. The fix is appropriate for the test cluster but would need re-architecting for production—for example, using a dedicated metrics bus or a shared database table for node statistics.

Input Knowledge Required to Understand This Message

To fully grasp message 906, a reader needs familiarity with several layers of the system:

Architecture knowledge. The three-layer topology (S3 frontend proxy → Kuri storage nodes → YugabyteDB) is essential context. The reader must understand that the S3 proxy distributes writes across Kuri nodes via round-robin, and that each Kuri node runs its own web UI and RPC server. Without this mental model, the distinction between "the round-robin works" and "the UI shows zeros" is meaningless.

Go HTTP routing semantics. The assistant's confidence that /api/stats would not be shadowed by / relies on understanding Go 1.22's http.ServeMux pattern matching. A reader unfamiliar with this would wonder why the order of handler registration doesn't matter.

Docker Compose networking. The nginx DNS caching issue is a well-known Docker Compose behavior. Containers within the same Compose project can resolve each other by service name, but the IP addresses are assigned at container start and can change on recreate. The webui container had cached the old IPs for kuri-1 and kuri-2, causing the port swap.

Environment variable configuration. The FGW_NODE_ID, FGW_BACKEND_NODES, and FGW_NODE_TYPE variables are the mechanism by which each container knows its identity and discovers its peers. The assistant's debugging relied on verifying these values in multiple places (container environment, settings files, nginx config).

The existing monitoring stack. The RIBS.ClusterTopology RPC method, the WebSocket RPC transport, and the React frontend that consumes the topology data are all part of the monitoring pipeline. The fix added a new HTTP endpoint to supplement the existing RPC-based monitoring, which means the reader must understand why a new endpoint was needed rather than extending the RPC protocol.## Output Knowledge Created by This Message

Message 906 is itself a piece of output knowledge—a summary that crystallizes the debugging session into actionable documentation. But beyond the message text, the work it describes produced several artifacts of lasting value:

The /api/stats endpoint. This is a new, lightweight, HTTP-based monitoring endpoint that returns each node's local statistics as JSON. It is simpler than the existing RPC-based monitoring and can be consumed by any HTTP client without WebSocket support. This endpoint becomes the building block for cross-node stats aggregation.

The cross-node stats aggregation pattern. The updated ClusterTopology method now iterates over all nodes in FGW_BACKEND_NODES, fetches their stats via HTTP, and merges them into a unified topology response. This pattern—local endpoint + remote fetching in the aggregator—is a common distributed monitoring design that can be reused for other metrics.

The nginx restart procedure. The debugging revealed that the webui container needs to be restarted after kuri container recreates. This is now documented implicitly in the assistant's actions, though it is not codified in the Docker Compose configuration. A future improvement would be to add a depends_on condition or a health check that forces nginx to re-resolve peer hostnames.

The verification procedure. The assistant established a repeatable test sequence: generate traffic with dd and curl, query the RPC endpoint with websocat, inspect the JSON output with jq, and compare stats across nodes. This procedure can be used by any developer to validate that the monitoring layer is correctly aggregating data.

The confirmed health of the round-robin. Perhaps most importantly, the message confirms that the S3 frontend proxy's round-robin load balancing is functioning correctly. The logs at message 873 showed alternating backend selections, and the aggregated stats now confirm that both nodes receive equal traffic (0.5 requests per second each after 10 uploads). This validates a core architectural assumption of the horizontally scalable design.

The Thinking Process Visible in Reasoning Parts

The subject message is a summary, not a reasoning trace, so the thinking process is not directly visible within it. However, the message is the culmination of a reasoning chain that is richly documented in the preceding context messages. By reading backward from message 906, we can reconstruct the assistant's thought process:

Hypothesis formation. At message 874, the assistant formed the hypothesis: "The issue is that the UI is not reflecting this — it's only showing traffic on kuri-1. The problem is that the ClusterTopology RPC is being called on kuri-1, and kuri-1 only knows about its own metrics. The metrics on kuri-2 aren't being aggregated." This is a precise, testable hypothesis grounded in the architecture.

Root cause analysis. At message 875, the assistant read the source code and confirmed the hypothesis: "I see the problem. On lines 244-248, the ClusterTopology method only fills in stats for the local node (when nodeID == selfNodeID). For remote nodes, it doesn't fetch their stats — it just shows them with zero values."

Design decision. At message 877, the assistant weighed alternatives: "The issue is that to fetch stats from other nodes, we'd need to make WebSocket RPC calls to them. This is a more complex change. For now, a simpler approach is to add a new lightweight RPC endpoint that returns just the node's local stats, and then have ClusterTopology call this endpoint for each remote node via HTTP." This is a classic engineering trade-off: simplicity over completeness, with the understanding that the simpler solution can be replaced later.

Iterative debugging. The assistant did not implement the fix blindly. After building and deploying, the assistant tested at message 893, then discovered the nginx port swap at message 902, diagnosed it by comparing environment variables and config files, and applied the restart fix. The final verification at message 905 confirmed that everything worked correctly.

The summary as closure. Message 906 is the assistant's way of saying "the loop is closed." The round-robin was working all along, but the monitoring layer was blind to it. Now the monitoring layer sees what the data path is doing. The message is both a status report and a postmortem, documenting what was wrong, what was changed, and what the system now looks like in its corrected state.

Conclusion

Message 906 is a small but revealing artifact of distributed systems engineering. It captures the moment when a monitoring blind spot was identified, diagnosed, and eliminated. The fix itself was modest—a new HTTP endpoint, a loop over peer nodes, and a container restart—but the reasoning behind it touches on fundamental principles of observability in distributed systems: the separation of data path and monitoring path, the need for cross-node aggregation, the pitfalls of container networking, and the importance of verifying that what the dashboard shows matches what the system actually does.

For a reader new to this project, message 906 offers a window into the discipline of building horizontally scalable storage. It shows that getting the architecture right is only half the battle—the other half is making sure you can see it working.