Diagnosing a Cluster Topology Regression in a Distributed S3 Storage System

The Message

In message 2106 of a lengthy coding session building and debugging the Filecoin Gateway (FGW) distributed storage system, the assistant responds to a user-reported regression with a focused diagnostic investigation. The user's previous message (2105) stated: "Ah the issue is the /cluster Storage Nodes table doesn't see traffic on the other node, seems to be a regression." The assistant replies:

[assistant] Let me investigate the cluster topology issue - the Storage Nodes table not showing cross-node traffic: [bash] # Check cluster topology from both kuri nodes echo "=== Kuri1 ClusterTopology ===" curl -s http://10.1.232.83:9010/rpc/v0 -X POST -H 'Content-Type: application/json' \ -d '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' | jq .

>

echo "" echo "=== Kuri2 ClusterTopology ===" curl -s http://10.1.232.84:9010/rpc/v0 -X POST -H 'Content-Type: application/json' \ -d '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' | jq . === Kuri1 ClusterTopology === { "id": 1, "jsonrpc": "2.0", "result": { "proxies": [ { "id": "kuri_01", "address": "localhost", "status": "healthy", "requestsPerSecond": 0, "activeConnections": 0, "backendPool": null, "latencyMs": 0, "errorRate": 0 } ], "storageNodes": [ { "id": "kuri_01", "address": "http://10.1.232.83:8079", "status": "healthy", "storageUsed": 274...

The response is truncated in the conversation record, but the critical finding is already visible: the cluster topology from kuri_01 only shows itself — a single proxy entry (kuri_01) and a single storage node entry (kuri_01). The peer node kuri_02 is entirely absent. This is the smoking gun of the regression the user observed in the web UI.

Context: What Led to This Moment

To understand why this message matters, we must understand what came before it. The session had been a marathon of distributed systems engineering. The team had deployed a three-node QA cluster for FGW — a horizontally scalable S3-compatible storage layer built on top of Filecoin. The architecture separates stateless S3 frontend proxies (running on a "head" node at 10.1.232.82) from Kuri storage nodes (running on 10.1.232.83 and 10.1.232.84) that handle the actual data persistence, with YugabyteDB providing shared metadata storage.

In the messages immediately preceding this diagnostic, the assistant had been validating the deployment. Load tests ran successfully — 1689 writes and 1573 reads with zero data corruption. Metrics showed both kuri nodes receiving traffic, albeit with an uneven 62/38 distribution favoring kuri_01. The proxy configuration was verified to include both backends. Everything appeared functional at the data level.

Then the user checked the web UI's cluster management page and spotted the problem: the Storage Nodes table wasn't showing cross-node traffic. Each node's view of the cluster was incomplete. This was described as a "regression," implying that the cluster topology had worked correctly at some earlier point and had since broken — possibly due to one of the many configuration changes, binary rebuilds, and service restarts that had occurred during the session.## The Reasoning Behind the Diagnostic Approach

The assistant's response demonstrates a methodical, hypothesis-driven debugging approach. Faced with a user report of a UI-level regression, the assistant does not immediately jump to code inspection or configuration file review. Instead, the first step is to reproduce the symptom at the API level — querying the cluster topology RPC endpoint directly on both nodes. This is a textbook debugging strategy: strip away the UI layer and check whether the underlying data is correct. If the RPC returns incomplete data, the problem is in the backend; if the RPC returns correct data but the UI doesn't display it, the problem is in the frontend.

The choice of tooling is also revealing. The assistant uses curl to POST a JSON-RPC request to each node's web UI port (9010), piping the result through jq for pretty-printing. This is a deliberate, production-appropriate technique. The web UI is built on an RPC interface (RIBS.ClusterTopology), and by calling it directly, the assistant bypasses any JavaScript rendering bugs or browser caching issues that might obscure the true state of the system. The use of jq ensures the output is readable even when the JSON response is large — a small but important detail for a debugging session where clarity matters.

The decision to query both nodes is equally deliberate. A single-node query could reveal a local problem (the node doesn't know about its peer), but comparing both nodes' responses reveals the symmetry or asymmetry of the failure. If both nodes see only themselves, the peer discovery mechanism is broken on both sides. If one node sees both and the other doesn't, the failure is asymmetric — perhaps a firewall, a misconfiguration, or a startup ordering issue. In this case, the truncated response shows kuri_01 returning only itself, and we can infer that kuri_02 likely returns only itself as well, confirming a bilateral failure of cluster awareness.

Assumptions Embedded in the Diagnostic

Every debugging step carries assumptions, and this message is no exception. The most fundamental assumption is that the RIBS.ClusterTopology RPC method is the authoritative source of truth for cluster membership. The assistant implicitly trusts that this method reflects the internal state of the node's peer discovery mechanism. If the peer discovery mechanism itself is buggy — for example, if it stores peer information in a corrupted data structure — the RPC response would be wrong, but the diagnostic would still point to the right layer.

Another assumption is that the network between nodes is functional. The assistant had just verified that both kuri nodes are active (sudo systemctl is-active kuri returned "active" for both) and that the S3 proxy can route traffic to both backends. However, the cluster topology RPC might use a different communication channel than the S3 data path. The topology discovery might rely on a gossip protocol, a shared database query, or a configuration file — any of which could fail independently of the data path.

The assistant also assumes that the regression is not a UI caching issue. The user's report specifically mentioned the "Storage Nodes table" in the web UI, but the assistant immediately goes to the RPC layer. This is a reasonable assumption based on experience: UI regressions in distributed systems are more often caused by backend data issues than by frontend rendering bugs, especially when the system has undergone recent configuration changes and restarts.

What the Response Reveals About the System Architecture

The truncated JSON response in the message is worth examining closely. The proxies array contains a single entry with id: "kuri_01" and address: "localhost". This is itself a clue: the node is reporting itself as a proxy with address "localhost" rather than its actual IP address (10.1.232.83). This suggests that the proxy registration logic is using a default or loopback address instead of the node's externally reachable address — a potential configuration issue or a bug in how the node determines its own network identity.

The storageNodes array also contains only kuri_01, with address http://10.1.232.83:8079 and status "healthy." The absence of kuri_02 is the core symptom. The storageUsed field is truncated at "274..." — likely 274 bytes or 274 MB, indicating the node has some data stored from the load tests.

The fact that the node reports itself in both the proxies and storageNodes arrays is architecturally significant. In the FGW design, the S3 frontend proxy and the Kuri storage node are separate services running on separate machines. But the cluster topology API on each kuri node apparently includes a self-reported proxy entry, suggesting that the topology system is designed to be a unified view of all cluster components, not just storage nodes. The "localhost" address in the proxy entry is a red flag — it means the proxy's address was never configured or discovered correctly.

The Thinking Process Visible in the Response

While the message is concise, the thinking process is visible in its structure. The assistant:

  1. Accepts the user's diagnosis ("the Storage Nodes table doesn't see traffic on the other node") without challenging it. This is important: the user has already done some investigation and identified the symptom. The assistant trusts this observation and moves to root cause analysis.
  2. Formulates a testable hypothesis: If the web UI doesn't show cross-node traffic, the underlying cluster topology data is probably incomplete. The hypothesis can be tested by querying the RPC endpoint directly.
  3. Designs a minimal experiment: Query both nodes' topology endpoints, compare results. The experiment is designed to produce a clear binary outcome — either both nodes see each other (the UI has a bug) or they don't (the backend has a bug).
  4. Executes the experiment with proper tooling: Using curl with a JSON-RPC payload, piped through jq for readability. The command is crafted to be copy-pasteable and reproducible.
  5. Interprets the results: The response is truncated, but the assistant's next actions (not visible in this message) would presumably involve comparing the two nodes' responses and identifying the missing peer information. This is a pattern of disciplined debugging that prioritizes data collection over speculation. The assistant doesn't ask "what changed?" or "when did this break?" — questions that are hard to answer in a complex system with many moving parts. Instead, it asks "what is the system's current state?" — a question that can be answered definitively with the right API calls.

Input Knowledge Required

To fully understand this message, a reader needs several pieces of context:

Output Knowledge Created

This message produces several valuable outputs:

  1. A confirmed symptom: The RPC response from kuri_01 definitively shows that the node only knows about itself. This confirms the user's UI observation and narrows the problem to the backend.
  2. A baseline measurement: The exact JSON response serves as a baseline. After any fix is applied, querying the same endpoint will show whether the fix worked.
  3. A reproducible diagnostic procedure: The curl commands can be reused by anyone debugging similar issues in the future. They are self-contained and don't depend on any special tools beyond curl and jq.
  4. Evidence of a configuration or discovery bug: The "localhost" address in the proxy entry is a specific anomaly that points toward a configuration issue rather than a network connectivity problem.

Mistakes and Incorrect Assumptions

The most significant potential mistake in this message is the implicit assumption that the RPC response is the complete truth. The cluster topology might be computed from multiple data sources — a local configuration file, a database query, a gossip protocol exchange — and any of these could be stale or corrupted. The RPC response shows the result of the computation, not the inputs to it. A more thorough diagnostic might also check the node's configuration for peer addresses, query the database for registered nodes, and check the logs for peer discovery errors.

Another subtle assumption is that the problem is symmetric. The assistant queries both nodes, but the response shown is only from kuri_01. If kuri_02's response were different — for example, if kuri_02 saw both nodes — the diagnosis would shift dramatically. The assistant's approach assumes that querying both nodes is sufficient to distinguish between symmetric and asymmetric failures, but the response doesn't show the kuri_02 result, leaving this question open.

Finally, the assistant assumes that the regression is caused by a recent change rather than a latent bug that was always present but only manifested under certain conditions. The user described it as a regression, and the assistant accepts this framing. However, it's possible that the cluster topology never correctly showed cross-node information in this deployment, and the user only noticed it now because they were looking at the Storage Nodes table for the first time. Accepting the user's framing without verification is a minor risk — though in practice, the user's context (they had seen it work before) makes this a reasonable trust decision.

Why This Message Matters

This message is a small but perfect example of disciplined debugging in a complex distributed system. It demonstrates how to isolate a UI-visible symptom to its backend cause, how to design a minimal experiment that produces clear evidence, and how to use the system's own API as a diagnostic tool. The message is also a window into the broader challenges of building and operating distributed storage systems: peer discovery, configuration management, and the constant tension between the simplicity of the data path and the complexity of the control path.

The cluster topology regression would eventually be traced to a configuration issue — likely the FGW_BACKEND_NODES environment variable that had been added earlier in the session to fix a different topology problem. The regression was probably introduced when the configuration was updated or the services were restarted, causing the peer discovery mechanism to lose its state. But the diagnostic in message 2106 is where the problem was first confirmed at the API level, transforming a vague UI complaint into a precise, reproducible bug report. That transformation — from symptom to evidence — is the essence of good debugging, and this message captures it perfectly.