When the Cluster Topology Returned Nothing: Diagnosing a Distributed System's Discovery Problem

The Message

{"id":1,"jsonrpc":"2.0","result":{"proxies":[],"storageNodes":[],"dataFlows":[]}}

This small JSON payload — an empty cluster topology — was the outcome of a diagnostic probe issued by the assistant during the deployment and validation of a three-node QA cluster for the Filecoin Gateway (FGW) distributed storage system. The full message reads:

[assistant] The API is returning HTML instead of JSON - looks like it's hitting the wrong endpoint. Let me check the actual API: [bash] ssh 10.1.232.83 "curl -s http://127.0.0.1:9010/rpc/v0 -X POST -H 'Content-Type: application/json' -d '{\"jsonrpc\":\"2.0\",\"method\":\"RIBS.ClusterTopology\",\"params\":[],\"id\":1}' 2>&1" {"id":1,"jsonrpc":"2.0","result":{"proxies":[],"storageNodes":[],"dataFlows":[]}}

At first glance, this looks like a simple debugging step — a developer probing an API endpoint to see why a web UI component isn't rendering. But this message sits at a critical juncture in the deployment of a horizontally scalable, multi-node distributed storage system. Understanding why it was written, what it reveals about the system's architecture, and how it shaped the subsequent debugging session offers a window into the realities of distributed systems operations.

Context: A QA Cluster Takes Shape

The message appears in the middle of Segment 11 of a long-running coding session focused on building and validating the FGW system. The assistant had just completed deploying a three-node QA cluster across physical machines: a head node running YugabyteDB (10.1.232.82), and two storage nodes running the "kuri" daemon (10.1.232.83 and 10.1.232.84). The deployment had been arduous — the assistant had resolved "dirty migration" states in the CQL database schemas, configured systemd services with secure credential vaulting, and verified that both kuri nodes were running with their S3 APIs, web UIs, and Prometheus metrics endpoints all responding.

The user's report was concise: ":9010 cluster topology doesn't render". The web UI on port 9010 included a cluster topology visualization, but it was showing nothing — a blank canvas where a network diagram of storage nodes and proxies should have appeared. The user was flagging a functional gap in the deployment, and the assistant needed to diagnose why the topology data wasn't populating.

The Diagnostic Path: Two Endpoints, One Answer

The assistant's reasoning, visible in the message text, follows a clear diagnostic arc. The first attempt was to hit what seemed like a plausible REST endpoint: /api/v0/topology on port 9010. This returned HTML — the web UI's own page markup — which immediately signaled that this was not the correct API path. The assistant recognized this instantly: "The API is returning HTML instead of JSON - looks like it's hitting the wrong endpoint."

This recognition is itself an interesting piece of systems knowledge. In many web frameworks, a request to an unknown or misrouted path will fall through to the default handler, which often returns the main application HTML. The assistant correctly interpreted this as a routing failure rather than a server error, and pivoted to the known-good RPC endpoint: /rpc/v0 using JSON-RPC over POST with the RIBS.ClusterTopology method.

This pivot reveals an important architectural assumption: the system uses JSON-RPC, not REST, for its internal API. The assistant knew this from prior work building the system — the RIBS (Remote Indexed Block Storage) layer exposes its diagnostic and management functions through a JSON-RPC interface. The correct invocation required specifying the method name (RIBS.ClusterTopology), passing an empty params array, and using a request ID of 1. This is a standard JSON-RPC 2.0 call.

The Empty Result: A Deeper Problem

The response was technically valid JSON-RPC — no error, no exception — but the result was an empty topology: no proxies, no storage nodes, no data flows. This was the root cause of the blank UI: the API was working correctly, but there was nothing to report.

This empty result is the message's most important output. It created new knowledge: the cluster topology feature requires explicit configuration to discover nodes. The system doesn't automatically detect peer nodes through gossip or broadcast protocols — at least not in this deployment configuration. The topology data must be populated through environment variables or some other configuration mechanism.

The assistant's subsequent actions (visible in messages 2034-2040) confirmed this diagnosis. Reading the source code in rbstor/diag.go revealed that ClusterTopology() reads the FGW_BACKEND_NODES environment variable to discover cluster members. Without this variable set, the function returns empty arrays. The fix was to add the backend node list to each kuri node's settings file and restart the services.

Assumptions and Mistakes

The message reveals several implicit assumptions. First, the assistant assumed that /api/v0/topology was a valid REST endpoint — it wasn't, and the assistant corrected this quickly. Second, the assistant assumed that the cluster topology would be populated automatically after the nodes were initialized and running. This assumption was wrong: the topology requires explicit configuration of peer nodes. Third, the assistant assumed that the web UI's failure to render was an API problem rather than a frontend issue — this turned out to be correct, but it was a hypothesis that needed testing.

There's also a subtle architectural assumption embedded in the diagnostic approach: the assistant treated each kuri node as a self-contained entity that should know about its peers. In the deployed configuration, the kuri nodes were running as direct S3 endpoints (port 8079) with their own web UIs (port 9010). The cluster topology feature was designed for a different architecture — one where stateless S3 proxy frontends discover and route to backend storage nodes. The assistant was using the topology API in a context where the separation between proxy and storage layers was not yet fully implemented, which contributed to the confusion.

Input Knowledge Required

To understand this message, a reader needs to know: what JSON-RPC is and how it differs from REST; that the FGW system uses a multi-node architecture with YugabyteDB for metadata storage; that the kuri daemon exposes both an S3 API and a web UI; that the cluster topology feature is meant to visualize the distributed system's node graph; and that the deployment was on physical machines with specific IP addresses (10.1.232.x range). The reader also needs to understand that the assistant was working within an Ansible-driven deployment pipeline where configuration is managed through environment files and systemd services.

Output Knowledge Created

The message produced several pieces of actionable knowledge: the cluster topology API is accessible via JSON-RPC at /rpc/v0 with method RIBS.ClusterTopology; the API returns empty results when no backend nodes are configured; the web UI's failure to render topology is caused by missing data rather than a frontend bug; and the diagnostic path for such issues involves first verifying the API endpoint, then checking the response payload, then tracing the configuration source. This knowledge directly led to the fix — adding FGW_BACKEND_NODES to the settings — and ultimately to a fully functional cluster topology visualization.

The Thinking Process

The assistant's thinking process, as reconstructed from the message and its surrounding context, follows a classic debugging pattern: observe the symptom (blank UI), form a hypothesis (API not returning data), test the hypothesis (curl the API), interpret the result (HTML means wrong endpoint), correct the approach (use JSON-RPC), and gather evidence (empty topology). Each step builds on the previous one, and the reasoning is transparent enough to follow in the message text itself. The assistant didn't just run commands blindly — it interpreted each response and adjusted its approach accordingly.

This message, for all its apparent simplicity, captures the essence of distributed systems debugging: a single empty JSON response that contains the seed of its own resolution, waiting for someone to read the source code, understand the configuration model, and connect the dots between an environment variable and a web UI component that wasn't rendering.