The Proxy That Wasn't: A Debugging Epiphany in Distributed Systems Topology

Introduction

In the middle of a complex debugging session for a horizontally scalable S3 architecture, a single message from an AI assistant captures one of the most revealing moments in distributed systems development: the instant when a fix that appears to work perfectly reveals a deeper, subtler problem. The message, delivered after implementing cross-node statistics aggregation for a two-node Kuri storage cluster, reads with the quiet thrill of discovery: "Now both storage nodes are showing requestsPerSecond: 0.5 — traffic is distributed and stats are being aggregated from remote nodes." But then comes the pivot: "Wait, I notice that proxies is showing kuri-2 but the request is going to kuri-1's web UI at port 9010."

This article examines that message in detail: why it was written, the reasoning it reveals, the assumptions it challenges, and the architectural insight it exposes about the nature of distributed system topology discovery.

The Message in Full

The subject message consists of an observation followed by a diagnostic command and its output:

Now both storage nodes are showing requestsPerSecond: 0.5 — traffic is distributed and stats are being aggregated from remote nodes.

Wait, I notice that proxies is showing kuri-2 but the request is going to kuri-1's web UI at port 9010. Let me check what's happening:

`` echo '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' | websocat -n1 ws://localhost:9011/rpc/v0 2>&1 | jq '.' { "id": 1, "jsonrpc": "2.0", "result": { "proxies": [ { "id": "kuri-1", "address": "localhost", "status": "healthy", "requestsPerSecond": 0.5, ... } ], "storageNodes": [ { "id": "kuri-1", "address": "http://kuri-1:8078", "status": "healthy", "storageUsed": 19447018394, ... } ] } } ``

The assistant queries port 9011 (kuri-2's web UI) after having just queried port 9010 (kuri-1's web UI). On kuri-1's UI, the proxies list showed kuri-2. On kuri-2's UI, the proxies list shows kuri-1. Each node reports the other node as the proxy, and itself as a storage node. Something is systematically misidentifying the role of every node in the cluster.

The Context: What Led to This Message

To understand why this message was written, we must trace the debugging arc that preceded it. The assistant had been building a three-layer horizontally scalable S3 architecture consisting of stateless S3 frontend proxies (port 8078), Kuri storage nodes (ports 7001/7002), and a shared YugabyteDB backend. A critical architectural error had been corrected earlier in the session: the assistant had originally been running Kuri nodes as direct S3 endpoints, violating the roadmap's requirement for separate stateless frontend proxy nodes. This was restructured into the proper three-layer hierarchy.

After that correction, the assistant implemented a real-time cluster monitoring dashboard with React frontend components. During testing, a problem emerged: the cluster topology UI only showed traffic on kuri-1, even though round-robin load balancing was distributing requests evenly between both storage nodes. The root cause was that the ClusterTopology RPC method, implemented in rbstor/diag.go, only reported local metrics. For remote nodes, it returned zero values.

The assistant's fix was elegant and pragmatic: add a lightweight /api/stats HTTP endpoint to each node that returns local metrics (node ID, groups count, storage used, requests per second), then modify the ClusterTopology method to fetch stats from remote nodes by making HTTP calls to their /api/stats endpoints. This required adding the endpoint to integrations/web/ribsweb.go, modifying rbstor/diag.go to make cross-node HTTP requests, rebuilding the Docker image, and restarting the cluster.

After rebuilding and generating test traffic, the assistant verified the fix by querying kuri-1's web UI at port 9010. The result showed requestsPerSecond: 0.5 on both storage nodes — the aggregation was working. But then the assistant noticed something anomalous in the response.## The Reasoning Process: From Triumph to Perplexity

The message captures a classic debugging rhythm: first, the satisfaction of a fix working as intended; second, the unsettling observation that something else is subtly wrong; third, the disciplined follow-up to investigate rather than dismiss.

The assistant's initial statement — "Now both storage nodes are showing requestsPerSecond: 0.5 — traffic is distributed and stats are being aggregated from remote nodes" — reflects genuine achievement. The cross-node HTTP stats aggregation is functioning. The round-robin load balancing is distributing writes evenly. The cluster is behaving as a distributed system should.

But the assistant's eye catches an anomaly in the JSON response. The proxies array contains an entry for kuri-2, but the query was directed at kuri-1's web UI on port 9010. Why would kuri-1 report a different node as the proxy? The assistant's reasoning is implicit but clear: if each node reports its own proxy identity, then querying kuri-1 should show kuri-1 in the proxies list. Seeing kuri-2 instead suggests that the role identification logic is inverted — each node is reporting its peer as the proxy rather than itself.

The follow-up command is textbook debugging methodology: query the other node (port 9011 for kuri-2) to see if the pattern is symmetric. The result confirms the suspicion: on kuri-2's UI, the proxies list shows kuri-1. Each node sees the other as the proxy and itself as a storage node. The system is consistently, systematically wrong.

The Underlying Assumption and Its Flaw

The assistant had made a reasonable but incorrect assumption about how node roles are determined. The ClusterTopology method in rbstor/diag.go parses the FGW_BACKEND_NODES environment variable to discover cluster peers. This variable contains entries like kuri-1:http://kuri-1:8078,kuri-2:http://kuri-2:8078. The method iterates over these entries and classifies each one: if the entry's ID matches the local node's FGW_NODE_ID, it's treated as "self" and reported as a storage node with local metrics; otherwise, it's treated as a remote node and reported as a proxy.

The flaw is in the classification logic. The FGW_BACKEND_NODES variable was originally designed for the S3 frontend proxy to discover backend storage nodes. But when the same variable is used by Kuri storage nodes themselves to discover peers, the semantics become ambiguous. A Kuri node sees its own entry in the list, classifies itself as a storage node, and classifies every other entry as a proxy — even though those other entries are also storage nodes, not proxies. The system has no way to distinguish between "a storage node that I should treat as a peer" and "a proxy node that routes requests to me."

This is a fundamental design tension in distributed systems: the same configuration mechanism is being repurposed for two different views of the topology. The S3 frontend proxy uses FGW_BACKEND_NODES to know where to route requests. The Kuri storage nodes use it to know who their peers are for monitoring. But the semantics are different: the proxy sees a list of backends; the storage node sees a list of peers, some of which may be proxies and some of which may be other storage nodes.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, one needs:

The architecture context: The system has three layers — S3 frontend proxies (stateless, port 8078), Kuri storage nodes (ports 7001/7002), and YugabyteDB. Each layer has distinct responsibilities.
The environment variable scheme: FGW_BACKEND_NODES lists cluster peers in id:url format. FGW_NODE_ID identifies the local node. FGW_NODE_TYPE distinguishes frontend from storage roles.
The monitoring architecture: The web UI runs on each Kuri node at port 9010/9011, serving a React frontend that calls RIBS.ClusterTopology RPC to get cluster state. The RPC is implemented in rbstor/diag.go.
The recent fix: The assistant had just added a /api/stats HTTP endpoint and modified ClusterTopology to fetch remote stats via HTTP, enabling cross-node metric aggregation.
The round-robin verification: Earlier testing confirmed that the S3 proxy distributes writes evenly between kuri-1 and kuri-2, so both nodes should show non-zero requestsPerSecond.

Output Knowledge Created by This Message

This message produces several valuable pieces of knowledge:

The cross-node stats aggregation works: Both storage nodes report requestsPerSecond: 0.5, confirming that the HTTP-based remote stats fetching is functional.
The proxy identification is inverted: The proxies field in the topology response contains the wrong node ID, indicating a systematic classification error in the topology discovery logic.
The error is consistent across nodes: Querying both kuri-1 (port 9010) and kuri-2 (port 9011) produces the same pattern — each node reports the other as the proxy. This rules out a transient or node-specific bug.
The fix is incomplete: While the stats aggregation works, the topology classification needs correction. The assistant has identified the next debugging target.

The Deeper Lesson: Topology is a Matter of Perspective

The most profound insight in this message is that topology in a distributed system is not an objective fact — it depends on who is asking and from what vantage point. The S3 frontend proxy sees a list of backend storage nodes. Each storage node sees a list of peers. But neither view inherently captures the distinction between "a node that serves storage" and "a node that proxies requests." The same configuration data, interpreted from different perspectives, yields different classifications.

This is a recurring challenge in distributed systems design. Configuration that works for one component may be semantically ambiguous when reused by another. The assistant's debugging journey — from fixing stats aggregation to discovering role misclassification — demonstrates the iterative nature of distributed systems development. Each fix reveals a new layer of subtlety, and each discovery refines the understanding of how the system actually behaves versus how it was designed to behave.

The message also illustrates a valuable debugging heuristic: when a fix appears to work but produces an unexpected result in a related area, investigate the anomaly immediately rather than moving on. The assistant could have declared victory — the stats were aggregating, the traffic was balanced — but instead noticed the proxy misidentification and followed up. This discipline is what separates surface-level fixes from genuine architectural understanding.

Conclusion

A single message in a debugging session — 20 lines of JSON output and a moment of realization — encapsulates the essence of distributed systems troubleshooting. The assistant built a fix that worked (cross-node stats aggregation) and simultaneously discovered a deeper problem (role misclassification). The message captures the transition from satisfaction to curiosity, from verification to investigation. It reveals the assumptions embedded in configuration design, the perspectival nature of topology, and the importance of questioning results that are "almost right." For anyone building or debugging distributed systems, this message is a miniature case study in the art of noticing when a working fix is telling you something more important than you expected to hear.