The Phantom Proxy Swap: Debugging a Cluster Topology Mismatch in a Distributed S3 Architecture

Introduction

In the middle of building a horizontally scalable S3-compatible storage system for a Filecoin Gateway, a developer encounters a perplexing anomaly. The monitoring dashboard, designed to show real-time cluster topology across multiple storage nodes, is displaying swapped proxy identities. Querying the web UI on port 9010—which should connect to storage node kuri-1—shows a proxy labeled kuri-2. Querying port 9011—which should connect to kuri-2—shows kuri-1. The nginx configuration is correct. The environment variables are correct. Yet the data is inverted. This single message captures a moment of disciplined debugging: the developer pauses, re-reads their own output, questions their assumptions, and methodically verifies the foundational configuration before proceeding further.

The Message in Full

The subject message, index 899 in the conversation, reads as follows:

The nginx config looks correct - port 9010 goes to kuri-1, port 9011 goes to kuri-2. But the query from 9010 shows proxy "kuri-2" and from 9011 shows proxy "kuri-1". That's strange...

Oh wait, I think I misread. Let me re-check. Actually looking at the response from port 9010 (which should be kuri-1): - Proxies show kuri-2 - that's wrong, it should show kuri-1

Let me check the environment variable: [bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec kuri-1 printenv | grep FGW_NODE_ID FGW_NODE_ID=kuri-1

[bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec kuri-2 printenv | grep FGW_NODE_ID FGW_NODE_ID=kuri-2

Context: What Led to This Moment

To understand this message, one must appreciate the architecture being built. The system follows a three-layer design: stateless S3 frontend proxies (which handle S3 API requests like PUT and GET), Kuri storage nodes (which actually store the data in CAR files and manage disk groups), and a shared YugabyteDB database (which stores object routing metadata). The frontend proxies use round-robin to distribute writes across storage nodes and YCQL lookups to find data on reads. Each storage node runs its own web UI for monitoring, exposed through an nginx reverse proxy on ports 9010 and 9011.

In the messages immediately preceding this one, the developer had been working on cross-node statistics aggregation. The ClusterTopology RPC method, which returns the full cluster state including all storage nodes and proxies, had initially only reported local metrics. The developer had just implemented a fix: adding a /api/stats HTTP endpoint to each node and modifying ClusterTopology to fetch stats from remote nodes via HTTP calls. After rebuilding the Docker image and restarting the containers, the developer tested the result and discovered that while both nodes now showed non-zero requestsPerSecond (indicating successful cross-node communication), the proxy identities appeared swapped.

The Reasoning Process: A Window Into Debugging Discipline

What makes this message remarkable is not the bug itself but the developer's cognitive process. The message captures a moment of self-correction that many experienced engineers will recognize.

The developer begins by stating a conclusion: "The nginx config looks correct." This is based on a direct read of the configuration file in message 898, which clearly maps port 9010 to kuri-1:9010 and port 9011 to kuri-2:9010. The developer has already verified this fact empirically—the nginx config is syntactically and semantically correct.

Then comes the anomaly: "But the query from 9010 shows proxy 'kuri-2' and from 9011 shows proxy 'kuri-1'." This is a contradiction. The developer has two pieces of evidence that cannot both be true under the current model: (1) the nginx config routes 9010→kuri-1 and 9011→kuri-2, and (2) the ClusterTopology response shows the opposite proxy assignment.

The developer's first instinct is to doubt their own observation: "Oh wait, I think I misread. Let me re-check." This is a crucial metacognitive move. Before assuming the system is broken, the developer checks whether they have correctly interpreted the data. They re-examine the JSON output from the two RPC calls and confirm: no, the data really is swapped. "Actually looking at the response from port 9010 (which should be kuri-1): Proxies show kuri-2 - that's wrong, it should show kuri-1."

Having confirmed the anomaly, the developer then performs the most fundamental diagnostic check: verifying that the environment variables are set correctly on each container. The FGW_NODE_ID environment variable is the source of truth for each node's identity. If this variable were wrong, the swap would be explained. The developer runs docker compose exec kuri-1 printenv | grep FGW_NODE_ID and docker compose exec kuri-2 printenv | grep FGW_NODE_ID. Both return the expected values: kuri-1 and kuri-2 respectively.

Assumptions and Their Implications

Several assumptions underpin this debugging session, and the developer is consciously testing them.

Assumption 1: The nginx config is the sole routing authority. The developer assumes that the mapping from external port to internal container is entirely determined by the nginx configuration they read. This is a reasonable assumption—nginx is a well-understood reverse proxy—but it ignores the possibility that Docker Compose port mappings, container networking, or even a stale nginx configuration could interfere. The developer does not check whether the nginx container has actually loaded the configuration file they edited.

Assumption 2: FGW_NODE_ID is the correct variable for node identity. The developer assumes that the FGW_NODE_ID environment variable is what the ClusterTopology method uses to determine which proxy to report. Looking at the code in rbstor/diag.go, the ClusterTopology method reads os.Getenv("FGW_NODE_ID") to determine the self node ID, and then iterates over FGW_BACKEND_NODES to discover peers. The proxy information is populated from the self node ID. So if kuri-1 has FGW_NODE_ID=kuri-1, it should report itself as a proxy with ID kuri-1. The fact that it reports kuri-2 instead suggests something deeper is wrong.

Assumption 3: The /api/stats endpoint returns the correct node ID. The developer had added a stats endpoint that returns {"nodeId":"...","groupsCount":...,"storageUsed":...,"requestsPerSecond":...}. When ClusterTopology fetches stats from remote nodes, it uses the returned nodeId to identify which node the stats belong to. If the stats endpoint on kuri-1 somehow returned nodeId: "kuri-2", that would explain the swap. But the developer verified in message 893 that kuri-1's stats endpoint returns {"nodeId":"kuri-1",...}.

Input Knowledge Required

To fully understand this message, a reader needs:

The architecture of the system: The three-layer design of S3 frontend proxies, Kuri storage nodes, and YugabyteDB, and how the monitoring UI fits into this picture.
The nginx routing setup: That ports 9010 and 9011 are exposed through an nginx container that proxies to the internal web UIs of kuri-1 and kuri-2 respectively.
The ClusterTopology RPC: That this method returns a JSON structure with proxies (showing S3 frontend proxy nodes) and storageNodes (showing Kuri storage nodes), and that each node reports its own proxy identity based on FGW_NODE_ID.
The recent changes: That the developer had just modified ClusterTopology to fetch stats from remote nodes via HTTP calls to /api/stats, and that cross-node communication had been verified working.
Docker Compose basics: Understanding docker compose exec to run commands inside containers and printenv to inspect environment variables.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

Verified environment variables: Both containers have correct FGW_NODE_ID values. The swap is not caused by misconfigured node identities.
Confirmed nginx routing: The nginx configuration correctly maps port 9010 to kuri-1 and port 9011 to kuri-2. The swap is not a routing issue.
A narrowed problem space: The bug must be in the ClusterTopology implementation itself—either in how it discovers peers, how it fetches remote stats, or how it assigns proxy identities in the response.
A documented debugging methodology: The message demonstrates a pattern of "check your assumptions, then check your observations, then check the fundamentals" that is valuable as a reproducible approach.

Mistakes and Incorrect Assumptions

The developer makes one notable mistake in this message: they do not immediately consider that the ClusterTopology method might be constructing the proxy list incorrectly. The code in rbstor/diag.go populates the proxies list by iterating over FGW_BACKEND_NODES and checking if each node is a proxy (by attempting an HTTP call). If the iteration order or the proxy-detection logic is flawed, the identities could become swapped.

Specifically, the ClusterTopology method (as modified in message 887) iterates over the backend nodes from FGW_BACKEND_NODES and for each one, tries to fetch /api/stats. If the stats endpoint returns a nodeId, that ID is used. But the proxy identity is determined by selfNodeID—the node's own FGW_NODE_ID. If kuri-1 is fetching stats from kuri-2 and using those stats to populate the proxy list, but then also including itself... the logic could produce the observed swap.

The developer's assumption that "the env vars are correct, so the proxy should be correct" overlooks the possibility that the code is not using the env var correctly. In fact, looking at the subsequent messages (not shown in the subject message), the developer eventually discovers that the ClusterTopology implementation has a bug where it assigns proxy identities based on the remote node's stats rather than the local node's identity. The fix involves ensuring that each node reports its own FGW_NODE_ID as the proxy ID, not the ID from the remote stats fetch.

The Broader Significance

This message captures a universal debugging experience: the moment when a developer encounters a contradiction between what should be true and what the data shows. The response is not to panic or make random changes, but to methodically verify each layer of the stack. The developer checks the configuration, re-reads the output, and verifies the environment variables—all before touching any code.

This discipline is especially important in distributed systems debugging, where the root cause can hide in any of a dozen interacting components. The nginx config could be stale. The environment variables could be wrong. The Docker Compose file could have a typo. The code could have a logic error. The developer's approach—start with the simplest, most fundamental checks and work outward—is a textbook example of effective debugging methodology.

The message also illustrates the value of writing out one's reasoning. The developer's internal monologue ("Oh wait, I think I misread. Let me re-check.") is captured in the conversation, creating a record not just of what was done but of how the developer thought about the problem. This metacognitive documentation is invaluable for anyone reviewing the session later, whether for learning, auditing, or continuing the work.

Conclusion

Message 899 is a small but revealing moment in a larger engineering effort. It shows a developer at a crossroads, faced with a contradiction that could indicate a fundamental flaw in the architecture or a simple misreading of the data. By methodically verifying the configuration and environment variables, the developer narrows the problem space and prepares for the next step: examining the code itself. The message is a testament to the importance of disciplined debugging, self-doubt as a virtue, and the value of writing down one's reasoning process. In the end, the phantom proxy swap is not a ghost in the machine but a logic error in the ClusterTopology implementation—and the developer's methodical approach ensures it is found and fixed efficiently.