The Phantom Proxy: Debugging a Topology Identity Crisis in a Distributed S3 Cluster

A Single Line of Code That Led Nowhere — and Everywhere

In the middle of an intense debugging session for a horizontally scalable S3 architecture, a developer paused to ask a seemingly simple question: Why is kuri-1's topology showing kuri-2 as the proxy? That question, captured in a single message at index 742 of a coding session, set off a chain of investigation that reveals more about the nature of distributed systems debugging than any textbook could. The message itself is brief — barely a line of commentary followed by a file read — but it sits at the crux of a much larger story about assumptions, stale state, and the invisible boundary between application code and infrastructure.

The Message in Full

The subject message reads:

Hmm, kuri-1's topology is still showing "kuri-2" as the proxy. Let me check the code: [read] /home/theuser/gw/rbstor/diag.go 00177| return groups 00178| } 00179| 00180| // ClusterTopology returns the current cluster layout 00181| // Reads FGW_BACKEND_NODES environment variable to discover cluster nodes 00182| func (r *rbs) ClusterTopology() iface.ClusterTopology { 00183| nodesConfig := os.Getenv("FGW_BACKEND_NODES") 00184| if nodesConfig == "" { 00185| return iface.ClusterTopology{ 00186| Proxies: []iface.ProxyInfo{}, 00187| StorageNodes: []iface.StorageNodeInfo{}, 00188| DataFlows: []ifa...

This is not a long message. It is not a dramatic reveal. It is a developer noticing something off, voicing the observation aloud, and reaching for the source code to understand why. But within this tiny gesture lies the entire art of debugging distributed systems.

The Context: A Cluster Emerging from Chaos

To understand why this message matters, we must understand what came before it. The developer had been building a three-layer S3 architecture consisting of stateless frontend proxies (the S3 API layer), Kuri storage nodes (the actual data persistence layer), and a shared YugabyteDB database for metadata. The architecture followed a roadmap that called for horizontal scalability: any number of frontend proxies could sit in front of any number of storage nodes, with routing handled by a configurable backend pool.

The session had already been a rollercoaster. Earlier, the developer had discovered that both Kuri nodes were crashing due to a Go 1.22 HTTP route conflict between HEAD / and GET /healthz — a subtle change in the standard library's ServeMux behavior that broke the health check endpoint. They had replaced the standard mux with a custom handler. They had fixed the web UI container by replacing a placeholder with an Nginx reverse proxy. They had discovered that the S3 proxy returned internal server errors because the S3Objects table lacked a node_id column, requiring manual schema correction in YugabyteDB. They had implemented a full cluster metrics collection system in rbstor/cluster_metrics.go with a rolling 10-minute window for throughput, latency, error rates, and active requests.

Most recently — and most relevant to our message — they had discovered that the React frontend was showing empty charts and "No cluster nodes configured" despite the backend returning perfectly valid data. The culprit was a classic Go-to-JavaScript mismatch: Go's default JSON marshaling uses PascalCase field names (Proxies, StorageNodes, Timestamps), while the React frontend expected camelCase (proxies, storageNodes, timestamps). The developer had painstakingly added json:"camelCase" tags to every struct in the interface definitions, rebuilt the Docker image, restarted the containers, and verified that the JSON now flowed in the correct format.

The Moment of Discovery

After the JSON fix was deployed, the developer ran a verification test. They queried the ClusterTopology RPC endpoint through kuri-1's web UI on port 9010:

{"id":1,"jsonrpc":"2.0","result":{"proxies":[{"id":"kuri-2",...}],"storageNodes":[...]}}

The JSON was now camelCase. The data was flowing. But something was wrong: the proxy listed was kuri-2, not kuri-1. The developer was querying kuri-1's web interface, yet the topology reported that kuri-2 was the active proxy. This was not a formatting issue — it was a semantic bug. The system was misidentifying its own components.

The developer's immediate reaction, captured in message 742, was to suspect the code. The ClusterTopology() function in rbstor/diag.go was responsible for reading the FGW_BACKEND_NODES environment variable and constructing the cluster layout. If the function was incorrectly parsing the node list or misassigning roles, that would explain why kuri-1 thought kuri-2 was the proxy. The developer opened the file to trace the logic.

The Assumption and Its Blind Spot

This is where the message becomes a case study in debugging psychology. The developer made a perfectly reasonable assumption: the bug was in the Go code. The ClusterTopology() function was the component that determined which nodes were proxies and which were storage nodes. If it was returning the wrong proxy ID, the code must have a bug.

The assumption was wrong. The code was correct.

What the developer discovered after reading the file was that ClusterTopology() reads FGW_BACKEND_NODES and FGW_NODE_ID from the environment. The environment variables on kuri-1 were correctly set:

FGW_NODE_ID=kuri-1
FGW_NODE_TYPE=storage
FGW_BACKEND_NODES=kuri-1:http://kuri-1:8078,kuri-2:http://kuri-2:8078

The code parsed these correctly. The issue was not in the Go application at all — it was in the infrastructure layer. The Nginx reverse proxy container that served the web UI on port 9010 had been configured with a stale configuration file. When the developer restarted the Kuri containers, the Nginx container was not restarted, so it continued to route requests based on an outdated configuration that pointed to kuri-2 instead of kuri-1.

The fix was trivial: docker restart test-cluster-webui-1. After that, the topology correctly showed kuri-1 as the proxy.

Why This Matters: The Two Layers of Debugging

This message illuminates a fundamental truth about debugging distributed systems: there are always two layers of state to consider — the application state and the infrastructure state. The developer had spent considerable effort fixing the application layer (JSON serialization, route conflicts, database schemas) and naturally assumed that any remaining bug must also be in the application layer. But the Nginx container was a piece of infrastructure that existed independently of the Go application code. It had its own configuration file, its own lifecycle, and its own state that could become stale.

The developer's instinct to "check the code" was correct in the abstract — one should always verify assumptions by reading the source — but the code turned out to be a red herring. The real answer was in the deployment configuration, not the application logic.

The Thinking Process Visible in the Message

The message reveals a structured debugging methodology even in its brevity:

Observation: The developer noticed an inconsistency in the output — kuri-1's topology showed kuri-2 as the proxy.
Hypothesis formation: The developer hypothesized that the ClusterTopology() function was the source of the error.
Evidence gathering: The developer read the source code of ClusterTopology() to understand its logic.
Analysis: The developer examined the environment variables to verify that the inputs to the function were correct.
Refinement: When the code appeared correct, the developer broadened the search to include the infrastructure layer. This is textbook systematic debugging. The developer did not jump to conclusions or randomly restart containers. They followed the data: from the RPC output, to the source code, to the environment variables, to the Nginx configuration. Each step eliminated a possible cause until only the infrastructure remained.

Input Knowledge Required

To fully understand this message, one needs:

Knowledge of the cluster architecture: The three-layer model of S3 proxies → Kuri storage nodes → YugabyteDB, and how FGW_BACKEND_NODES encodes the cluster topology.
Understanding of Go JSON serialization: How Go's encoding/json package maps struct fields to JSON keys, and why adding json:"camelCase" tags changes the output.
Familiarity with Docker Compose: How containers are created, how environment variables are injected, and why restarting one container does not automatically restart others.
Knowledge of Nginx reverse proxying: How Nginx routes HTTP requests to backend services based on configuration files.
Understanding of RPC debugging: How websocat and JSON-RPC are used to probe the internal state of running services.

Output Knowledge Created

The message and its resolution produced several valuable pieces of knowledge:

The topology discovery logic was confirmed correct: The ClusterTopology() function in rbstor/diag.go correctly parses FGW_BACKEND_NODES and FGW_NODE_ID to determine node roles.
The Nginx configuration lifecycle was identified as a dependency: The web UI container's Nginx configuration is loaded at startup and does not update automatically when the backend configuration changes.
A debugging heuristic was reinforced: When the application code appears correct, check the infrastructure layer — containers, configuration files, and network routing.
The cluster monitoring pipeline was validated end-to-end: With the Nginx restart, the full chain from metrics collection through JSON-RPC to React frontend was confirmed working.

The Broader Lesson

Message 742 is a microcosm of distributed systems debugging. It shows how easily a developer can be led down the wrong path by a reasonable assumption. The code should have been the source of the bug — it was the component responsible for topology discovery, it had been recently modified, and it was the obvious place to look. But the actual bug was in a completely different layer: a stale Nginx configuration in a container that had not been restarted.

This is the kind of bug that haunts production systems. It is not a logic error or a type mismatch. It is a state synchronization problem between components that have no formal dependency relationship. The Kuri container and the Nginx container are both part of the same Docker Compose setup, but there is no mechanism to ensure that restarting one triggers a restart of the other. The developer had to discover this gap empirically.

The message also demonstrates the value of reading source code even when the code is not the problem. By reading ClusterTopology(), the developer confirmed that the application logic was sound, which narrowed the search space to the infrastructure layer. Without that confirmation, the developer might have wasted time adding debugging output to the Go code or rewriting the topology discovery logic. The act of reading the code was itself a debugging tool.

Conclusion

Message 742 is a single data point in a long debugging session, but it captures a universal experience in distributed systems development: the moment when the data says one thing, the code says another, and the developer must bridge the gap between them. The developer's response — to read the source code, verify the inputs, and systematically eliminate possibilities — is a model of disciplined debugging. The resolution — a simple container restart — is a reminder that in distributed systems, the simplest explanation is often the one that crosses layer boundaries.

The phantom proxy that showed kuri-2 where kuri-1 should have been was not a bug in the code. It was a bug in the deployment, a mismatch between the running state of the infrastructure and the expectations of the application. And the only way to find it was to follow the data, question every assumption, and never stop at the first layer of explanation.