The One-Line Fix That Restored Order: Debugging a Port Swap in a Distributed S3 Cluster

The Message

[assistant] [bash] sleep 2 && curl -s http://localhost:9010/api/stats | jq .
{
  "nodeId": "kuri-1",
  "groupsCount": 1,
  "storageUsed": 19447018394,
  "requestsPerSecond": 0
}

At first glance, this message appears trivial: a simple curl command hitting a stats endpoint, with a JSON response confirming that the node behind port 9010 identifies itself as kuri-1. But this single line of output represents the resolution of a subtle and frustrating class of bug — the phantom port swap — and it marks the moment a distributed debugging session finally clicked back into alignment. To understand why this message matters, we must trace the reasoning that led to it, the assumptions that were challenged along the way, and the quiet satisfaction of watching a two-character identifier confirm that the system is once again telling the truth about itself.

The Context: Building a Horizontally Scalable S3 Architecture

The assistant and user had been deep in the trenches of building a horizontally scalable S3-compatible storage system for the Filecoin Gateway. The architecture follows a three-layer design: stateless S3 frontend proxies (port 8078) that route requests to independent Kuri storage nodes, which in turn share object routing metadata via a YugabyteDB cluster. The test cluster had two Kuri storage nodes (kuri-1 and kuri-2), each with its own web UI for monitoring, exposed through an nginx reverse proxy on ports 9010 and 9011 respectively.

The immediate problem being investigated was cross-node metric aggregation. The ClusterTopology RPC endpoint, which powers the monitoring dashboard, was only showing local statistics. The assistant had just implemented a fix: adding a lightweight /api/stats HTTP endpoint to each Kuri node, and modifying ClusterTopology to fetch stats from remote nodes via HTTP calls. After rebuilding the Docker image and restarting the containers, the assistant began verifying the fix.

The Discovery: A Silent Port Swap

The first verification attempt (message 896) looked promising — both storage nodes showed requestsPerSecond: 0.5, indicating traffic was being distributed and stats aggregated. But something odd caught the assistant's eye: the proxies array in the response from port 9010 (which should be kuri-1's web UI) showed kuri-2 as the proxy. This was suspicious. A quick check of port 9011 confirmed the inversion — it showed kuri-1 as the proxy. The ports were swapped.

This is the kind of bug that can drive a developer to despair. The nginx configuration file was correct: port 9010 explicitly proxied to kuri-1:9010, and port 9011 to kuri-2:9011. The environment variables on each container were correct: FGW_NODE_ID=kuri-1 on kuri-1, FGW_NODE_ID=kuri-2 on kuri-2. The config files on disk were correct. Everything said the right thing, but the system was lying. Port 9010 was returning kuri-2's data, and port 9011 was returning kuri-1's data.

The Diagnosis: Nginx DNS Caching and Container Restarts

The assistant's debugging process reveals a methodical approach to troubleshooting. First, the assistant verified the nginx configuration by reading the generated config file — it was correct. Then the assistant checked the environment variables inside each container using docker compose exec — also correct. Then the assistant checked the settings files on disk — correct again.

At this point, the assistant could have gone down a rabbit hole of debugging the stats handler code, checking for race conditions, or investigating whether the FGW_NODE_ID environment variable was being read at the wrong time. Instead, the assistant recognized the pattern. The summary at the top of the conversation had already documented a known issue: "nginx webui container sometimes needs restart after kuri nodes restart to pick up correct routing."

This is a crucial insight. The nginx container had been running since before the kuri nodes were restarted with the new Docker image. During the restart, the kuri containers got new IP addresses on the Docker internal network. Nginx, which resolves upstream hostnames at startup (or caches DNS lookups), was still pointing to the old IP addresses — or more precisely, the DNS resolution order had changed, causing the hostname kuri-1 to resolve to the IP of the kuri-2 container, and vice versa.

The fix was elegantly simple: restart the nginx container. Message 903 shows the assistant running docker compose restart webui. Then, after a two-second sleep to allow the container to initialize, the assistant runs the verification command in message 904.

The Verification: A Single Curl Command

The command itself is straightforward: sleep 2 && curl -s http://localhost:9010/api/stats | jq .. The sleep 2 gives the nginx container time to start accepting connections. The curl hits the stats endpoint through the nginx proxy on port 9010. The jq . formats the JSON output for readability.

The response is the moment of truth:

{
  "nodeId": "kuri-1",
  "groupsCount": 1,
  "storageUsed": 19447018394,
  "requestsPerSecond": 0
}

The nodeId field reads "kuri-1". Port 9010 is now correctly routing to kuri-1. The port swap is resolved. The fix worked.

Why This Message Matters

This message is a masterclass in operational debugging. It demonstrates several important principles:

1. Trust but verify. The assistant had just restarted the webui container and could have assumed the fix worked. Instead, the assistant explicitly verified by querying the endpoint. In distributed systems, assumptions are the enemy of reliability.

2. Know your known issues. The conversation summary had documented the nginx restart issue. When the assistant encountered the port swap, rather than debugging from scratch, the assistant recognized the pattern and applied the known fix. Good documentation of operational quirks pays dividends.

3. The simplest fix is often the right one. Before diving into code changes, DNS configuration, or network debugging, the assistant tried a container restart. This is the digital equivalent of "turn it off and on again" — and it worked.

4. Single points of verification are powerful. The entire verification is one field in one JSON response: nodeId: "kuri-1". A single boolean condition — does the node ID match the expected value? — confirms the entire fix. The assistant didn't need to run a full integration test or check every endpoint. One targeted query was sufficient.

Assumptions and Potential Pitfalls

The assistant made several assumptions in this debugging sequence:

The nginx config was correct. This was verified by reading the file, but the assistant assumed the config hadn't been corrupted or overwritten during the container restart cycle.
The stats endpoint was reliable. The assistant assumed that if the endpoint returned nodeId: "kuri-1", then the node was genuinely kuri-1. This is a reasonable assumption, but in theory the FGW_NODE_ID environment variable could have been stale or misread.
A two-second sleep was sufficient. The assistant assumed that two seconds was enough time for the nginx container to fully initialize. If the container had a slow startup, the curl might have failed or returned a stale response.
The port swap was the only issue. The assistant assumed that once the ports were correctly mapped, the cross-node stats aggregation would work correctly. In fact, the earlier verification (message 896) had already shown that both nodes were reporting stats — the only problem was the port mapping.

The Deeper Lesson: Debugging Is About Eliminating Variables

What makes this message interesting is not the command itself, but the chain of reasoning that led to it. The assistant had spent the previous twenty messages building a cross-node stats aggregation system. The code changes were complex: new HTTP endpoints, modified RPC handlers, environment variable parsing, and HTTP client calls between containers. When the first verification showed swapped ports, the assistant could have assumed the code changes were wrong and started debugging the new code.

Instead, the assistant recognized that the symptom (swapped ports) was inconsistent with the code change (which added a new endpoint but didn't modify routing). The most likely cause was an operational issue — stale nginx routing — not a code bug. By isolating the variable (nginx routing) and testing it independently (querying the stats endpoint directly), the assistant confirmed that the code was correct and the infrastructure needed a restart.

This is the essence of systems debugging: when a complex change produces unexpected results, don't assume the change is wrong. Test each layer independently. Verify the infrastructure before questioning the code. And always check the simple things first — sometimes a container restart is all you need.

Conclusion

Message 904 is a quiet victory. It doesn't contain dramatic code changes or architectural revelations. It's a single curl command and a JSON response. But that response — "nodeId": "kuri-1" — represents the successful resolution of a subtle infrastructure bug, the validation of a complex cross-node stats aggregation feature, and the restoration of trust in the test cluster's monitoring infrastructure. In the world of distributed systems debugging, sometimes the most satisfying fix is the one that requires no code changes at all — just a restart and a verification.