The Case of the Swapped Ports: A Debugging Epiphany in Distributed Systems Monitoring

Introduction

In the middle of building a horizontally scalable S3-compatible storage system, a developer encountered a puzzling contradiction: the configuration looked correct, but the data told a different story. This article examines a single message from a coding session—message 903—where the assistant discovered that two web UI ports in a Docker Compose test cluster were returning data from the wrong backend nodes. This seemingly small observation triggered a cascade of debugging insights and revealed deeper truths about distributed system monitoring, configuration management, and the subtle ways that state can diverge from expectation in containerized environments.

The Message

Here is the subject message in its entirety:

The ports are swapped! Port 9010 is returning kuri-2's data and port 9011 is returning kuri-1's data. The nginx config looks correct but something is off. Let me restart the webui container:

>

`` $ cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose restart webui 2>&1 Container test-cluster-webui-1 Restarting Container test-cluster-webui-1 Started ``

The Context: A Cluster Monitoring Infrastructure Under Construction

To understand why this message matters, we must step back into the broader project. The developer was building a horizontally scalable S3 architecture for a Filecoin Gateway system. The architecture consisted of three layers: stateless S3 frontend proxies that accepted client requests, Kuri storage nodes that held the actual data, and a shared YugabyteDB database that stored object routing metadata. Each Kuri node operated independently with its own keyspace, while the S3 proxy routed requests to the appropriate backend based on object metadata.

A significant portion of the recent work had focused on implementing a cluster monitoring dashboard. The developer had added real-time metrics collection, I/O throughput charts, latency distribution visualizations, and a cluster topology view that distinguished between S3 frontend nodes (shown in blue) and Kuri storage nodes (shown in green). The monitoring system relied on a WebSocket RPC interface and a new /api/stats HTTP endpoint that returned per-node statistics such as storage used, groups count, and requests per second.

The test cluster was orchestrated with Docker Compose and included two Kuri nodes (kuri-1 and kuri-2), an S3 proxy, a YugabyteDB instance, and an nginx container that served as a reverse proxy for the web UIs. The nginx configuration mapped port 9010 to kuri-1's web interface and port 9011 to kuri-2's web interface. This mapping was explicitly defined in the generated nginx configuration file and appeared to be correct upon inspection.

The Discovery: When Configuration and Reality Diverge

The immediate trigger for this message was a discrepancy the developer noticed while debugging the cluster topology display. Earlier, they had verified that cross-node communication worked—kuri-1 could successfully fetch the /api/stats endpoint from kuri-2 using the internal Docker network hostname kuri-2:9010. They had also confirmed that the FGW_NODE_ID environment variables were correctly set to kuri-1 for the kuri-1 container and kuri-2 for the kuri-2 container. The settings files in the configuration directory also contained the correct values.

Yet when the developer queried the web UI endpoints through the nginx proxy, something was clearly wrong. Querying port 9010 (which should have reached kuri-1) returned data identifying itself as kuri-2, and querying port 9011 (which should have reached kuri-2) returned data identifying itself as kuri-1. The ports were swapped.

This discovery was made through a simple but effective debugging technique: the developer used curl to fetch the /api/stats endpoint through each public port and inspected the nodeId field in the JSON response. The command sequence shown in the preceding messages reveals this:

curl -s http://localhost:9010/api/stats | jq .
# Returns: {"nodeId": "kuri-2", ...}

curl -s http://localhost:9011/api/stats | jq .
# Returns: {"nodeId": "kuri-1", ...}

This was a moment of clarity—an "aha" realization that the developer expressed with the exclamation "The ports are swapped!" The observation contradicted the nginx configuration, which explicitly mapped port 9010 to kuri-1 and port 9011 to kuri-2.

Assumptions and Their Consequences

The developer operated under several assumptions during this debugging session. First, they assumed that the nginx configuration file accurately reflected the runtime behavior of the nginx container. This is a reasonable assumption—configuration files are the source of truth for reverse proxy behavior—but it failed to account for the possibility that the nginx container might have stale routing information or that the configuration file might not have been reloaded after changes.

Second, the developer assumed that restarting the webui container would fix the problem. This assumption was based on the observation that "nginx webui container sometimes needs restart after kuri nodes restart to pick up correct routing," which was listed as a known issue in the project summary. The developer had internalized this operational quirk and reached for the restart command as a first-line troubleshooting step.

Third, there was an implicit assumption that the container orchestration layer (Docker Compose) would maintain consistent state between the nginx proxy and the backend services. In a properly functioning system, restarting backend services should not cause the reverse proxy to swap its routing table. The fact that this was a known issue suggests a deeper architectural fragility in the test cluster setup.

The Restart Decision: A Pragmatic Response

The developer's decision to restart the webui container was pragmatic and grounded in operational experience. Rather than diving into nginx configuration debugging, examining container networking, or checking for race conditions in the startup sequence, the developer chose the fastest path to resolution: restart the proxy container.

This decision reflects a common pattern in distributed systems debugging: when the configuration looks correct but behavior is wrong, the problem is often a stale state or a caching issue. Restarting the container forces a fresh read of the configuration and re-establishes connections to backend services. In this case, the restart was expected to cause nginx to re-resolve the backend hostnames (kuri-1:9010 and kuri-2:9010) and correctly map them to the appropriate ports.

The command used was straightforward:

cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose restart webui

This restarted the webui container without recreating it, preserving the container's filesystem and configuration while restarting the nginx process inside.

Input Knowledge Required

To fully understand this message, one needs to know several pieces of context:

  1. The test cluster architecture: Two Kuri storage nodes (kuri-1, kuri-2) with web UIs served through an nginx reverse proxy on ports 9010 and 9011.
  2. The monitoring infrastructure: A newly added /api/stats HTTP endpoint that returns per-node JSON statistics including a nodeId field.
  3. The nginx configuration: A generated configuration file that explicitly maps port 9010 to http://kuri-1:9010 and port 9011 to http://kuri-2:9010.
  4. The known issue: The project documentation listed "nginx webui container sometimes needs restart after kuri nodes restart to pick up correct routing" as a known problem.
  5. The debugging history: The developer had just implemented cross-node stats fetching and was verifying that the cluster topology display showed accurate data from both nodes.

Output Knowledge Created

This message produced several valuable pieces of knowledge:

  1. A confirmed bug: The port swapping behavior was now a documented, reproducible issue in the test cluster.
  2. A debugging technique: Using the /api/stats endpoint with curl and jq to verify which node is responding on which port proved to be an effective diagnostic method.
  3. A remediation step: Restarting the webui container was confirmed as a potential fix for the swapped port issue.
  4. A question for further investigation: The fact that the nginx config looked correct but behavior was wrong suggests a deeper issue—perhaps related to DNS caching, container startup ordering, or nginx's upstream resolution behavior.

The Thinking Process

The developer's thinking process in this message is a model of effective debugging. It begins with observation (the stats show wrong node IDs), moves to hypothesis formation (the ports are swapped), checks the configuration for consistency (the nginx config looks correct), acknowledges the anomaly (something is off), and executes a remediation step (restart the webui container).

The reasoning is notable for its economy. The developer does not over-investigate the root cause at this moment. They recognize that the port swapping could have multiple causes—stale DNS cache, incorrect upstream resolution, a race condition during container startup—and choose the most efficient path to resolution. This is a pragmatic tradeoff between investigation time and system recovery time.

The message also reveals a developer who is deeply familiar with the system's quirks. The known issue about nginx needing restarts after kuri node restarts was documented from previous experience. This operational knowledge shaped the response and made the debugging faster than it would have been for someone new to the project.

Broader Significance

This message captures a universal experience in distributed systems development: the moment when the system's behavior contradicts its configuration. In a perfect world, configuration files are authoritative and runtime behavior follows them exactly. In practice, distributed systems accumulate subtle state inconsistencies—cached DNS entries, stale connections, misordered startup sequences—that cause behavior to diverge from configuration.

The port swapping issue also highlights the importance of building observability into distributed systems. Without the /api/stats endpoint and the nodeId field in the JSON response, the developer might have spent much longer diagnosing why the cluster topology display showed incorrect data. The monitoring infrastructure that the developer had just built became the tool that revealed the configuration bug.

Finally, this message demonstrates that debugging is not always about finding the root cause. Sometimes it is about recognizing a pattern, applying a known fix, and moving on. The developer did not investigate why the ports were swapped—they simply restarted the container and continued working. In a production system, this might be insufficient, but in a test cluster used for development and iteration, it was the right call.

Conclusion

Message 903 captures a small but significant moment in the development of a distributed S3 storage system. The discovery that ports 9010 and 9011 were returning data from the wrong backend nodes was a puzzle that contradicted the explicit nginx configuration. The developer's response—a pragmatic container restart informed by operational experience—reflects the reality of debugging distributed systems where state and configuration can diverge in unexpected ways. This message serves as a microcosm of the broader challenges in building and operating multi-node systems: the configuration is never the whole story, and the system's behavior is the only truth that matters.