The Moment of Discovery: Debugging a Phantom Port Swap in a Distributed S3 Cluster

Introduction

In the midst of building a horizontally scalable S3-compatible storage system, a developer encountered one of the most elusive classes of bugs in distributed systems: a symptom that looks like a configuration error but is actually a transient state issue caused by container lifecycle ordering. Message 898 in this coding session captures the precise moment of noticing the anomaly and the beginning of a diagnostic chain that would ultimately reveal a stale DNS cache inside an nginx reverse proxy container.

The message itself is deceptively brief—a single observation followed by a command to inspect a configuration file. But it sits at the inflection point of a debugging journey, where raw data gives way to hypothesis formation, and where the developer's mental model of the system is tested against observed reality.

The Subject Message

The full text of the subject message reads:

I see - from kuri-1's perspective (port 9010), the proxy shows kuri-2, and from kuri-2's perspective (port 9011), the proxy shows kuri-1. This is because the nginx routing might be swapped. Let me check:

>

``` [bash] cat /data/fgw2/config/nginx.conf events { worker_connections 1024; }

>

http { # kuri-1 web UI on port 9010 server { listen 9010;

>

location / { proxy_pass http://kuri-1:9010; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; ... ```

Context and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the events that led to it. The developer had just completed a significant feature: cross-node metric aggregation for the cluster monitoring dashboard. The ClusterTopology RPC method, which returns the state of all storage nodes and proxies in the cluster, had been updated to fetch live statistics from remote nodes via a new /api/stats HTTP endpoint rather than showing zeros for non-local nodes.

After rebuilding the Docker image and restarting the containers, the developer generated test traffic and verified that both Kuri storage nodes were showing requestsPerSecond: 0.5—evidence that the round-robin load balancing was working and that stats were being aggregated correctly. This was a moment of success.

But then came the observation that triggered the subject message. When querying the cluster topology through kuri-1's web UI on port 9010, the proxy list showed kuri-2. When querying through kuri-2's web UI on port 9011, the proxy list showed kuri-1. This was backwards. The developer's immediate hypothesis was that the nginx routing configuration had the ports swapped—that port 9010 was actually routing to kuri-2 and vice versa.

The motivation for writing this message was the cognitive dissonance between the expected behavior (port 9010 → kuri-1, port 9011 → kuri-2) and the observed behavior (port 9010 shows kuri-2's proxy, port 9011 shows kuri-1's proxy). The developer needed to resolve this contradiction before proceeding.

The Reasoning Process: Hypothesis Formation

The message reveals a specific chain of reasoning. The developer had just verified that the /api/stats endpoint worked correctly when accessed directly inside the Docker network—wget http://kuri-2:9010/api/stats from inside kuri-1's container returned kuri-2's data correctly. This meant the cross-node communication path was functional.

The anomaly only appeared when going through the external ports (9010 and 9011), which are served by an nginx reverse proxy. The developer's mental model of the system architecture included this nginx layer as the entry point for web UI traffic. If the internal API was correct but the external view was wrong, the most likely culprit was the routing configuration in nginx.

This is a classic debugging heuristic: when a system has multiple layers, and behavior differs between internal and external access points, the boundary layer between them is the prime suspect. The developer's hypothesis—"the nginx routing might be swapped"—was the simplest explanation that fit the data.

Assumptions Made

The developer made several assumptions in this message, most of which were reasonable:

  1. The nginx config file is the authoritative source of routing truth. The developer assumed that reading the generated nginx.conf would definitively reveal whether ports were mapped correctly. This assumption was correct—the config file did show the intended mapping (9010 → kuri-1, 9011 → kuri-2).
  2. The anomaly is a static configuration problem. The initial hypothesis assumed a permanent misconfiguration. This turned out to be incorrect—the config was correct, but the nginx container had stale DNS resolution cached from before the Kuri containers were recreated.
  3. The symptom is symmetric. The developer noted that both ports showed the opposite node's data, which is consistent with a swapped mapping. This was a useful observation that narrowed the possibilities.
  4. The nginx config is the right place to look first. Given the symptom pattern, this was the correct starting point for investigation.

The Mistake: A Classic Distributed Systems Pitfall

The developer's initial assumption—that the nginx routing was statically misconfigured—was incorrect. The nginx.conf file showed the correct mapping: port 9010 proxied to http://kuri-1:9010 and port 9011 proxied to http://kuri-2:9010. The configuration was not the problem.

The actual issue was a stale DNS cache inside the nginx container. Docker Compose's internal DNS resolves container names to IP addresses, but these IP addresses can change when containers are recreated. The nginx container had been started before the Kuri containers were recreated with --force-recreate, so it cached the old IP addresses. When the Kuri containers got new IPs, nginx was still routing to the old ones—effectively swapping which node responded on which port from the client's perspective.

This is a well-known class of issue in containerized environments: DNS caching in reverse proxies can cause routing to stale endpoints after container restarts. The fix was simply restarting the nginx container (as seen in message 903), which flushed the DNS cache and resolved the correct IPs.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. The system architecture: Two Kuri storage nodes (kuri-1, kuri-2) each running a web UI on internal port 9010, with an nginx reverse proxy exposing them on external ports 9010 and 9011 respectively.
  2. The recent changes: The developer had just added a /api/stats endpoint and updated ClusterTopology to fetch remote stats via HTTP. The Docker image had been rebuilt and containers recreated.
  3. The debugging context: The developer had verified cross-node communication worked (kuri-1 could fetch stats from kuri-2 internally) but noticed the external ports showed swapped identities.
  4. The tooling: Docker Compose for container orchestration, nginx as a reverse proxy, and the project's convention of using FGW_NODE_ID environment variables for node identity.
  5. The monitoring infrastructure: The cluster topology RPC returns a list of proxies and storage nodes with live metrics, and the web UI renders this data.

Output Knowledge Created

This message, though brief, creates several important pieces of knowledge:

  1. A documented symptom: The observation that external ports 9010 and 9011 appear to serve the wrong node's data is now recorded, along with the initial hypothesis.
  2. A diagnostic trace: The command to inspect nginx.conf establishes a diagnostic pattern for future debugging—when external and internal views disagree, check the routing layer.
  3. A testable hypothesis: The "swapped nginx routing" hypothesis is explicitly stated, making it possible to verify or falsify through config inspection.
  4. A branching point in the debugging session: This message marks the transition from "it works" to "wait, something is off," which is a critical juncture in any development session.

The Thinking Process Revealed

The subject message reveals a developer thinking in real time. The opening phrase "I see" indicates a moment of pattern recognition—the developer is looking at the output of two RPC calls and noticing the inconsistency. The phrasing "from kuri-1's perspective (port 9010), the proxy shows kuri-2" shows the developer reasoning from the user's point of view, mentally mapping external ports to internal services.

The phrase "This is because the nginx routing might be swapped" is a hypothesis in progress. The word "might" signals uncertainty—the developer is proposing an explanation but hasn't confirmed it yet. The follow-through—"Let me check"—is the crucial step that separates productive debugging from speculation. The developer immediately moves to gather evidence by reading the configuration file.

The choice to read /data/fgw2/config/nginx.conf rather than the template file or the docker-compose configuration is telling. The developer wants to see the actual rendered configuration that nginx is using, not the source template. This shows an understanding that what matters is the runtime state, not the declarative configuration.

Broader Significance

This message exemplifies a pattern that recurs throughout software engineering: the moment when observed behavior contradicts expected behavior, and the developer must decide which of their mental models is wrong. In this case, the developer's model of the system had a correct nginx config but didn't account for DNS caching as a source of transient misrouting.

The debugging journey that follows this message (messages 899–906) is a textbook example of systematic elimination: check the config (it's correct), check the environment variables (they're correct), check the logs (not helpful), check the actual runtime state via direct API calls (ports are swapped at runtime), then apply the fix (restart nginx). Each step narrows the possibilities until the true cause is isolated.

For readers unfamiliar with distributed systems debugging, this message demonstrates that the most obvious hypothesis is not always correct, and that the real culprit often lies in the interaction between components rather than in any single component's configuration. The nginx config was correct; the DNS cache was stale. The symptom looked like a configuration error but was actually a state management issue.

Conclusion

Message 898 is a snapshot of a developer at the threshold of discovery. It captures the moment when a successful feature deployment reveals an unexpected side effect, and when a simple hypothesis ("the routing is swapped") begins a chain of investigation that leads to a deeper understanding of the system's runtime behavior. The message is brief, but it contains the seeds of the entire debugging arc that follows: the observation, the hypothesis, the evidence-gathering command, and the implicit understanding that the truth lies in the runtime state, not the static configuration.