The Smoking Gun: How Two Curl Commands Exposed a Silent Nginx Routing Swap

Introduction

In the midst of a complex debugging session for a horizontally scalable S3 storage architecture, a single message containing two curl commands and their JSON responses served as the decisive moment that unraveled a subtle but critical infrastructure bug. The message, delivered by an AI assistant debugging a distributed storage test cluster, is deceptively brief—just a pair of HTTP requests to a newly created /api/stats endpoint on two different ports. Yet within those four lines of JSON output lies the entire resolution to a puzzle that had consumed the preceding thirty-seven messages of investigation. This article examines that message in depth: why it was written, what assumptions it challenged, the thinking process it reveals, and the knowledge it produced.

The Message

The subject message reads in its entirety:

[assistant] OK so the configs are correct. Let me debug by checking what the /api/stats endpoint returns: [bash] curl -s http://localhost:9010/api/stats | jq . { "nodeId": "kuri-2", "groupsCount": 1, "storageUsed": 19524382341, "requestsPerSecond": 0 }

>

[bash] curl -s http://localhost:9011/api/stats | jq . { "nodeId": "kuri-1", "groupsCount": 1, "storageUsed": 19447018394, "requestsPerSecond": 0 }

On its surface, this looks like a routine diagnostic step. The assistant has just added a lightweight /api/stats HTTP endpoint to each Kuri storage node, rebuilt the Docker image, and restarted the cluster. Now it is verifying that the endpoint works. But the output reveals something deeply wrong: port 9010, which the nginx configuration explicitly maps to kuri-1, is returning stats with nodeId: "kuri-2". Conversely, port 9011, mapped to kuri-2, returns nodeId: "kuri-1". The ports are swapped.

The Context: Thirty-Seven Messages of Confusion

To understand why this message was written, one must appreciate the debugging journey that preceded it. The assistant had been building a three-layer horizontally scalable S3 architecture: a stateless S3 frontend proxy (port 8078) that routes requests to two independent Kuri storage nodes, which in turn store data in a shared YugabyteDB. A monitoring web UI, served through an nginx reverse proxy, exposes each Kuri node's dashboard on ports 9010 and 9011 respectively.

The trouble began when the assistant noticed that the cluster monitoring UI was not reflecting reality. The round-robin S3 proxy was correctly distributing traffic between both Kuri nodes—uploads alternated between kuri-1 and kuri-2 as expected. Yet the ClusterTopology RPC endpoint, queried through the web UI, showed traffic only on one node. The assistant spent messages 865 through 901 chasing this discrepancy through multiple layers of the system.

First came the logging investigation. The assistant checked whether the S3 proxy's logging was properly configured, discovered that the gw/s3frontend/backend logger wasn't initialized, and added the appropriate environment variable to docker-compose.yml. After restarting, the logs confirmed that the proxy was indeed distributing requests to both backends. The problem was not in the proxy.

Next came the realization that the ClusterTopology RPC implementation in rbstor/diag.go only reported metrics for the local node. For remote nodes, it showed zero values. This was a fundamental design limitation: each Kuri node only knew its own statistics. The assistant implemented a cross-node stats aggregation mechanism, adding a /api/stats HTTP endpoint to each node and modifying ClusterTopology to fetch remote stats via HTTP calls to sibling nodes.

After rebuilding the Docker image and restarting the cluster, the assistant tested the cross-node communication. It worked—kuri-1 could query kuri-2's stats endpoint and vice versa. But then a new anomaly appeared: the ClusterTopology response from port 9010 showed proxy "kuri-2" instead of "kuri-1", and port 9011 showed proxy "kuri-1" instead of "kuri-2". The assistant checked environment variables (FGW_NODE_ID was correct on both nodes), checked the generated settings files (also correct), and checked the nginx configuration (which mapped 9010→kuri-1 and 9011→kuri-2, seemingly correct). Everything looked right on paper, yet the behavior was wrong.## The Reasoning Behind the Message

The subject message represents a deliberate narrowing of the investigative scope. After thirty-seven messages of checking configurations, environment variables, log files, and RPC responses, the assistant had exhausted the obvious suspects. The environment variables were correct. The config files were correct. The nginx configuration was correct. Yet the system was behaving as if the nodes were swapped.

The assistant's reasoning, visible in the preceding message (901), shows a systematic debugging approach: "Let me check the config files" → grep FGW_NODE_ID /data/fgw2/config/kuri-1/settings.env /data/fgw2/config/kuri-2/settings.env → both files contain the correct values. This eliminates the possibility that the configuration generation script (gen-config.sh) produced incorrect files.

With all high-level configuration verified, the assistant needed to determine whether the problem was in the nginx routing layer or in the Kuri nodes themselves. The newly created /api/stats endpoint was the perfect diagnostic tool because it returns the node's own identity directly from the process's environment—no nginx involvement, no RPC layer, no aggregation logic. By curling this endpoint through the nginx proxy (ports 9010 and 9011), the assistant could isolate whether nginx was routing to the wrong backend.

The choice to use curl with jq for formatting is significant. The assistant could have used wget (as in message 893) or inspected the raw HTML. But curl with jq produces clean, parseable JSON output that makes the nodeId field immediately visible. This is a deliberate diagnostic design: the simplest possible query that answers the most fundamental question—"which node am I actually talking to?"

The Assumptions Under Test

This message implicitly tests several assumptions that had been held throughout the debugging session:

  1. The nginx configuration is correct. The generated nginx.conf maps port 9010 to kuri-1:9010 and port 9011 to kuri-2:9010. The assistant had verified this by reading the file directly. But the curl results prove this assumption wrong—or rather, they prove that something between the nginx configuration and the actual routing is broken.
  2. The environment variables are correctly propagated. The assistant had verified FGW_NODE_ID inside each container via docker compose exec. Yet the /api/stats endpoint, which reads FGW_NODE_ID from the environment, returns the wrong identity on each port. This suggests either the environment variable is being overridden at runtime, or the nginx routing is indeed swapping the nodes.
  3. The stats endpoint returns the truth. This assumption is validated by the message itself. The /api/stats endpoint is a thin HTTP handler that serializes the node's identity and basic metrics. There is no caching, no aggregation, no indirection. If it says the node is "kuri-2", then the process serving that request believes it is kuri-2.

The Thinking Process Revealed

The assistant's thinking process in this message is a model of disciplined debugging. The opening phrase "OK so the configs are correct" signals a conclusion drawn from the previous investigation—the settings files and environment variables are not the source of the problem. This is an important mental checkpoint: the assistant is consciously ruling out a class of explanations before proceeding.

The phrase "Let me debug by checking what the /api/stats endpoint returns" reveals the strategic pivot. The assistant is moving from static configuration verification (reading files) to dynamic behavior verification (making live HTTP requests). This is the classic debugging heuristic: when static analysis fails to explain a bug, observe the system in motion.

The two curl commands are executed sequentially, not in parallel. The assistant queries port 9010 first, observes the unexpected result (kuri-2), and then queries port 9011 to confirm the pattern (kuri-1). The symmetry of the swap—9010 returns kuri-2, 9011 returns kuri-1—is itself a clue. A random misconfiguration might produce asymmetric results (e.g., both ports returning the same node), but a clean swap suggests a systematic inversion somewhere in the routing layer.

The Knowledge Produced

This message produces several pieces of critical knowledge:

  1. The nginx proxy is swapping the backends. Port 9010, configured to route to kuri-1, is actually reaching kuri-2. Port 9011, configured to route to kuri-2, is actually reaching kuri-1. This is a definitive finding because the /api/stats endpoint is a direct reflection of the process's identity.
  2. The problem is in the routing layer, not the application layer. Both Kuri nodes are running correctly and reporting their true identities. The application code, the configuration files, and the environment variables are all consistent. The bug must be in how nginx routes incoming connections to the containers.
  3. The cross-node stats aggregation is working correctly. The earlier confusion in the ClusterTopology RPC—where port 9010 showed proxy "kuri-2" and port 9011 showed proxy "kuri-1"—was not a bug in the aggregation logic. It was a correct reflection of the swapped routing. The aggregation code was fetching stats from the node it thought was the sibling, but because the web UI was hitting the wrong node, the "local" and "remote" perspectives were inverted.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of the test cluster's architecture: the three-layer design (S3 proxy → Kuri nodes → YugabyteDB), the port mapping scheme (8078 for S3, 9010/9011 for web UIs via nginx, 7001/7002 for LocalWeb), and the role of the nginx reverse proxy in routing web traffic to the appropriate Kuri node. The reader must also understand that each Kuri node has a unique identity (FGW_NODE_ID) that it reports through the /api/stats endpoint, and that the nginx configuration file (nginx.conf) explicitly maps port 9010 to kuri-1 and port 9011 to kuri-2.

Additionally, the reader benefits from knowing the debugging history: the assistant had previously verified that the S3 proxy's round-robin was working correctly, that the environment variables inside each container were correct, and that the generated settings files contained the right node IDs. This context establishes why the assistant was so confident that the configs were correct, making the swapped output all the more surprising.

The Broader Significance

This message is a textbook example of the "verify the plumbing" debugging technique. When higher-level abstractions (configuration files, environment variables, RPC responses) all appear consistent but the system behaves incorrectly, the correct next step is to insert a minimal, trustworthy probe at the lowest possible level. The /api/stats endpoint is exactly such a probe: it has no business logic, no caching, no dependencies on other services. It simply reads the process's environment and reports it.

The message also illustrates the importance of symmetric testing. Querying both ports and observing the complementary swap (9010→kuri-2, 9011→kuri-1) provides far more information than querying a single port. The symmetry confirms that the bug is in the routing layer rather than a random failure, and it narrows the search space to the nginx configuration or the Docker Compose network setup.

In the broader context of the session, this message is the turning point. After this discovery, the assistant would go on to investigate the nginx configuration generator and discover that the gen-config.sh script was producing a configuration where the server_name directives or proxy_pass URLs were swapped relative to the intended mapping. The fix would involve correcting the configuration generation logic to ensure that port 9010 consistently routes to kuri-1 and port 9011 to kuri-2.

Conclusion

The subject message at index 902 is a masterclass in diagnostic minimalism. In just two curl commands and their responses, the assistant definitively isolated a routing-layer bug that had been masquerading as an application-layer problem for dozens of messages. The message demonstrates the power of building simple, trustworthy diagnostic endpoints into complex distributed systems, the importance of symmetric testing, and the discipline of systematically ruling out explanations before moving to the next hypothesis. For anyone debugging a distributed system, this message serves as a reminder: when the configs look right but the behavior is wrong, go straight to the source and ask the simplest possible question.