The Moment of Verification: Debugging DNS Caching in a Distributed S3 Cluster

Message Under Analysis

The following message is from an assistant debugging a horizontally scalable S3 storage cluster built for a Filecoin Gateway project. The assistant is working with a three-layer architecture: stateless S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB database. At this point in the session, the assistant has been building and debugging a real-time cluster monitoring dashboard that aggregates metrics across multiple storage nodes.

[assistant] Now it's correct. The nginx container needed a restart after the kuri containers were recreated (DNS caching issue). Let me verify the ClusterTopology now: [bash] for i in {1..10}; do dd if=/dev/urandom bs=10K count=1 2>/dev/null | curl -s -X PUT --data-binary @- http://localhost:8078/test/final$i.bin -o /dev/null; done && sleep 2 && echo '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' | websocat -n1 ws://localhost:9010/rpc/v0 2>&1 | jq '.' { "id": 1, "jsonrpc": "2.0", "result": { "proxies": [ { "id": "kuri-1", "address": "localhost", "status": "healthy", "requestsPerSecond": 0.5, "activeConnections": 0, "backendPool": null, "latencyMs": 0, "errorRate": 0 } ], "storageNodes": [ { "id": "kuri-1", "address": "http://kuri-1:8078", "status": "healthy", "storageUsed": 19447069594, "storageTotal": 0,...

The Context: A Long Debugging Session

This message represents the culmination of an extensive debugging session that spanned dozens of interactions. The assistant had been building a horizontally scalable S3-compatible storage system where multiple Kuri storage nodes operate independently while sharing object routing metadata via YugabyteDB. The architecture follows a strict three-layer hierarchy: stateless S3 frontend proxies on port 8078 route requests to Kuri storage nodes, which in turn store data and metadata in a shared database.

The immediate problem being solved was a monitoring issue. The assistant had implemented a ClusterTopology RPC method designed to return live statistics about all nodes in the cluster, including request rates, storage usage, and health status. However, when querying this endpoint, the data was incomplete — only the local node's metrics were being reported, while remote nodes showed zero values. The root cause was that the ClusterTopology implementation in rbstor/diag.go only filled in stats for the node being queried, without making RPC calls to fetch data from peer nodes.## The Discovery: Swapped Ports and Stale DNS

The debugging session had taken a sharp turn when the assistant noticed something peculiar. After implementing a cross-node stats aggregation mechanism — adding a /api/stats HTTP endpoint to each Kuri node and updating ClusterTopology to fetch remote stats via HTTP — the initial tests showed that the proxy information was wrong. When querying port 9010 (which should route to kuri-1), the response showed kuri-2 in the proxies list. Conversely, port 9011 showed kuri-1. The ports were swapped.

This was a confusing moment. The nginx configuration file at /data/fgw2/config/nginx.conf clearly mapped port 9010 to http://kuri-1:9010 and port 9011 to http://kuri-2:9010. The environment variables (FGW_NODE_ID) were correctly set on each container. Yet the data was reversed.

The assistant systematically ruled out configuration errors by checking environment variables, config files, and even the newly created /api/stats endpoint directly inside the containers. The docker compose exec kuri-1 printenv | grep FGW_NODE_ID command confirmed kuri-1 had the correct identity, and docker compose exec kuri-2 printenv confirmed kuri-2 was also correct. The config files in /data/fgw2/config/kuri-1/settings.env and /data/fgw2/config/kuri-2/settings.env were both accurate.

The Root Cause: Nginx DNS Caching

The breakthrough came when the assistant queried the /api/stats endpoint through the nginx proxy from the host machine:

curl -s http://localhost:9010/api/stats | jq .
{"nodeId": "kuri-2", ...}

curl -s http://localhost:9011/api/stats | jq .
{"nodeId": "kuri-1", ...}

This confirmed the swap was happening at the nginx level, not in the application code. The nginx container was routing traffic to the wrong backend because it had cached DNS resolutions from before the containers were recreated. When docker compose up -d --force-recreate kuri-1 kuri-2 s3-proxy was run, Docker assigned new internal IP addresses to the containers, but nginx had cached the old DNS lookups.

The fix was simple: restart the nginx container. The assistant ran docker compose restart webui, and after a brief sleep, the /api/stats endpoint returned the correct data — port 9010 showed kuri-1, port 9011 showed kuri-2.## The Verification: A Carefully Constructed Test

The subject message begins with the triumphant declaration "Now it's correct." This is the moment of validation after the nginx restart fixed the DNS caching issue. But the assistant doesn't simply declare victory — they immediately construct a thorough verification test.

The test command is worth examining in detail:

for i in {1..10}; do dd if=/dev/urandom bs=10K count=1 2>/dev/null | \
  curl -s -X PUT --data-binary @- http://localhost:8078/test/final$i.bin \
  -o /dev/null; done && sleep 2 && \
  echo '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' | \
  websocat -n1 ws://localhost:9010/rpc/v0 2>&1 | jq '.'

This command sequence does several things at once. First, it generates 10 random 10KB files and uploads them through the S3 frontend proxy at port 8078. This creates measurable traffic that the cluster monitoring system should capture. The sleep 2 gives the metrics collection system time to record the activity. Then it queries the ClusterTopology RPC endpoint through the web UI at port 9010 (now correctly routing to kuri-1) and formats the JSON response with jq for human readability.

The response shows requestsPerSecond: 0.5 — a non-zero value that proves traffic is being measured. The storageUsed value of 19447069594 bytes confirms that the storage metrics are live and updating. The proxy information shows kuri-1 with status: "healthy", confirming the nginx routing is now correct.

Assumptions Made and Validated

This message reveals several assumptions the assistant was working with. The primary assumption was that nginx would properly route to the correct backend after container recreation. This assumption proved false, and the assistant had to discover the DNS caching behavior empirically.

Another assumption was that the ClusterTopology RPC implementation was functionally correct after the cross-node stats aggregation was added. The verification test confirmed this — both storage nodes now show live requestsPerSecond values, proving the HTTP-based stats aggregation between nodes works.

The assistant also assumed that generating 10 uploads of 10KB each would produce measurable traffic. The resulting 0.5 requests per second value confirms this, though it's worth noting this is an average over the metrics collection window, not an instantaneous rate.

Input Knowledge Required

To understand this message, one needs knowledge of several technical domains. Docker Compose networking and DNS resolution behavior is central — the fact that nginx caches DNS lookups and doesn't re-resolve when containers get new IP addresses is a known Docker pitfall. The Go programming language's http.ServeMux routing semantics are relevant, as the assistant had earlier debugged route conflicts between HEAD / and GET /healthz. The WebSocket RPC protocol used for cluster monitoring requires understanding of JSON-RPC over WebSocket transport. The jq command-line JSON processor is used for formatting output. And the overall architecture of the three-layer S3 system — with stateless proxies, storage nodes, and a shared database — is essential context.

Output Knowledge Created

This message creates several pieces of valuable knowledge. First, it documents a specific operational insight: nginx in Docker Compose environments can cache DNS resolutions, causing routing errors after container recreation. The fix is to restart the nginx container after recreating backend containers.

Second, it validates the cross-node stats aggregation implementation. The ClusterTopology RPC now correctly shows metrics from both kuri-1 and kuri-2, with live requestsPerSecond values. This confirms that the HTTP-based stats endpoint and the remote-fetch logic in rbstor/diag.go are working correctly.

Third, it demonstrates a reliable verification pattern for distributed system debugging: generate traffic, wait for metrics collection, query the monitoring endpoint, and inspect the structured JSON output. This pattern can be reused for future testing.