Instrumenting for Insight: Debugging Round-Robin Routing in a Distributed S3 Proxy

Introduction

In distributed systems, verifying that load balancing actually works is often harder than implementing it. Code paths that look correct in static analysis can behave unexpectedly under real traffic, and the gap between "should work" and "does work" is where debugging lives. This article examines a single message from a coding session building a horizontally scalable S3-compatible storage system—a moment where the developer runs a targeted diagnostic test after adding instrumentation to debug an uneven traffic distribution between two storage backend nodes.

The message is a bash command that uploads six small files to an S3 frontend proxy, part of a larger debugging sequence triggered by the user's observation that all traffic was being routed to only one of two backend storage nodes (kuri-1), while kuri-2 received none. The assistant had already inspected the round-robin routing code, concluded the logic was correct, and added info-level logging to track backend selection decisions. This test upload is the moment of verification—the attempt to generate observable evidence that would either confirm the routing works or reveal the true nature of the bug.

The Message

sleep 5 && for i in {1..6}; do dd if=/dev/urandom bs=1K count=1 2>/dev/null | \
  curl -s -X PUT --data-binary @- http://localhost:8078/test/rrtest$i.bin \
  -o /dev/null && echo "uploaded rrtest$i"; done

Output:

uploaded rrtest1
uploaded rrtest2
uploaded rrtest3
uploaded rrtest4
uploaded rrtest5
uploaded rrtest6

All six uploads succeeded. The files were named with the prefix rrtest—a deliberate signal that this is a round-robin test, not ordinary load generation. The real result, however, is not visible in this message's output: it lives in the container logs of the s3-proxy, where the newly added logging statements would reveal which backend handled each request.

Context and Motivation

To understand why this message exists, we must trace the debugging chain that led to it. The user had reported (in message 849) that while the cluster monitoring UI showed traffic on the Frontend Proxies table, all of it was going to kuri-1 with none reaching kuri-2. This is a critical observation for a horizontally scalable system: if writes only target one node, the architecture fails to distribute load, and the second node becomes idle storage rather than an active participant in the cluster.

The assistant's response reveals a specific debugging methodology. Rather than immediately assuming the round-robin algorithm was broken, the assistant first inspected the code to understand the routing logic. Reading server/s3frontend/server.go and server/s3frontend/backend_pool.go (messages 850–859), the assistant verified that:

  1. The SelectRoundRobin() method existed and was being called for PUT requests
  2. Both backends (kuri-1 and kuri-2) were configured in the FGW_BACKEND_NODES environment variable
  3. Both backends were reporting healthy via their /healthz endpoints
  4. The configuration files for kuri-2 were correctly generated The code looked correct. Both backends were in the pool, both were healthy, and the round-robin selector should have alternated between them. This is a classic debugging dilemma: the symptom is real (user sees all traffic on one node), but the code appears correct. The assistant's chosen next step was to add instrumentation—info-level logging statements to server.go—so that each round-robin selection would be recorded in the proxy's logs.## The Reasoning Behind the Test Design The test command in message 863 is not arbitrary. Every detail reveals deliberate design: The sleep 5 ensures the s3-proxy container has fully started after the docker compose up -d --force-recreate s3-proxy command in message 862. Container orchestration systems report "running" before the process inside is ready to accept connections; the sleep prevents false negatives from timing errors. The loop of six iterations is intentional. With two backends and a round-robin algorithm, six requests should produce three requests per backend if distribution is perfectly even. More importantly, six is enough to observe a pattern: if the bug is that the round-robin counter is stuck or resetting, six requests might show all going to the same backend, or alternating between one backend and itself (if the second backend is never selected). The 1 KB file size is a deliberate choice. Using dd if=/dev/urandom bs=1K count=1 produces exactly 1024 bytes of random data. This is small enough to be fast (no need to wait for large uploads) but large enough to be a realistic S3 object. The small size also minimizes the chance of timing-related issues (connection timeouts, slow transfers) that could complicate the diagnostic signal. The rrtest filename prefix serves as a marker. In a production system, logs might contain many different request patterns. By naming files rrtest1 through rrtest6, the developer makes it trivially easy to grep for these specific requests in the proxy logs. This is a form of traceability that anticipates the next step: examining logs to correlate each upload with a backend selection decision. The -o /dev/null flag discards the response body. The developer only cares about the HTTP status code (implicitly checked by curl -s which suppresses errors but returns non-zero on failure) and the echo confirmation. The actual S3 response content is irrelevant to the diagnostic.

Assumptions Embedded in the Test

Every test carries assumptions, and this one is no exception. Understanding them is crucial to evaluating whether the test can actually prove or disprove the hypothesis.

Assumption 1: The logging change was applied correctly. The assistant edited server/s3frontend/server.go in message 860, rebuilt the Docker image in message 861, and recreated the s3-proxy container in message 862. The test assumes that this chain of operations succeeded—that the new binary with logging is what's actually running. If the Docker build cached an old layer, or if --force-recreate didn't actually replace the container, the logs would be silent and the test would produce no diagnostic signal.

Assumption 2: The round-robin selection is the only routing path for PUT requests. The test assumes that all PUT requests go through proxyRoundRobin. If there's an alternative code path (e.g., a default fallback that always picks the first backend), the test might show even distribution through round-robin while real traffic follows a different path. The assistant had verified this by reading the server code, but the assumption remains.

Assumption 3: The monitoring UI is accurate. The user's original observation (all traffic to kuri-1) came from the cluster monitoring dashboard. The test assumes that the dashboard's data source (the ClusterTopology RPC endpoint) correctly reflects actual request distribution. If the metrics collection has a bug—for example, if kuri-2's metrics endpoint isn't being polled—the dashboard could show zero traffic even when kuri-2 is handling requests.

Assumption 4: Six requests are sufficient to detect a pattern. If the round-robin is working correctly, six requests should produce 3:3 distribution. But if the bug is intermittent (e.g., a race condition that sometimes skips a backend), six requests might not be enough to trigger it. The test assumes the bug is deterministic and reproducible on demand.

What the Test Does Not Tell Us

The output of this message—six successful uploads—confirms that the S3 proxy is accepting and processing PUT requests. It does not, by itself, reveal which backend handled each request. The true diagnostic value of this test lives in the container logs, which would need to be examined separately.

This is a deliberate separation of concerns. The test generates traffic under controlled conditions; the log analysis (a separate step) reads the evidence. This two-phase approach is common in distributed systems debugging: first create a clean signal, then inspect the recording.

The Thinking Process Visible in the Sequence

Looking at the broader conversation context, we can reconstruct the assistant's reasoning chain:

  1. Observation: User reports uneven traffic distribution (message 849).
  2. Hypothesis formation: The round-robin logic might be broken, or one backend might be unhealthy.
  3. Verification of preconditions: Check that both backends are configured (message 858), both are healthy (message 853), and the round-robin code path exists (messages 850–857).
  4. Instrumentation decision: Since the code appears correct but the symptom persists, add logging to expose the runtime behavior (message 860).
  5. Deployment of instrumentation: Rebuild Docker image and restart the proxy container (messages 861–862).
  6. Controlled test: Generate traffic with identifiable markers to produce log entries (message 863, the subject of this article).
  7. Expected next step: Examine the s3-proxy logs to see which backend was selected for each request, confirming or refuting the round-robin hypothesis. This sequence demonstrates a disciplined debugging approach: resist the temptation to change the algorithm until you have evidence that the algorithm is the problem. The assistant's first instinct was to read the code, not to rewrite it. Only when the code review failed to find the bug did the assistant add instrumentation—the minimum change needed to gather data.

The Broader Architectural Context

This debugging episode occurs within a specific architectural moment. The system being built is a three-layer horizontally scalable S3 architecture:

S3 Frontend Proxy (stateless, port 8078)
  ├── Kuri Node 1 (storage, port 7001)
  └── Kuri Node 2 (storage, port 7002)
       └── Shared YugabyteDB (S3 object routing metadata)

The frontend proxy is designed to be stateless and horizontally scalable—you should be able to run multiple proxy instances, each routing to the same pool of Kuri storage nodes. The round-robin write distribution is a core architectural feature: without it, new objects would accumulate on a single node, defeating the purpose of having multiple storage nodes.

The fact that the user noticed uneven distribution during testing is valuable. It means the monitoring dashboard is working correctly (it surfaced the anomaly), and it means the test cluster is realistic enough to expose real behavior. A bug found in testing is a bug that won't be found in production.

Conclusion

Message 863 appears, on its surface, to be a mundane bash loop uploading six small files. But in context, it is the culmination of a careful diagnostic process: the moment when instrumentation meets traffic, when hypothesis meets evidence. The developer designed this test with specific parameters—file size, iteration count, naming convention, timing—to produce a clean signal that could be correlated with the newly added logging statements in the proxy server.

The test's true output is not the six "uploaded" confirmations, but the log entries that would reveal whether the round-robin algorithm is functioning as intended. This two-phase approach—generate traffic, then inspect logs—is a pattern that recurs throughout distributed systems debugging. It reflects a deeper understanding that in complex systems, you cannot observe behavior without first creating the conditions for observation.

The round-robin debugging episode also illustrates a broader truth about software engineering: the hardest bugs are often the ones where the code looks correct. When static analysis fails to find the defect, the only option is to add instrumentation and observe the system in motion. Message 863 is that moment of observation—a small, carefully designed experiment in the service of understanding how the system actually behaves, as opposed to how it was designed to behave.