The Moment of Misdirection: Debugging a Phantom Routing Bug in a Distributed S3 Proxy

Introduction

In the course of building a horizontally scalable S3-compatible storage system, a single message from an AI assistant marks a fascinating inflection point. The user reports: "There is traffic visible on Frontend Proxies table, but all of it goes only to kuri-1, none to kuri-2." The assistant's reply begins with a confident assertion: "That's expected based on how the cluster is currently configured." This message, at index 850 in a long coding session, is a masterclass in how even experienced developers can be led astray by plausible assumptions. What follows is a deep debugging journey that ultimately reveals the real problem was not in the routing logic at all, but in the monitoring system's inability to aggregate metrics across distributed nodes.

The Message Itself

The target message is brief but consequential:

[assistant] That's expected based on how the cluster is currently configured. Let me check the S3 proxy routing logic: [grep] round.?robin|backend|routing" Found 100 matches /home/theuser/gw/server/s3frontend/server.go: Line 14: // FrontendServer is a stateless S3 proxy that routes requests to Kuri backend nodes Line 17: backendPool BackendPool Line 25: func NewFrontendServer(auth s3.Authenticator, backendPool BackendPool, router ObjectRouter, multipartTracker MultipartTracker, nodeID string) FrontendServer { Line 28: backendPool: backendPool, Line 69: // Check if we have any healthy backends Line 70: if s.backendPo...

The assistant begins by asserting that the observed behavior—all traffic hitting only kuri-1—is "expected." This is a strong claim. It suggests the assistant believes the architecture is intentionally configured to send all traffic to a single node, or that the round-robin algorithm is not yet implemented. But the very next action—grepping for "round.robin|backend|routing"—betrays uncertainty. If the behavior were truly expected, why investigate?

The Reasoning and Motivation

To understand why this message was written, we must reconstruct the assistant's mental model at this moment. The session up to this point had been a marathon of building and debugging a three-layer distributed S3 architecture. The architecture consists of:

  1. S3 Frontend Proxy (stateless, port 8078) — routes S3 API requests to backend Kuri storage nodes
  2. Kuri Storage Nodes (kuri-1 on port 7001, kuri-2 on port 7002) — independent storage nodes with per-node database keyspaces
  3. Shared YugabyteDB — stores S3 object routing metadata The assistant had just completed deploying the test cluster, verifying that all services were running, and generating test traffic. The user then observed that the monitoring UI showed all traffic going to kuri-1 and none to kuri-2. The assistant's first instinct was to assume this was a configuration issue—perhaps the round-robin wasn't wired up yet, or the backend pool was hardcoded to a single node. The phrase "That's expected" is a hedge. It could mean: - "I designed it that way intentionally" (unlikely given the architecture goals) - "That's a known limitation of the current configuration" - "That's what I would expect given how things are set up right now" Given the assistant's subsequent actions, the most charitable reading is that the assistant genuinely believed the routing might not yet be distributing traffic, and was about to discover why. But the assertion of "expected" also reveals a cognitive bias: when a system behaves unexpectedly, developers often assume the behavior is intentional rather than a bug.

The Assumptions Made

This message rests on several implicit assumptions:

Assumption 1: The user's observation is accurate and reflects the actual routing behavior. The assistant accepts at face value that traffic is only hitting kuri-1. This turns out to be partially correct—the UI shows only kuri-1 traffic—but the actual routing was working fine. The problem was in the monitoring, not the routing.

Assumption 2: The round-robin logic is the likely culprit. By grepping for routing-related code, the assistant assumes the issue lies in the proxy's backend selection algorithm. This is a reasonable starting point, but it leads down a rabbit hole of adding debug logging, rebuilding Docker images, and restarting containers—all before discovering the real problem.

Assumption 3: The S3 proxy's logging is sufficient to diagnose the issue. The assistant adds info-level logging to track round-robin selections, rebuilds the Docker image, and restarts the proxy. This takes several iterations because the logging framework requires explicit logger configuration in the environment variables.

Assumption 4: Both backends are equally capable of serving requests. The assistant checks that kuri-2 has a /healthz endpoint and that both backends report healthy. This assumption is correct—both nodes are operational.

The Mistakes and Incorrect Assumptions

The most significant mistake is the initial assertion that the behavior is "expected." This framing sets up a debugging session that chases the wrong problem for several iterations. The assistant spends time:

  1. Reading the backend pool code to verify round-robin logic
  2. Checking container logs for error messages
  3. Adding debug-level logging to the proxy
  4. Rebuilding the Docker image and restarting containers
  5. Generating test traffic to verify round-robin is working All of this effort confirms that round-robin is working—traffic is being distributed evenly between kuri-1 and kuri-2 at the proxy level. But the UI stubbornly shows only kuri-1 traffic. The real mistake was not the routing logic but a fundamental architectural oversight in the monitoring system. The ClusterTopology RPC endpoint, which the UI queries to display node statistics, only reports metrics for the local node. When the UI connects to kuri-1's web interface (port 9010), it queries kuri-1's metrics, which only knows about requests that kuri-1 itself has handled. The metrics from kuri-2 are siloed in kuri-2's process and never aggregated. This is a classic distributed systems debugging trap: the monitoring system shows an incomplete picture, and developers assume the production system is broken rather than the monitoring system. The assistant falls into this trap completely, spending roughly 30 messages (indices 850–880) chasing a phantom routing bug before discovering the aggregation problem.

Input Knowledge Required

To understand this message, one needs:

  1. The architecture of the system: A three-layer design with stateless S3 proxies in front of independent Kuri storage nodes, sharing a YugabyteDB for metadata.
  2. The round-robin routing algorithm: The BackendPool.SelectRoundRobin() method in server/s3frontend/backend_pool.go that distributes PUT requests across available backends.
  3. The monitoring architecture: The ClusterTopology RPC method in rbstor/diag.go that returns node statistics, and the React frontend in integrations/web/ribswebapp/ that displays them.
  4. The logging framework: The project uses go-log with configurable log levels via environment variables like RIBS_LOGLEVEL.
  5. The Docker Compose test infrastructure: How the test cluster is configured with environment variables for backend node discovery (FGW_BACKEND_NODES).

Output Knowledge Created

This message initiates a debugging process that produces several valuable outputs:

  1. Confirmation that round-robin routing works: The proxy correctly alternates between kuri-1 and kuri-2 for PUT requests.
  2. Discovery of the metrics aggregation gap: The ClusterTopology implementation only reports local node stats. Lines 244–248 of rbstor/diag.go only fill in statistics when nodeID == selfNodeID, leaving remote nodes with zero values.
  3. A plan for fixing the monitoring: The assistant begins implementing an HTTP /api/stats endpoint on each node so that ClusterTopology can make RPC calls to remote nodes and aggregate their metrics.
  4. Improved debug logging: The assistant adds info-level logging to the round-robin selection code, making future debugging easier.
  5. A corrected mental model: The assistant learns that in distributed systems, the monitoring layer is not a neutral observer—it is itself a distributed component with its own blind spots.

The Thinking Process Visible in the Reasoning

The assistant's reasoning unfolds in several stages across the subsequent messages:

Stage 1: Acceptance and investigation. "That's expected... Let me check the S3 proxy routing logic." The assistant assumes the routing is the problem and begins reading code.

Stage 2: Verification of backends. The assistant checks that both kuri-1 and kuri-2 are healthy and that the FGW_BACKEND_NODES environment variable includes both nodes. This confirms the configuration is correct.

Stage 3: Adding instrumentation. The assistant adds logging to the proxyRoundRobin function, rebuilds the Docker image, and restarts the proxy. This is a classic debugging technique: when you can't see what's happening, add more visibility.

Stage 4: The reveal. After generating test traffic and checking logs, the assistant sees:

Added backend {"id": "kuri-1", "url": "http://kuri-1:8078"}
Added backend {"id": "kuri-2", "url": "http://kuri-2:8078"}

Both backends are registered. The round-robin is working. The assistant then has an "aha" moment: "The round-robin is working - traffic is being distributed evenly between kuri-1 and kuri-2 (alternating). The issue is that the UI is not reflecting this."

Stage 5: Root cause analysis. The assistant reads the ClusterTopology implementation and identifies the exact lines where only local node stats are populated. This is the true root cause.

Conclusion

Message 850 is a study in how distributed systems debugging can mislead even careful engineers. The assistant's confident assertion that the behavior was "expected" was wrong, but it was a productive wrongness—it led to a thorough investigation that ultimately uncovered a deeper architectural issue. The real lesson is not about routing algorithms or round-robin implementation, but about the subtle ways that monitoring systems can create illusions of incorrect behavior. In a distributed system, the view from a single node is always incomplete, and the tools we build to observe the system are themselves part of the system, subject to the same limitations and blind spots as the components they monitor.

The assistant's journey from "That's expected" to "the round-robin is working, the UI is broken" is a microcosm of distributed systems debugging: start with the most obvious hypothesis, test it rigorously, and when the evidence contradicts it, be willing to question your assumptions about where the problem actually lives.