The Moment of Diagnostic Clarity: When the Data Path Works but the Observability Path Doesn't

Introduction

In distributed systems debugging, the hardest bugs are often not the ones where something is broken, but the ones where two separate subsystems—the data path and the observability path—tell contradictory stories. Message 874 in this coding session captures precisely such a moment: a flash of diagnostic insight where an engineer realizes that the round-robin load balancing is working correctly, but the monitoring dashboard is lying because it only sees local metrics. This article examines that message in depth, exploring the reasoning, assumptions, and knowledge boundaries that shaped this critical realization.

The Subject Message

The message opens with a confident declaration:

The round-robin is working - traffic is being distributed evenly between kuri-1 and kuri-2 (alternating). The issue is that the UI is not reflecting this - it's only showing traffic on kuri-1.

>

The problem is that the ClusterTopology RPC is being called on kuri-1, and kuri-1 only knows about its own metrics. The metrics on kuri-2 aren't being aggregated.

>

Let me check how ClusterTopology is implemented

The assistant then proceeds to read the diag.go file to inspect the ClusterTopology implementation.

Context and Motivation: Why This Message Was Written

To understand why this message matters, we need to reconstruct the debugging journey that led to it. The conversation leading up to message 874 tells a story of systematic investigation with a surprising twist.

The user had reported a puzzling observation (message 849): "There is traffic visible on Frontend Proxies table, but all of it goes only to kuri-1, none to kuri-2." This was a legitimate concern—if the whole point of building a horizontally scalable S3 architecture is to distribute load across multiple storage nodes, seeing all traffic land on a single node would indicate a fundamental routing failure.

The assistant's first instinct was entirely reasonable: suspect the round-robin logic in the S3 frontend proxy. The assistant added debug-level logging to track which backend was selected for each request (message 860), rebuilt the Docker image, restarted the cluster, and generated test traffic (messages 861-864). But the logs showed nothing—the debug messages weren't appearing because the log level was set to info, not debug (messages 865-868).

After adjusting the log level configuration in docker-compose.yml and redeploying (messages 870-872), the assistant finally saw the logs that would change everything:

s3-proxy-1  | Added backend {"id": "kuri-1", "url": "http://kuri-1:8078"}
s3-proxy-1  | Added backend {"id": "kuri-2", "url": "http://kuri-2:8078"}

Both backends were registered. The round-robin was working. The data path was fine.

This is the moment message 874 was born—the moment of diagnostic pivot. The assistant realized that the problem wasn't in the request routing at all, but in the monitoring subsystem. The UI was displaying incomplete information because the ClusterTopology RPC endpoint, when called on kuri-1, only returned metrics from kuri-1's local perspective.

The Reasoning Process: How the Insight Unfolded

The thinking visible in message 874 is elegant in its simplicity. The assistant makes a logical leap that separates two concerns that had been conflated:

  1. The data path: Does the S3 proxy correctly distribute PUT requests across backends?
  2. The observability path: Does the monitoring UI correctly aggregate and display metrics from all nodes? By confirming that both backends were registered in the pool and that traffic was being distributed (the logs showed alternating selections), the assistant could definitively rule out a data-path bug. The remaining possibility was an observability-path bug—and the most likely culprit was immediately identifiable: the ClusterTopology RPC was being served by kuri-1, and kuri-1 only had access to its own local metrics. This is a classic distributed systems debugging pattern. When a system has multiple independent nodes, each with its own metrics, and a monitoring endpoint that only reports local state, the dashboard will always look like only one node is doing work—even when the actual workload is perfectly balanced. The monitoring becomes a self-fulfilling illusion of imbalance.

Assumptions Made by the Assistant

Several assumptions underpin the reasoning in message 874:

Assumption 1: The round-robin is actually working. The assistant had seen logs confirming both backends were added to the pool, but hadn't yet verified that individual requests were being routed to different nodes. The assertion that traffic was "alternating" was based on the presence of both backends in the pool and the round-robin algorithm's deterministic behavior, not on direct per-request observation. This was a reasonable inference but still an assumption.

Assumption 2: The ClusterTopology RPC is served by kuri-1. The assistant assumed that the Web UI at :9010 (which serves the ClusterTopology RPC) is connected to kuri-1's backend, not kuri-2's. This was correct—the docker-compose configuration had the web UI pointing to kuri-1's RPC endpoint.

Assumption 3: The metrics are not being aggregated across nodes. This was the core insight, and it was correct. The ClusterTopology implementation in diag.go only filled in stats for the local node (when nodeID == selfNodeID), leaving remote nodes with zero values.

Assumption 4: The fix requires fetching stats from remote nodes via RPC. This assumption shaped the subsequent implementation work (messages 875-885), where the assistant began building a /api/stats endpoint and modifying ClusterTopology to make HTTP calls to each remote node. This was a reasonable architectural choice, though it added complexity to what could have been a simpler fix (e.g., having the S3 proxy report metrics to a shared database).

Mistakes and Incorrect Assumptions

While the core insight in message 874 was correct, there were subtle issues worth examining:

The assumption that the UI was "only showing traffic on kuri-1" was itself a misinterpretation of what the UI was showing. The Frontend Proxies table was showing traffic on the proxy (proxy-1), not on the storage nodes. The Storage Nodes table was showing both kuri-1 and kuri-2, but with zero traffic on kuri-2. The user's original complaint was about the storage node metrics, not the proxy metrics. The assistant correctly identified the root cause—metrics isolation—but the initial framing conflated the proxy-level view with the storage-node-level view.

The assistant assumed that adding debug logging and rebuilding would be quick. In practice, the debugging cycle took multiple iterations (messages 860-873) because of the log level configuration issue. This is a common pitfall: adding logging is only useful if the log output is actually captured and displayed. The assistant had to update the docker-compose environment variables to include GOLOG_LOG_LEVEL for the frontend logger before the debug output appeared.

The assumption that the fix would be straightforward. The subsequent messages (875-885) reveal that implementing cross-node metrics aggregation was more complex than initially expected. The RIBS interface didn't have a NodeID method, requiring the assistant to read the environment variable directly. The LSP errors in message 880 and 883 show the iterative fix cycle.

Input Knowledge Required to Understand This Message

To fully grasp message 874, a reader needs:

Knowledge of the system architecture: The horizontally scalable S3 system has three layers: stateless S3 frontend proxies (which handle routing), Kuri storage nodes (which store data), and a shared YugabyteDB (which stores routing metadata). Each Kuri node has its own metrics, and the monitoring UI connects to one node's RPC endpoint.

Knowledge of the round-robin implementation: The BackendPool.SelectRoundRobin() method in server/s3frontend/backend_pool.go uses an atomic counter to distribute requests across healthy backends. The assistant had verified that both kuri-1 and kuri-2 were in the pool.

Knowledge of the ClusterTopology implementation: The ClusterTopology() method in rbstor/diag.go reads the FGW_BACKEND_NODES environment variable to discover cluster nodes, then fills in metrics for each node. Crucially, it only populates metrics for the local node—remote nodes get zero values.

Knowledge of Docker Compose and container networking: The test cluster runs multiple containers (yugabyte, kuri-1, kuri-2, s3-proxy, webui) with internal DNS resolution via Docker Compose service names. The FGW_BACKEND_NODES environment variable uses these service names (e.g., kuri-1:http://kuri-1:8078).

Output Knowledge Created by This Message

Message 874 creates several valuable pieces of knowledge:

A clear diagnostic separation between data-path and observability-path bugs. This is a reusable debugging pattern for distributed systems. When the UI shows uneven load distribution, the first question should be: "Is the monitoring endpoint aggregating metrics from all nodes, or only reporting local state?"

A specific bug report: The ClusterTopology RPC endpoint does not aggregate metrics from remote storage nodes. This is a concrete, actionable finding that directly informs the next development work.

A design insight for the monitoring architecture: The current design assumes that the node serving the Web UI (kuri-1) has complete knowledge of all cluster metrics. In reality, each node only knows its own metrics. The monitoring system needs either (a) a pull model where the UI fetches metrics from each node individually, or (b) a push model where nodes report metrics to a shared store.

A debugging methodology: The sequence of steps—add logging, rebuild, restart, generate traffic, check logs, form hypothesis, verify hypothesis—is a model of systematic debugging. The key lesson is that when logs don't appear, check the log level configuration before assuming the code isn't running.

The Thinking Process Visible in the Reasoning

The reasoning in message 874 shows a clear diagnostic arc:

  1. Observation: The UI shows traffic only on kuri-1.
  2. Initial hypothesis: The round-robin routing is broken.
  3. Evidence gathering: Add logging, rebuild, restart, generate traffic.
  4. Evidence analysis: Logs show both backends are registered and healthy.
  5. Hypothesis revision: The round-robin is working; the UI is wrong.
  6. New hypothesis: The ClusterTopology RPC only reports local metrics.
  7. Verification plan: Read the ClusterTopology implementation to confirm. This is textbook diagnostic reasoning—the assistant doesn't jump to conclusions but systematically eliminates possibilities. The critical skill on display is the ability to distinguish between "the system isn't working" and "the system is working but the measurement is wrong." In distributed systems, this distinction is often the hardest to make because the monitoring tools are themselves part of the distributed system and subject to the same consistency and visibility limitations.

Conclusion

Message 874 is a small but pivotal moment in a larger debugging session. It represents the point where the assistant correctly identified that the problem was not in the data path (which was functioning correctly) but in the observability path (which was incomplete). This insight saved what could have been hours of fruitless debugging of the round-robin logic and redirected effort toward the real issue: cross-node metrics aggregation.

The message also illustrates a deeper truth about distributed systems engineering: monitoring and observability are not neutral windows into system behavior. They are themselves distributed systems with their own failure modes, consistency models, and blind spots. A dashboard that shows all traffic going to one node might be telling the truth about the dashboard's data source, even while the actual system is perfectly balanced. The skill of knowing when to trust your instruments and when to question them is what separates productive debugging from wild goose chases.