The Art of Debugging Distributed Systems: A Single Health Check That Changed Everything

Introduction

In the course of building a horizontally scalable S3-compatible storage system, a seemingly trivial debugging message reveals the intricate thought process behind diagnosing distributed system failures. The message in question is deceptively brief — just two shell commands checking whether two backend nodes respond to a health check endpoint. Yet this simple diagnostic step sits at a critical juncture in a debugging session, where the developer must reconcile what the code should do with what the system actually does.

The message, reproduced exactly as it appeared in the conversation, is:

[assistant] Let me check if kuri-2 has a /healthz endpoint:
[bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec s3-proxy wget -q -O- http://kuri-1:8078/healthz 2>&1
OK
[bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec s3-proxy wget -q -O- http://kuri-2:8078/healthz 2>&1
OK

To understand why this message matters, we must reconstruct the full context of the debugging session and the architecture it operates within.

The Architecture Under Test

The system being built is a three-layer horizontally scalable S3 storage architecture. At the top sits a stateless S3 frontend proxy (listening on port 8078) that accepts client requests and routes them to backend Kuri storage nodes. Below the proxy are two Kuri nodes — independent storage servers that each own their own data and keyspace in a shared YugabyteDB cluster. The S3 proxy is designed to distribute writes across these backends using a round-robin strategy, while reads are routed based on object metadata stored in a shared S3 objects table.

This architecture means that the S3 proxy is the single point of entry for all client traffic. If the proxy fails to distribute traffic evenly, the entire system's horizontal scalability claim is undermined. The health of the backend pool — the set of Kuri nodes that the proxy can route requests to — is therefore mission-critical.

The User's Observation

The debugging session leading up to this message began with a user report (message 849): "There is traffic visible on Frontend Proxies table, but all of it goes only to kuri-1, none to kuri-2." This is a serious finding. The round-robin routing logic, which the assistant had implemented and verified in code, was supposed to distribute requests evenly across both Kuri nodes. Yet in practice, all traffic was landing on kuri-1 alone.

The assistant's first response (message 850) was revealing: "That's expected based on how the cluster is currently configured." This was an incorrect assumption — a moment where the developer's mental model of the system diverged from reality. The assistant then immediately pivoted to investigation, reading the backend pool source code and checking the S3 proxy logs. This self-correction is a hallmark of effective debugging: when a hypothesis fails, abandon it and gather more data.

The Critical Debugging Step

Message 853 represents the next logical step in the investigation. The assistant had read the backend_pool.go source code and confirmed that the round-robin logic appeared correct. The S3 proxy logs showed "Backend nodes: 2 configured" but revealed nothing about individual request routing. At this point, the assistant needed to form a new hypothesis about why traffic wasn't reaching kuri-2.

The most parsimonious explanation was that the backend pool's health checking mechanism was marking kuri-2 as unhealthy. In distributed systems, a common pattern is for a proxy to periodically check backend health by issuing HTTP requests to a designated endpoint (typically /healthz). If a backend fails to respond or returns a non-OK status, the proxy removes it from the pool and stops routing traffic to it. This would perfectly explain the observed behavior: kuri-2 might be alive but failing its health checks, causing the proxy to silently route all traffic to the only healthy backend, kuri-1.

The assistant's choice to test this hypothesis directly — by running wget commands from inside the s3-proxy container — is methodologically sound. By executing the health check from the same network context as the proxy itself, the assistant eliminates network-level discrepancies. The proxy might resolve hostnames differently, or firewall rules might apply differently from within the Docker network. Running the check inside the container reproduces the exact conditions under which the proxy performs its own health checks.

The Results and Their Implications

Both commands returned "OK". This is a classic debugging outcome: the simplest hypothesis was ruled out, narrowing the search space. Kuri-2 was alive, reachable, and responding to health checks. The backend pool should have considered it healthy and included it in round-robin rotation.

This negative result is valuable precisely because it eliminates an entire class of possible causes. The issue is not:

Kuri-2 being down or unresponsive
Network connectivity between the proxy and kuri-2
The /healthz endpoint being missing or misconfigured on kuri-2
A fundamental health check failure The debugging session must now move to deeper causes: perhaps the round-robin counter is stuck, perhaps the backend pool is not actually using the health check results in its routing decisions, or perhaps there is a race condition in how backends are registered. The assistant's subsequent actions (visible in messages 854–856) confirm this trajectory — checking kuri-2's configuration, adding debug logging, and examining the server code more carefully.

Assumptions and Potential Pitfalls

This message reveals several assumptions that the assistant is operating under. First, the assistant assumes that both Kuri nodes expose a /healthz endpoint on port 8078 — the same port used by the S3 proxy itself. This is a reasonable assumption given the architecture, but it's worth noting that the health check endpoint might live on a different port or path. The fact that both returned "OK" confirms the assumption was correct for this system.

Second, the assistant assumes that the health check endpoint is the mechanism by which the backend pool determines backend health. This is a standard pattern in Go HTTP proxies, but the actual health checking logic in backend_pool.go might use a different mechanism — perhaps TCP-level checks, or a custom RPC endpoint. The assistant is implicitly betting that /healthz is the right thing to check.

Third, the assistant assumes that running wget from within the s3-proxy container accurately simulates the proxy's own health check behavior. While this is a good approximation, subtle differences could exist — for example, the proxy might use a different HTTP client with different timeout settings, or it might check a different endpoint path.

The Thinking Process Visible in the Message

The message's opening line — "Let me check if kuri-2 has a /healthz endpoint" — reveals the assistant's mental model. The phrasing "has a /healthz endpoint" is interesting because it suggests the assistant is not just checking whether the endpoint responds, but whether it exists at all. This implies a suspicion that perhaps kuri-2 was configured differently from kuri-1, or that the health check endpoint wasn't wired up correctly for the second node.

The two commands are ordered deliberately: check kuri-1 first as a control, then check kuri-2 as the test subject. If kuri-1 returned OK and kuri-2 returned something else, the diagnosis would be clear. By running both, the assistant establishes a baseline for what a healthy response looks like.

The use of wget -q -O- (quiet mode, output to stdout) is a pragmatic choice for a container environment where curl might not be installed. The 2>&1 redirect captures any error output, ensuring that connection failures or DNS resolution errors would be visible alongside the response body.

Input Knowledge Required

To fully understand this message, one needs to know:

The three-layer architecture: S3 proxy → Kuri nodes → YugabyteDB
That the S3 proxy uses a backend pool with round-robin routing
That health checks are a standard mechanism for managing backend pools
That /healthz is a conventional health check endpoint in Go web services
Docker Compose networking: containers can reference each other by service name
That the s3-proxy container has network access to both kuri-1 and kuri-2

Output Knowledge Created

This message produces several pieces of actionable knowledge:

Both Kuri nodes expose a functioning /healthz endpoint
Both nodes are reachable from the s3-proxy container's network context
The health check mechanism is not the cause of the traffic imbalance
The debugging effort must shift to other potential causes (routing logic, state management, configuration)

Conclusion

A message consisting of just two shell commands and their output might seem trivial at first glance. But in the context of debugging a distributed system, it represents a carefully considered diagnostic step that eliminates an entire class of potential causes. The assistant's decision to test health checks from within the proxy container, the ordering of control and test cases, and the interpretation of the "OK" responses all reflect a systematic approach to debugging that is far more sophisticated than the message's brevity suggests.

This single debugging step demonstrates a fundamental truth about engineering complex systems: the most valuable insights often come not from the answers you find, but from the questions you know how to ask — and the wrong hypotheses you know how to eliminate.