The Art of Debugging Distributed Systems: Tracing a Missing Backend in an S3 Proxy Cluster

Introduction

In the course of building a horizontally scalable S3-compatible storage system, a seemingly simple observation by the user — "There is traffic visible on Frontend Proxies table, but all of it goes only to kuri-1, none to kuri-2" — triggered a focused debugging session that reveals much about the challenges of distributed systems development. The subject message, a brief three-line exchange from the assistant, captures a pivotal moment in this debugging process: the moment when a developer pivots from examining log levels to verifying the fundamental configuration of the system. This article unpacks that single message, exploring the reasoning, context, assumptions, and knowledge required to understand it, and what it reveals about debugging complex multi-service architectures.

The Message in Full

The target message, sent by the assistant at index 858 in the conversation, reads as follows:

The log level is info, not debug. Let me add some info-level logging to track which backend is selected. But first, let me check if both backends are actually in the pool by adding debug output: [bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec s3-proxy env | grep BACKEND FGW_BACKEND_NODES=kuri-1:http://kuri-1:8078,kuri-2:http://kuri-2:8078

At first glance, this appears to be a routine debugging step — checking an environment variable inside a running container. But the message is dense with meaning. It represents a decision point, a hypothesis being tested, and a shift in debugging strategy. To fully appreciate it, we must understand the architecture being debugged, the sequence of events that led to this moment, and the reasoning that produced this particular command.

The Architecture Under Test

The system under development is a horizontally scalable S3-compatible storage architecture built for the Filecoin Gateway. The architecture follows a three-layer design:

1. S3 Frontend Proxy (stateless, port 8078) — a lightweight proxy that accepts S3 API requests and routes them to backend storage nodes
2. Kuri Storage Nodes (independent, ports 7001/7002) — the actual storage engines, each with its own database keyspace
3. Shared YugabyteDB — a distributed SQL database that stores object routing metadata The S3 frontend proxy uses a round-robin algorithm to distribute write requests across the available Kuri backends. For reads, it performs a YCQL lookup to determine which node holds the requested object. This design allows the frontend to remain stateless and horizontally scalable, while the storage nodes operate independently. The test cluster, orchestrated via Docker Compose, consists of two Kuri nodes (kuri-1 and kuri-2), a shared YugabyteDB instance, the S3 frontend proxy, and an nginx-based web UI. The environment variable FGW_BACKEND_NODES is the mechanism by which the frontend proxy discovers its available backends — it contains a comma-separated list of node ID-to-URL mappings.

The Debugging Context

The user's observation that all traffic goes to kuri-1 and none to kuri-2 is a classic symptom in distributed systems: the load balancer or proxy appears to be ignoring one of its backends. The possible causes are numerous:

• The backend might not be registered in the pool
• The backend might be marked as unhealthy
• The round-robin algorithm might have a bug
• The health check endpoint might be failing for one node
• The configuration might be incorrect The assistant's response to the user's observation begins in message 850, where they state "That's expected based on how the cluster is currently configured" — a somewhat puzzling statement, since the architecture is explicitly designed to distribute traffic across all backends. This might reflect an assumption that the user was seeing expected behavior from a previous configuration, or it might be a conversational hedge while the assistant begins investigating. Regardless, the assistant immediately begins examining the routing logic by searching for round-robin and backend-related code. By message 852, the assistant has confirmed that the S3 proxy logs show "Backend nodes: 2 configured," suggesting both backends are registered. In messages 853-854, they verify that both kuri-1 and kuri-2 respond to health checks at their S3 endpoints. This eliminates several possible causes: both nodes are healthy and both are configured.## The Pivot: From Log Levels to Configuration This brings us to the subject message. The assistant has just examined the S3 proxy logs and found them unhelpfully sparse — only two lines of output showing the proxy started with 2 backends configured. The logging level is set to info, which means debug-level messages (which would include round-robin selection events) are suppressed. The natural next step would be to either change the log level to debug or add info-level logging to the round-robin selection code. But the assistant does something interesting here. They say: "Let me add some info-level logging to track which backend is selected. But first, let me check if both backends are actually in the pool by adding debug output." That "but first" is the critical reasoning moment. Before modifying code, before rebuilding containers, before diving into the logging infrastructure — the assistant takes the simplest possible verification step: check the environment variable directly inside the running container. This is a textbook debugging heuristic: when a system isn't behaving as expected, verify the most basic assumptions first. The assumption being tested is: "Does the proxy even know about both backends?" The command executed is:

cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec s3-proxy env | grep BACKEND

This command does several things at once:

1. It navigates to the test-cluster directory where the Docker Compose configuration lives
2. It sets the FGW_DATA_DIR environment variable (used by Docker Compose for volume mounts and configuration paths)
3. It uses docker compose exec to run a command inside the running s3-proxy container
4. It pipes the container's environment variables through grep to find any that contain "BACKEND" The result: FGW_BACKEND_NODES=kuri-1:http://kuri-1:8078,kuri-2:http://kuri-2:8078 This confirms that both backends are indeed configured in the environment. The proxy should know about both kuri-1 and kuri-2. The configuration is correct.

The Deeper Reasoning

The assistant's message reveals a sophisticated debugging strategy. Rather than immediately jumping to code modification (which would require editing, rebuilding the Docker image, and restarting containers — a multi-minute cycle), the assistant first exhausts all "zero-cost" verification steps. Checking an environment variable inside a running container takes seconds and requires no code changes. This is especially important in a Docker-based development environment where the build-test cycle can be slow.

The message also reveals an important assumption: the assistant assumes that the FGW_BACKEND_NODES environment variable is the authoritative source of backend configuration for the proxy. This is a reasonable assumption given the architecture, but it's worth noting that there could be other configuration sources — a configuration file, a database query, or a runtime registration mechanism — that might override or supplement this environment variable. The assistant implicitly trusts that the environment variable is the sole source of truth for backend discovery.

There's also an implicit assumption about the health check mechanism. The assistant has already verified (in message 853) that both kuri-1 and kuri-2 respond to HTTP health checks at their S3 endpoints. But the proxy's internal health tracking might use a different mechanism — perhaps a gRPC health check, a database heartbeat, or a custom protocol. The assistant assumes that the HTTP /healthz endpoint is the correct health check mechanism, which is consistent with the code examined earlier.

What Knowledge Is Required

To understand this message, a reader needs:

1. Docker Compose familiarity: Understanding that docker compose exec runs a command inside a running container, and that FGW_DATA_DIR is an environment variable passed to Docker Compose for configuration.
2. Distributed systems concepts: Understanding the three-layer architecture (proxy → storage nodes → database) and the round-robin routing pattern.
3. Go logging conventions: Understanding that info level suppresses debug messages, and that the Go go-log library (from IPFS) uses hierarchical logger names.
4. S3 protocol knowledge: Understanding that S3 PUT requests are typically routed to any available backend, while GET requests must be routed to the specific backend holding the object.
5. The project's configuration model: Understanding that FGW_BACKEND_NODES is the environment variable that configures backend discovery for the S3 proxy.

What Knowledge Is Created

This message produces several valuable pieces of knowledge:

1. Confirmed configuration: Both backends are correctly configured in the environment variable. The proxy has the information it needs to route to both kuri-1 and kuri-2.
2. Narrowed hypothesis space: Since the configuration is correct and both backends are healthy, the problem must lie elsewhere — perhaps in the round-robin algorithm itself, the health tracking logic, or a race condition during startup.
3. Debugging strategy validated: The "check configuration first" approach has ruled out one major class of problems quickly and efficiently, without requiring code changes or container rebuilds.
4. Logging gap identified: The existing logging infrastructure doesn't provide visibility into round-robin selections at the info level, which means debugging routing issues requires either changing log levels or adding new log statements.

The Thinking Process

The reasoning visible in this message follows a clear pattern:

1. Observe symptom: All traffic goes to kuri-1, none to kuri-2.
2. Form hypothesis: Perhaps the proxy doesn't know about kuri-2.
3. Test hypothesis: Check the environment variable that configures backends.
4. Interpret result: Both backends are configured. Hypothesis rejected.
5. Plan next step: Since configuration is correct, the next step is to add logging to track round-robin selections in real-time. The "but first" construction is particularly revealing. It shows that the assistant is consciously prioritizing verification steps, choosing the simplest and fastest check before committing to a more expensive operation (code modification + rebuild + restart). This is the hallmark of an experienced debugger who has learned that the most obvious cause is often the simplest one to check, and that checking it first can save significant time.

Mistakes and Incorrect Assumptions

While the message itself doesn't contain explicit mistakes, there are several assumptions worth examining critically:

1. The environment variable is the sole configuration source: The assistant assumes that FGW_BACKEND_NODES is the only way backends are configured. In a more complex system, there might be additional configuration sources (a config file, a database, a service discovery mechanism) that could override or supplement this variable.
2. The proxy reads configuration at startup: The assistant assumes that the proxy reads the environment variable at startup and doesn't update it dynamically. If the proxy supports hot-reload of configuration, the environment variable might have been correct at startup but the proxy might have updated its internal state based on health check results.
3. Health check success equals routing eligibility: The assistant has confirmed that both backends respond to health checks, but this doesn't necessarily mean the proxy considers them healthy. The proxy might use a different health check mechanism (e.g., a custom RPC, a database query, or a gRPC health check) that could be failing for kuri-2.
4. The round-robin counter is initialized correctly: Even with both backends configured and healthy, the round-robin algorithm might have a bug — perhaps it starts with an index of 0 and never increments, or perhaps it has an off-by-one error that skips the second backend.

Conclusion

The subject message, while brief, captures a moment of disciplined debugging in a complex distributed system. The assistant's decision to verify the configuration before modifying code reflects a mature understanding of the debugging process: always check the simplest explanation first. The message also reveals the importance of having good observability into system configuration — the ability to inspect environment variables inside a running container proved invaluable in quickly ruling out a major class of problems.

This debugging session would continue with the assistant adding info-level logging to the round-robin selection code (messages 859-860), rebuilding the Docker image (message 861), and restarting the containers (message 862). But the subject message represents the critical turning point where a potentially time-consuming code modification was deferred in favor of a quick configuration check — a decision that ultimately saved time and narrowed the problem space efficiently.

For developers working on distributed systems, this message serves as a reminder that the most effective debugging often begins not with code changes, but with verifying the fundamental assumptions about how the system is configured and what information it has available to make decisions.