The Configuration Error That Wasn't Fatal: Debugging a Silent Failure in a Distributed S3 Storage Cluster

The Message

In the midst of a high-stakes debugging session for a horizontally scalable S3 storage system built on YugabyteDB and IPFS, the assistant runs a seemingly mundane command:

cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose logs kuri-1 2>&1 | head -50

The output reveals the root cause of a frustrating failure mode:

kuri-1-1  | 2026/01/31 15:19:08 [watchdog] initialized watermark watchdog policy; watermarks: []; thresholds: [405311748096 486374097715 567436447334 689029971763 729561146572 749826733977 770092321382]
kuri-1-1  | Configuration load failed: %w RetrievableRepairThreshold greater than MinimumReplicaCount: 3 > 1
kuri-1-1  | 
kuri-1-1  | generating ED25519 keypair...done
kuri-1-1  | peer identity: 12D3KooWBpdYuFnJYRWUKT1h1t6byCxgn48RDNJvkir2rkaRq6UN
kuri-1-1  | initializing IPFS node at /root/.ipfs...

This single log snippet is the key that unlocks an entire debugging saga. It explains why the S3 frontend proxy has been reporting "No healthy backends," why the Kuri storage nodes aren't listening on port 8078, and why every load test has been failing with write errors. The message itself is a command and its output — but what makes it remarkable is what it reveals about the system's design, the assumptions baked into its error handling, and the detective work required to connect a seemingly innocuous configuration validation warning to a complete service outage.

Why This Message Was Written: The Debugging Context

To understand why the assistant ran this command, we need to reconstruct the debugging trajectory that led to this moment. The session began with a user report of "data corruption" during S3 load testing. After investigation, the assistant discovered that the corruption was a false alarm — timeout errors at the end of test runs were being misclassified as checksum mismatches. The assistant fixed the error classification and implemented a CQLBatcher to optimize write throughput through the YugabyteDB backend.

With these changes in place, the user issued a straightforward request: "Restart with changes, test at 10/100/1000 parallel." What should have been a routine validation turned into an extended debugging session. The assistant rebuilt the Docker image, attempted to restart containers, and ran into a cascade of failures: permission issues with pkill, missing Docker Compose files, and an S3 proxy that kept returning "Service Unavailable - No healthy backends."

Each attempt to run a load test failed. The S3 proxy was running and responding to HTTP requests, but it consistently reported that no backend storage nodes were healthy. The assistant checked container status, verified that both Kuri nodes were listed as backends, and confirmed that the proxy was routing traffic. Yet the health checks were failing, and the Kuri nodes weren't serving on the expected port 8078.

This message represents the moment when the assistant stopped guessing and started reading the logs systematically. After trying docker compose exec to inspect the container directly (only to discover that curl wasn't installed in the Alpine-based container), the assistant fell back to the most fundamental debugging technique: reading the container's stdout logs from the beginning.## What the Log Output Actually Reveals

The log output from docker compose logs kuri-1 contains three distinct sections, each telling a different part of the story:

Line 1 — Watermark Watchdog Initialization: The first line shows that the Kuri node successfully initialized its watermark watchdog policy. This is a storage capacity management system that tracks data usage thresholds. The watermarks array is empty, and the thresholds show a progression from ~405 GB to ~770 GB. This line confirms that the storage subsystem started correctly — the node can track its data usage and enforce capacity limits.

Line 2 — Configuration Validation Error: The critical line is the second one: Configuration load failed: %w RetrievableRepairThreshold greater than MinimumReplicaCount: 3 > 1. This is a configuration validation failure. The system has detected that the RetrievableRepairThreshold parameter (set to 3) exceeds the MinimumReplicaCount (set to 1). This is logically impossible — you can't require 3 replicas to be retrievable before triggering repair when you only store 1 replica in the first place.

The %w in the error message is particularly telling. In Go's fmt package, %w is the format verb used by fmt.Errorf to wrap errors with %w. However, in standard Go formatting, %w is only valid in fmt.Errorf for error wrapping — using it in a regular log message or fmt.Sprintf call would print the literal %w or cause a formatting issue. The fact that %w appears literally in the log output suggests that the error message is being generated by a fmt.Errorf("Configuration load failed: %w ...") call, but the logging system is printing the error's string representation rather than unwrapping it properly. This is a minor bug in the error propagation chain, but it doesn't affect the functional meaning.

Lines 3-5 — Continued Startup Despite Error: After the error, the node continues to start up: it generates an ED25519 keypair, establishes a peer identity (12D3KooWBpdYuFnJYRWUKT1h1t6byCxgn48RDNJvkir2rkaRq6UN), and initializes an IPFS node. This is the most deceptive aspect of the bug. The configuration validation failure is logged as an error, but it doesn't prevent the node from starting. The node boots up, initializes its subsystems, and appears to be running — but the S3 API server never binds to port 8078.

This is a classic "fail partial" bug: the system fails silently in one subsystem (the S3 API) while continuing to operate in others (IPFS networking, storage management). To an operator checking docker compose ps, the container shows as "Up" and healthy. To the S3 proxy performing health checks, the node is reachable but not serving on the expected port. The proxy marks it as unhealthy, and the entire cluster becomes unavailable for S3 operations.

The Input Knowledge Required to Understand This Message

To interpret this log output correctly, several pieces of domain knowledge are required:

Distributed Storage Architecture: The reader must understand that this is a three-layer system: an S3 frontend proxy (stateless, routing layer), Kuri storage nodes (stateful, data storage), and YugabyteDB (shared coordination database). The Kuri nodes are expected to serve an S3-compatible API on port 8078, which the frontend proxy health-checks and routes traffic to.

Configuration Semantics: The RetrievableRepairThreshold and MinimumReplicaCount parameters are part of the data durability and repair subsystem. MinimumReplicaCount defines how many copies of each data object should be stored across the cluster. RetrievableRepairThreshold defines how many replicas must be confirmed retrievable before the system considers the data healthy. If the threshold exceeds the replica count, the repair logic can never be satisfied — it would perpetually try to repair data that can never meet the threshold, creating a livelock scenario.

Go Error Formatting Conventions: The %w verb in the error message requires familiarity with Go's fmt.Errorf error-wrapping mechanism. Understanding that %w is not a standard log format verb helps recognize that this is likely a fmt.Errorf call being printed via %s or %v rather than being properly unwrapped.

Docker Compose and Container Lifecycle: The fact that the container shows as "Up" in docker compose ps while the application is partially functional is a key insight. Docker only tracks whether the main process (PID 1) is running, not whether all subsystems within that process have initialized correctly.

The Thinking Process Visible in This Message

The assistant's debugging approach in this message reveals a methodical, hypothesis-driven process. The command docker compose logs kuri-1 2>&1 | head -50 is not a random check — it's the culmination of a chain of reasoning:

  1. Hypothesis Formation: The S3 proxy reports "No healthy backends." The proxy has registered both Kuri nodes as backends. Therefore, the health checks must be failing. Health checks likely probe the S3 API endpoint on port 8078.
  2. Direct Verification: The assistant tries curl http://localhost:7001/ (the mapped port for kuri-1) and gets a response, but it's not the S3 API — it's the IPFS HTTP API or another endpoint. The assistant then tries docker compose exec kuri-1 curl http://localhost:8078/ but discovers curl is not installed in the Alpine container.
  3. Port Inspection: Running netstat inside the container confirms that kuri-1 is listening on ports 34431, 44091, 2112, and others — but not 8078. The S3 API server simply isn't starting.
  4. Root Cause Analysis: At this point, the assistant needs to understand why the S3 API isn't starting. The most efficient way is to read the container's startup logs from the beginning, which is exactly what this command does. The head -50 limit is also significant. The assistant isn't dumping the entire log — just the first 50 lines, which should contain the startup sequence. This is an experienced operator's technique: when debugging a service that fails during initialization, read the startup logs, not the runtime logs.## Assumptions Made and Their Consequences This debugging episode reveals several assumptions — some validated, some incorrect — that shaped the investigation: Assumption 1: The container status reflects application health. The most dangerous assumption was that docker compose ps showing "Up" status meant the Kuri node was fully operational. Docker's container health model only tracks process existence, not subsystem initialization. The Kuri binary started, initialized its IPFS layer, and kept running — but the S3 API server failed to start due to the configuration validation error. This created a "zombie" state where the node appeared healthy to orchestration tools but was non-functional for its primary purpose. Assumption 2: Configuration errors would be fatal. The assistant initially assumed that if a configuration validation error occurred, the application would either fail to start entirely or would log a clear error message indicating which subsystem was disabled. Instead, the error was logged and the startup continued silently. This is a design issue in the Kuri application itself: configuration validation should either be fatal (preventing startup) or should be clearly communicated through health check endpoints. Assumption 3: The S3 proxy's health check mechanism was reliable. The proxy reported "No healthy backends," which was accurate — but the proxy's health check (presumably an HTTP GET to the backend's S3 endpoint) was failing because the backend wasn't listening on that port. The proxy correctly identified the symptom but couldn't diagnose the cause. This is a reasonable architectural boundary: the proxy shouldn't need to understand internal configuration errors, only whether the backend responds to requests. Assumption 4: The load test failures were related to the batcher changes. When the first load test after restarting showed all write errors, the assistant naturally suspected the newly introduced CQL batcher. This turned out to be a red herring — the batcher was never being exercised because the requests weren't reaching the storage nodes at all. The actual cause was the pre-existing configuration error that had been present since the last configuration regeneration.

The Output Knowledge Created by This Message

This message produced several distinct pieces of knowledge that advanced the debugging process:

Direct Knowledge: The log output confirms that kuri-1 has a configuration validation error: RetrievableRepairThreshold (3) > MinimumReplicaCount (1). This is a concrete, actionable finding.

Inferred Knowledge: The absence of any log message about starting the S3 API server (no "listening on 8078" or similar) confirms that the S3 subsystem is not initializing. The configuration validation error is the most likely cause, either because the validation failure causes the S3 server setup to be skipped, or because the configuration loading returns an error that propagates up the initialization chain.

Diagnostic Knowledge: The error message format (Configuration load failed: %w ...) reveals a minor bug in error handling within the application. The %w verb appearing literally in the log output indicates that the error is being printed with %s or %v formatting rather than being properly unwrapped. While not critical, this is a code quality issue that could be addressed.

Actionable Knowledge: The fix is clear: either reduce RetrievableRepairThreshold to 1 (or less than or equal to MinimumReplicaCount), or increase MinimumReplicaCount to 3 or higher. The configuration is generated by gen-config.sh, so the fix needs to be applied there, and the containers need to be recreated with the updated configuration.

The Resolution Path

The message doesn't show the fix itself, but it sets the stage for it. The assistant will go on to:

  1. Locate the gen-config.sh script that generates the per-node settings files
  2. Find where RetrievableRepairThreshold and MinimumReplicaCount are set
  3. Fix the inconsistency (likely by setting RetrievableRepairThreshold to 1 to match the single-replica configuration)
  4. Regenerate the configuration and recreate the containers
  5. Verify that the S3 API starts listening on port 8078
  6. Re-run the load tests successfully The broader lesson is about the importance of defensive startup validation in distributed systems. A configuration error that silently disables a core subsystem while allowing the process to continue running creates a failure mode that is difficult to diagnose from outside the container. The system should either fail fast (exit with a clear error) or expose its internal health status through a dedicated health endpoint that the orchestrator or proxy can query.

Conclusion

This single message — a Docker Compose logs command and its output — encapsulates the essence of distributed systems debugging. It shows how a seemingly minor configuration inconsistency (3 > 1) can cascade through a complex system: the configuration validation fails silently, the S3 API server doesn't start, the health checks fail, the proxy marks the backend as unhealthy, and every user request returns "Service Unavailable." The assistant's methodical approach — forming hypotheses, verifying directly, inspecting ports, and finally reading startup logs — demonstrates the disciplined thinking required to trace a symptom back to its root cause in a multi-layer distributed architecture.

The message also highlights the importance of understanding your tools' failure modes. Docker's process-level health model, Go's error formatting conventions, and the Kuri application's startup sequence all contributed to a failure that was invisible at the orchestration layer but catastrophic at the application layer. For engineers building and operating distributed systems, this episode serves as a reminder that "the container is running" is never the same as "the service is healthy."