The Art of the Small Fix: Debugging a Docker Container Wait Loop in a Distributed S3 Test Cluster

Introduction

In the course of building a horizontally scalable S3-compatible storage architecture for the Filecoin Gateway, an assistant and user found themselves in a rapid-fire debugging session. Over the span of a few minutes, three distinct operational bugs were discovered and fixed in a test cluster's startup infrastructure. The subject of this article—message 294 in the conversation—is the third and final fix in this chain: a correction to a shell script's container-waiting logic that was causing the startup process to hang indefinitely. While the message itself is brief, it sits at the terminus of a revealing debugging trajectory that illuminates how assumptions about infrastructure tooling can silently compound, and how even a one-line fix requires a sophisticated understanding of container orchestration internals.

The Subject Message

Here is the message in its entirety:

Fixed! The script now properly detects when db-init has already exited.

>

Changes: - First checks if db-init has already exited (docker-compose up waits for dependencies) - If already exited, checks the exit code immediately - Only enters the wait loop if db-init is still running - More flexible pattern matching for "Exited" status

>

To test: ``bash ./stop.sh /data/fgw2 ./start.sh /data/fgw2 ``

>

The script should now immediately recognize that db-init completed and proceed to check the Kuri nodes.

The assistant's internal reasoning preceding this message reveals the thought process: "Good, the syntax is valid. The fix handles both cases: 1. If db-init already exited (as shown in the user's output), it checks immediately 2. If db-init is still running, it waits for it to exit. This should resolve the issue where the script was waiting unnecessarily when db-init had already completed."

The Debugging Chain: Three Bugs in Five Minutes

To understand why this message was written, one must understand the three-bug cascade that preceded it. The test cluster infrastructure—a Docker Compose configuration with two Kuri storage nodes, a shared YugabyteDB instance, and a database initialization service—had been built in a previous session. When the user began testing it, problems emerged in rapid succession.

Bug 1: Permission Errors. The init-data.sh script used chmod on data directories that YugabyteDB had created as root inside Docker. When the user ran the script as a regular user, chmod failed because root-owned files are not modifiable by non-root users. The assistant fixed this by suppressing chmod errors with 2>/dev/null and adding a warning message. This was a pragmatic fix: the permissions didn't actually matter for Docker's operation since containers run as root internally, but the error was breaking the script's flow due to the set -e flag.

Bug 2: Database Already Exists. After fixing the permission issue, the user ran the cluster again. This time, the db-init service failed because it tried to run CREATE DATABASE filecoingw on a PostgreSQL instance where the database already existed from a previous run. The assistant fixed this by appending 2>/dev/null || echo 'Database may already exist' to the SQL command, making the initialization idempotent.

Bug 3: The Waiting Game. With the first two bugs fixed, the user ran the cluster a third time. The db-init container now exited successfully (shown as "Exited" in the docker-compose output), but the start.sh script's wait loop didn't recognize this. The script entered a 30-iteration loop, printing "Waiting for db-init... (1/30)", "(2/30)", and so on, even though the container had already finished. The user's terminal output shows it reached iteration 8 before the user interrupted to report the issue.

This is where message 294 enters the picture. The assistant had just edited the script and validated its syntax. Message 294 is the confirmation and explanation of that fix.

The Root Cause: A Subtle Assumption About Docker Output

The wait loop's original logic was straightforward: it ran docker-compose ps db-init and grepped for "Exited (0)" to detect successful completion, or "Exited ([1-9])" to detect failure. This seems reasonable—after all, docker-compose ps does show exit codes for stopped containers. But there were two problems.

First, the docker-compose ps command, by default, only shows running containers. A container that has already exited is not shown unless you use the -a flag (for "all"). This was actually a separate bug that the assistant had fixed in a previous iteration of the script, but the fix had introduced a new issue: even with -a, the output format for an exited container doesn't always include the parenthesized exit code in the way the grep pattern expected. Depending on the Docker version and the container's state, the output might show just "Exited" without "(0)", or it might show it on a different column.

Second, and more fundamentally, the script's architecture assumed a linear flow: start containers, then wait for db-init. But docker-compose up itself handles dependency ordering. Because the db-init service depended on yugabyte being healthy, Docker Compose waited for YugabyteDB before starting db-init. And because db-init was a short-lived container that ran a SQL command and exited, by the time docker-compose up finished and returned control to the script, db-init had already completed. The script then entered a wait loop for a container that was already done—a classic race condition where the monitoring code arrives after the event it's monitoring.

The Fix: Restructuring the Wait Logic

The assistant's fix restructured the wait logic to handle this asynchronous reality. Instead of assuming the container was still running and entering a polling loop, the script now first checks whether db-init has already exited. If it has, it immediately inspects the exit code. Only if the container is still running does it enter the wait loop.

This is a small but significant architectural insight: in distributed systems and container orchestration, you cannot assume the order of events. A service that was supposed to run "during startup" may have already finished by the time your monitoring code runs. The fix also used more flexible pattern matching for the "Exited" status, accommodating variations in Docker's output format across versions.

Assumptions, Mistakes, and Lessons

The debugging chain reveals several assumptions that turned out to be incorrect:

Assumption 1: Docker Compose startup is synchronous. The original script treated docker-compose up -d as a blocking operation that completed only after all services were running. In reality, docker-compose up returns as soon as the containers are created and started, not when they finish their work. For long-running services this is fine, but for initialization containers that run and exit, it creates a timing mismatch.

Assumption 2: docker-compose ps output is consistent. The grep pattern assumed a specific textual format that included exit codes in parentheses. Docker's output formatting can vary, and the pattern was too brittle.

Assumption 3: The wait loop would always find a running container. The loop was designed to poll until the container exited, but it didn't account for the case where the container had already exited before the loop began.

Assumption 4: Error suppression is sufficient for idempotency. The fix for Bug 2 (database already exists) used error suppression rather than a proper conditional check. While this worked, it was a pragmatic shortcut rather than a robust solution. A more thorough approach would have been to check for database existence before creating it.

Input and Output Knowledge

To understand this message, a reader needs input knowledge of: Docker Compose's service lifecycle and dependency resolution; the behavior of docker-compose ps and its output format; shell scripting patterns for container health checks; the concept of idempotent initialization in containerized environments; and the specific architecture of the test cluster (db-init as a short-lived setup container).

The message creates output knowledge about: a corrected startup script that properly handles asynchronous container completion; a reusable pattern for waiting on short-lived containers in Docker Compose; the specific fix applied to the start.sh script; and a debugging methodology that traces through a chain of cascading failures.

The Thinking Process

The assistant's reasoning, visible in the internal notes, shows a methodical debugging approach. Each bug was identified from user-provided error output, traced to its root cause, fixed with minimal changes, and validated. The reasoning for message 294 specifically shows the assistant synthesizing the two cases that the fix must handle (already exited vs. still running) and confirming that the syntax check passed. The thinking is concise but covers the essential logic: "If db-init already exited... it checks immediately. If db-init is still running, it waits for it to exit."

Conclusion

Message 294 is a small message about a small fix, but it encapsulates a profound lesson about infrastructure software: the gap between what we assume containers do and what they actually do is where bugs breed. The three-bug cascade—permissions, idempotency, timing—is a microcosm of the challenges in distributed systems development. Each fix was individually simple, but diagnosing them required understanding the full lifecycle of Docker containers, the behavior of orchestration tools, and the subtle ways that assumptions compound. In the end, the test cluster started correctly, and the work of building the distributed S3 architecture could continue.