The Invisible Container: A Lesson in Docker Compose Diagnostics

Introduction

In the course of building a horizontally scalable S3-compatible storage architecture for the Filecoin Gateway, a developer encountered a frustratingly subtle bug: a shell script that refused to stop waiting for a container that had already finished. The fix, when it finally came, was a single flag change—replacing docker-compose ps with docker-compose ps -a—but the journey to that discovery reveals deep truths about how Docker Compose manages container lifecycle, how assumptions about tool behavior can silently derail automation, and how even experienced developers can be tripped up by the gap between what a command shows and what a command can show.

The message in question, message 300 in the conversation, is the assistant's announcement of this fix. It is deceptively brief: a few lines explaining that exited containers don't appear in docker-compose ps by default, and that the -a flag is needed to see all containers including stopped ones. But this message sits at the end of a multi-step debugging chain that had already consumed several iterations of attempted fixes, each one failing because the root cause had not yet been properly identified. Understanding why this particular fix worked—and why the previous attempts did not—requires examining the full context of the test cluster infrastructure, the assumptions baked into the startup script, and the specific behavior of Docker Compose's process management.

Context: Building a Test Cluster for a Distributed S3 Architecture

The broader project was ambitious: a horizontally scalable S3-compatible storage system built on the Filecoin network, using Kuri storage nodes as backend data servers and stateless S3 frontend proxies for request routing. The test cluster, defined in a Docker Compose file, was designed to simulate this architecture with two Kuri nodes, a shared YugabyteDB instance for metadata coordination, and a database initialization service (db-init) that created the required PostgreSQL database and CQL keyspace before the Kuri nodes attempted to connect.

The startup sequence was critical. YugabyteDB had to be healthy first. Then db-init had to run to create the database schema. Only then could the Kuri nodes start and connect to the database. The start.sh script orchestrated this by launching all services with docker-compose up -d, then entering a polling loop that checked the status of each service in order.

The script had already been through several rounds of debugging. First, the init-data.sh script had to be fixed to suppress permission errors when chmod failed on root-owned files created by Docker. Then the db-init container itself had to be fixed to handle the case where the database already existed from a previous run. Each fix addressed a visible failure, but the underlying orchestration logic remained fragile.

The Bug: A Loop That Never Terminates

The symptom was clear from the user's terminal output. After docker-compose up completed, the output showed:

 ✔ Container test-cluster-db-init-1  Exited

The db-init container had exited. But the startup script then entered a wait loop:

🔍 Waiting for database initialization...
  Waiting for db-init... (1/30)
  Waiting for db-init... (2/30)
  Waiting for db-init... (3/30)
  ...
  Waiting for db-init... (20/30)

The script was checking docker-compose ps db-init for an "Exited (0)" status, but the container was never found. The loop continued until it timed out after 30 iterations, and the script never detected that initialization had completed successfully.

The user's frustration was palpable. They had already reported this issue once before, and the assistant had attempted a fix in message 292—adding logic to check if the container had already exited before entering the loop. But that fix had failed too. The user ran the script again and got the same result: the loop kept spinning.

The Diagnostic Breakthrough

What changed between the failed fix and the successful one was a direct observation. In message 296, the assistant ran a command that the user had suggested:

docker-compose ps

The output was revealing:

NAME                      IMAGE                                COMMAND                  SERVICE    CREATED              STATUS                        PORTS
test-cluster-yugabyte-1   yugabytedb/yugabyte:2024.2.5.0-b59   "/sbin/tini -- bin/y…"   yugabyte   About a minute ago   Up About a minute (healthy)   5433/tcp, ...

The db-init container was not listed. It had exited, and docker-compose ps—by default—only shows running containers. The exited container was invisible to the very command the script was using to check its status.

This is the critical insight that the previous attempts had missed. The assistant's first fix (message 292) had added a pre-check: "if docker-compose ps db-init | grep -q 'Exited'; then..." But this pre-check was checking docker-compose ps (without -a), which would never show an exited container. The pre-check itself was doomed to fail. The script would always fall through to the wait loop, which also used docker-compose ps (without -a), and the loop would never find the container either.

The assistant's second fix (message 298) changed the pre-check to use docker-compose ps -a, but the wait loop still used docker-compose ps without the flag. The pre-check might have worked if the container had already exited before the loop started, but in practice, the timing meant the container was still running when the pre-check ran, and by the time the loop started, the container had exited and become invisible again.

The Fix: One Character, Profound Difference

Message 300 announces the corrected fix. The assistant changed every occurrence of docker-compose ps to docker-compose ps -a throughout the script. The -a flag (short for --all) tells Docker Compose to show all containers, including those that have exited. With this change, the script could now detect the db-init container regardless of its state—running, exited with success, or exited with failure.

The message itself is concise, almost to a fault. It reads:

Fixed! Changed docker-compose ps to docker-compose ps -a to see exited containers.

The issue: Exited containers don't show in docker-compose ps by default - you need the -a flag to see all containers including stopped ones.

There is no lengthy explanation, no post-mortem analysis of why the previous fix failed. The assistant simply states the correction and moves on. But the brevity belies the depth of the debugging journey that led to this point.

Why This Bug Was So Deceptive

Several factors conspired to make this bug particularly hard to diagnose.

First, the Docker Compose documentation is ambiguous. The docker-compose ps command's default behavior—showing only running containers—is documented, but it's easy to miss when you're focused on the higher-level orchestration logic. Many developers naturally assume that ps shows all containers, analogous to docker ps -a which shows all containers by default (actually, docker ps also shows only running containers by default, but the parallel is instructive).

Second, the bug was timing-dependent. The db-init container was designed to exit quickly—it ran a SQL command and finished. In some runs, the script's pre-check might catch it while it was still running, and the loop would then wait for it to exit. But the loop itself used docker-compose ps without -a, so once the container exited, it disappeared from view. The loop would never see the "Exited (0)" status it was looking for.

Third, the previous fix created a false sense of progress. The assistant's first attempt (message 292) looked correct on paper. It added a pre-check, it handled both success and failure cases, and it passed syntax validation. But it was built on an incorrect assumption about how docker-compose ps behaves. The fix was tested in isolation (syntax check passed), but not end-to-end with the actual Docker Compose output.

Fourth, the error message was misleading. The script didn't fail—it just hung. There was no error output, no crash, no obvious signal that something was wrong. The user had to wait 30 loop iterations (approximately 30 seconds) before the script timed out and moved on. This made the bug feel like a performance issue rather than a logic error.

The Assumptions That Failed

The debugging chain reveals several incorrect assumptions that the assistant made:

That docker-compose ps shows exited containers. This was the root assumption. The assistant had written the original wait logic assuming that docker-compose ps db-init would show the container even after it exited. When that proved false, the first fix attempted to work around it without actually verifying the assumption.
That the pre-check would catch the container before it exited. The first fix assumed that by checking docker-compose ps before entering the loop, the script would find the container while it was still running. But the pre-check used the same docker-compose ps command without -a, so if the container had already exited (which it had, as shown in the docker-compose up output), the pre-check would fail too.
That the fix was complete after the first attempt. The assistant announced the fix in message 294 with confidence: "The script should now immediately recognize that db-init completed." But the user's test in message 295 showed the same behavior. The assistant had not actually tested the fix against a real Docker Compose environment.
That the issue was with timing rather than visibility. The assistant's initial analysis focused on the timing of when the container exited versus when the script checked. The real issue was not timing but visibility—the container was invisible to the command being used.

Input Knowledge Required

To understand this message fully, a reader needs:

Docker Compose basics: Understanding that docker-compose up -d starts services in the background, and that docker-compose ps shows the status of those services.
The -a flag semantics: Knowing that docker-compose ps defaults to showing only running containers, and that -a (or --all) is required to see exited containers.
Shell scripting patterns: Familiarity with grep -q for pattern matching, loop constructs in bash, and exit code checking.
The test cluster architecture: Understanding that db-init is a one-shot container that creates a database and exits, as opposed to a long-running service like YugabyteDB or the Kuri nodes.
The debugging history: Knowing that this was the third attempt to fix the same symptom, and that previous fixes had failed because they didn't address the root cause.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

A corrected startup script that properly handles the lifecycle of one-shot initialization containers.
A reusable debugging pattern: When a container's status isn't showing up in docker-compose ps, try docker-compose ps -a to see all containers.
A documented behavior of Docker Compose: The message explicitly states that exited containers don't show in docker-compose ps by default, which is a fact that many developers encounter only through painful experience.
A template for future debugging: The sequence of steps—observe the symptom, check the command output directly, compare with expectations, identify the gap—is a model for systematic debugging.

The Thinking Process

The assistant's reasoning, visible in the message headers and the surrounding context, shows a progression from assumption-driven debugging to observation-driven debugging.

In message 291, the assistant's reasoning starts with: "The user is pointing out that the script is waiting unnecessarily. The db-init already exited successfully." This is a correct observation, but the proposed fix is based on an assumption: "The issue is that the grep pattern 'Exited (0)' might not match if the output format is different."

In message 296, the assistant takes a different approach. Instead of guessing at the pattern, it runs the actual command: docker-compose ps. The output reveals the truth: the container isn't there at all. The reasoning in message 297 shows the moment of insight: "Ah, I see! The db-init container already exited and was removed (it's not showing in docker-compose ps because it has restart: 'no' and already completed)."

This is the critical shift. The assistant moves from "maybe the pattern doesn't match" to "the container isn't in the output at all." The fix follows directly from this observation.

Conclusion

Message 300 is a small message with a large lesson. It demonstrates that the most effective debugging is often not about writing more code or adding more logic, but about observing the system's actual behavior and adjusting your assumptions to match reality. The single-character change from docker-compose ps to docker-compose ps -a is trivial in isolation, but it represents the culmination of a debugging process that exposed a fundamental misunderstanding about how Docker Compose reports container state.

For the broader project, this fix was a prerequisite for the major architecture correction that followed. Once the startup script could reliably detect database initialization, the team could move on to the more fundamental issues: the Kuri nodes failing because no external module was configured, and the architectural error of running Kuri nodes as direct S3 endpoints instead of using separate stateless frontend proxies. But without this fix, every test run would have been blocked by a script that never stopped waiting for a container that had already finished its work.

In the end, the invisible container was never truly invisible. It was always there, sitting quietly in Docker's internal state, waiting for someone to use the right flag to see it.