The Invisible Container: Debugging a Silent Failure in Docker Compose Status Checks

Introduction

In the middle of building a horizontally scalable S3-compatible storage cluster, a developer encounters a frustrating problem: a startup script keeps waiting for a container that has already finished its job. The container shows as "Exited" in the Docker Compose output, yet the script's wait loop spins through 30 retries without ever detecting success. This is the moment captured in message 296 of a coding session — a brief but revealing debugging step where the assistant runs a single docker-compose ps command to understand why its logic is failing. What seems like a trivial check actually exposes a subtle but critical gap between how Docker Compose reports container status and how shell scripts interpret that output. This article unpacks that moment, examining the reasoning, assumptions, mistakes, and knowledge boundaries that surround this single diagnostic action.

The Message in Full

The subject message reads:

[assistant] ## Agent Reasoning
The user is saying the script is still waiting for db-init even though it already exited. Let me check the docker-compose ps output to understand what's happening. The issue is likely that the grep pattern isn't matching correctly.
[bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker-compose ps 2>/dev/null
NAME                      IMAGE                                COMMAND                  SERVICE    CREATED              STATUS                        PORTS
test-cluster-yugabyte-1   yugabytedb/yugabyte:2024.2.5.0-b59   "/sbin/tini -- bin/y…"   yugabyte   About a minute ago   Up About a minute (healthy)   5433/tcp, 6379/tcp, 7000/tcp, 7100/tcp, 7200/tcp, 9000/tcp, 9042/tcp, 9100/tcp, 10100/tcp, 11000/tcp, 12000/tcp

At first glance, this is simply a developer running a diagnostic command. But the output tells a deeper story: only one container appears — the YugabyteDB node — while the db-init container, which the script is supposed to be waiting for, is entirely absent from the listing. This absence is the key to understanding the bug.

Context and Motivation: Why This Message Was Written

To understand why this message exists, we need to trace the chain of events leading up to it. The developer (working with an AI assistant) has been building a test cluster for a distributed S3 storage system called "Filecoin Gateway" (FGW). The architecture involves multiple Kuri storage nodes backed by a shared YugabyteDB database, with a stateless S3 frontend proxy layer for request routing.

The cluster uses Docker Compose for orchestration, and a critical part of the startup sequence is a db-init service that creates the required PostgreSQL database and CQL keyspace before the Kuri nodes attempt to connect. This service is designed to run once and exit — it creates the database, then terminates. The start.sh script has a wait loop that polls docker-compose ps looking for the string "Exited (0)" in the db-init container's status, which would indicate successful initialization.

The user has just run start.sh and observed that the script is stuck in its wait loop, printing "Waiting for db-init... (1/30)" through "(20/30)" and beyond, even though the Docker Compose output at startup clearly showed "Container test-cluster-db-init-1 Exited." The container has finished its work, but the script cannot see that fact.

The assistant's reasoning in the message is straightforward: "The user is saying the script is still waiting for db-init even though it already exited. Let me check the docker-compose ps output to understand what's happening. The issue is likely that the grep pattern isn't matching correctly." This is a hypothesis-driven debugging step — the assistant suspects a pattern-matching problem in the grep expression and runs docker-compose ps to examine the actual output format.

The Critical Discovery: What the Output Reveals

The output of docker-compose ps is revealing — but not for the reason the assistant initially expected. The command returns only one container:

test-cluster-yugabyte-1   ...   Up About a minute (healthy)

The db-init container is not listed at all. This is the root cause of the bug, and it's a different problem than a grep pattern mismatch. The docker-compose ps command, by default, only shows running containers. Once a container has exited — even with exit code 0 — it disappears from the default output. The script's wait loop was checking docker-compose ps db-init | grep -q "Exited (0)", but since the exited container doesn't appear in the output, the grep never finds a match, and the loop runs until it exhausts its 30 retries.

This is a classic Docker Compose gotcha. The ps command has a -a (or --all) flag that includes stopped and exited containers, but the default behavior is to show only running services. The assistant's earlier fix attempt (in message 292) had tried to address this by adding an initial check for already-exited containers, but that fix was still using docker-compose ps without the -a flag, so it suffered from the same invisibility problem.

Assumptions and Misconceptions

Several assumptions are at play in this message, and understanding them is key to appreciating the debugging process.

Assumption 1: The container would appear in docker-compose ps output after exiting. This is the most fundamental assumption underlying both the assistant's initial fix and the ongoing debugging. The assistant assumed that an exited container would still be visible in the default ps output, just with a different status string. In reality, Docker Compose hides exited containers by default, making them invisible to naive status checks.

Assumption 2: The grep pattern was the problem. The assistant's stated hypothesis is that "the grep pattern isn't matching correctly." This is a reasonable guess based on the symptoms — the script is looping without finding the expected string. But it's an assumption about the format of the output rather than its existence. The pattern might be perfectly fine; there's simply no output to match against.

Assumption 3: The previous fix was sufficient. In message 292, the assistant had already modified start.sh to add an early check for already-exited containers. The assumption was that this fix would handle the case where db-init finishes before the wait loop starts. But the fix inherited the same docker-compose ps call without -a, so it was checking for a container that had already become invisible.

Assumption 4: The user's environment was set up correctly. The assistant runs the diagnostic command with FGW_DATA_DIR=/data/fgw2, matching the user's invocation. But the command also includes 2>/dev/null to suppress stderr, which could hide error messages that might provide additional clues about what's happening.

Input Knowledge Required

To fully understand this message, a reader needs several pieces of contextual knowledge:

  1. Docker Compose basics: Understanding that docker-compose ps lists containers and their statuses, and that the -a flag changes what's shown.
  2. The test cluster architecture: Knowing that db-init is a one-shot initialization container that creates a database and exits, and that the startup script needs to wait for it before proceeding.
  3. Shell scripting patterns: Recognizing that grep -q is used for silent pattern matching in conditional checks, and that docker-compose ps <service> filters output to a specific service.
  4. The previous debugging iterations: The permission fix (message 283), the database-already-exists fix (message 287), and the first attempt at fixing the wait loop (message 292) all inform why the assistant is now looking at the ps output.
  5. The FGW project context: This is a horizontally scalable S3 storage system with Kuri storage nodes, a stateless frontend proxy layer, and YugabyteDB for metadata coordination.

Output Knowledge Created

This single diagnostic command produces several important pieces of knowledge:

  1. Confirmation that db-init is not in the default ps output: The most important finding. The absence of db-init from the listing proves that the container has exited and been removed from the default view.
  2. Evidence that the grep-pattern hypothesis is wrong: The output format shows "Up About a minute (healthy)" for the running container, which is clean and parseable. The issue isn't pattern complexity; it's data absence.
  3. A clear path to the real fix: The solution is now obvious — use docker-compose ps -a (or check container exit codes via docker inspect or docker-compose run exit codes) to detect completed containers.
  4. Validation that the cluster infrastructure is otherwise healthy: The YugabyteDB container is "Up About a minute (healthy)", confirming that the database is running correctly and the network is operational.
  5. A debugging methodology demonstration: The message shows the value of running the exact same command the script uses, rather than theorizing about what it might output. This direct inspection approach is a best practice for diagnosing status-checking logic.

The Thinking Process: A Window into Debugging

The assistant's reasoning reveals a structured debugging approach. The chain of thought progresses through several stages:

Stage 1 — Symptom recognition: The user reports that the script is still waiting at iteration 20 of 30. The assistant immediately recognizes this as abnormal — the container should have exited long before.

Stage 2 — Hypothesis formation: The assistant forms an initial hypothesis: "The issue is likely that the grep pattern isn't matching correctly." This is a reasonable starting point, as pattern matching is a common source of shell scripting bugs.

Stage 3 — Evidence gathering: Rather than continuing to theorize, the assistant runs a diagnostic command to examine the actual output. This is the key action in the message — moving from speculation to observation.

Stage 4 — Pattern recognition (implicit): The output shows only one container. While the assistant doesn't explicitly comment on the absence of db-init in the reasoning shown, the act of running the command and displaying its output sets up the next step in the debugging process. The reader can infer that the assistant now sees the real problem.

What's notable about this thinking process is what it doesn't include. The assistant doesn't immediately jump to the -a flag solution. It doesn't pre-judge the output. It simply runs the command and presents the evidence. This is a disciplined debugging approach — let the data speak before forming conclusions.

The Broader Architectural Context

This debugging moment sits within a larger story of architectural evolution. The test cluster being built is not a simple monolith — it's a distributed system with multiple interacting services, each with its own lifecycle. The db-init container is a dependency orchestrator, ensuring that the database schema exists before storage nodes attempt to connect. The fact that its exit status is invisible to the default docker-compose ps output is a subtle but important detail that affects the reliability of the entire startup sequence.

In production, this kind of status-checking logic is critical. Automated deployment scripts, health checks, and orchestration systems all depend on accurate container status reporting. A bug like this — where a successfully completed initialization step is invisible to the monitoring system — could lead to false negatives, unnecessary retries, or cascading startup failures in more complex deployments.

Conclusion

Message 296 is a small but instructive moment in the software development process. It captures the precise instant when a hypothesis meets reality — when the assistant runs a diagnostic command expecting to see a pattern-matching problem, and instead discovers a more fundamental issue about container visibility in Docker Compose. The message demonstrates the importance of direct observation in debugging, the value of questioning assumptions about tool behavior, and the subtle ways that default settings can create invisible failures.

The fix that follows from this discovery is straightforward: add the -a flag to docker-compose ps calls that need to detect exited containers. But the insight that leads to that fix — that exited containers are invisible by default — is the real knowledge created in this message. It's a reminder that in complex systems, the most important debugging step is often the simplest: look at what's actually happening, rather than what you assume should be happening.