The Invisible Container: How a Missing -a Flag Exposed Docker's Default Behavior

Introduction

In the middle of building a horizontally scalable S3-compatible storage cluster for the Filecoin Gateway, a seemingly minor bug emerged that would stall the entire startup pipeline. The test cluster infrastructure—composed of Docker Compose services including YugabyteDB, Kuri storage nodes, and a database initialization container—was failing to progress past a single waiting step. The start.sh script kept looping, printing "Waiting for db-init..." up to twenty times, never detecting that the database initialization had actually completed successfully. The root cause, as the assistant would discover in message 297, was a subtle but critical misunderstanding of how docker-compose ps behaves by default: it hides exited containers. The fix required the -a flag to reveal containers that had already finished their work and been removed from the active process list.

This message captures a classic debugging moment—the transition from a wrong mental model to a correct one, triggered by the simple act of running the very command the script was using and seeing what it actually returned.

The Subject Message

[assistant] ## Agent Reasoning Ah, I see! The db-init container already exited and was removed (it's not showing in docker-compose ps because it has restart: "no" and already completed). The script is checking if db-init is in the ps output, but since it already exited successfully, it's not there anymore.

>

The fix is to check docker-compose ps -a (which shows all containers including exited ones) instead of just docker-compose ps.

>

[read] /home/theuser/gw/test-cluster/start.sh

>

00086| 00087| # Wait for database initialization 00088| echo "🔍 Waiting for database initialization..." 00089| # Check if already completed (docker-compose up waited for it) 00090| if docker-compose ps db-init 2>/dev/null | grep -q "Exited"; then 00091| # Check exit code 00092| EXIT_CODE=$(docker-compose ps -a db-init 2>/dev/null | grep "Exited" | sed 's/.Exited (\([0-9]\)).*/\1/') 00093| if [ "$EXIT_CODE" = "0" ] || [ -z "$EXIT_CODE" ]; then 00094| echo "✅ Dat...

Context: The Test Cluster Pipeline

To understand why this message matters, one must understand the architecture being built. The test cluster consisted of several Docker Compose services that needed to start in a specific order. First, YugabyteDB (the shared metadata database) had to be healthy. Then, a db-init container would run a one-shot initialization script to create the required PostgreSQL database and CQL keyspace. Only after that initialization succeeded could the Kuri storage nodes start and connect to the database.

The start.sh script orchestrated this pipeline. It launched all services with docker-compose up -d, then entered a polling loop that checked whether db-init had exited successfully. The script used docker-compose ps db-init | grep -q "Exited" to detect completion. But as the user's terminal output showed, this check was failing even though db-init had clearly exited—Docker Compose's own output showed "Container test-cluster-db-init-1 Exited" during the startup phase.

The user's frustration was evident: after twenty iterations of waiting, they reported "still an issue, check docker-compose ps etc in that dir." This prompted the assistant to investigate by running the actual command manually.

The Moment of Discovery

The assistant's reasoning in message 297 reveals the exact moment of insight. They ran docker-compose ps (without -a) and saw that db-init was not listed at all. The output showed only the YugabyteDB container:

NAME                      IMAGE     COMMAND   SERVICE   STATUS
test-cluster-yugabyte-1   yugabyte  ...       yugabyte  Up About a minute (healthy)

The db-init container was simply absent from the listing. This was the critical observation. The assistant immediately connected this to the restart: "no" policy configured on the db-init service. Containers with restart: "no" that exit successfully are removed from the active container list. They still exist on disk, but docker-compose ps—which by default shows only running containers—does not display them.

The assistant's reasoning shows a clear cause-and-effect chain:

1. The script checks docker-compose ps db-init for "Exited" status
2. But docker-compose ps (without -a) only shows running containers
3. Since db-init already exited and was removed from the active list, it doesn't appear
4. Therefore the grep never finds "Exited" and the loop continues indefinitely The fix was elegantly simple: use docker-compose ps -a (the -a flag meaning "all containers, including stopped ones") instead of bare docker-compose ps.

Assumptions and Mistakes

This debugging episode reveals several layers of assumptions that turned out to be incorrect.

First assumption: The assistant assumed that docker-compose ps shows all containers associated with a Compose project. This is a natural assumption—the word "ps" in Unix traditionally shows all processes, and Docker's own docker ps shows running containers by default but docker ps -a shows all. The Compose variant follows the same convention, but the assistant had not internalized this distinction when writing the startup script.

Second assumption: The assistant assumed that the previous fix (in message 292) was sufficient. That fix had added a preliminary check for whether db-init had already exited before entering the wait loop. But it used the same docker-compose ps command, which suffered from the same blind spot. The fix addressed the symptom (the loop running unnecessarily) but not the root cause (the command not finding the container at all).

Third assumption: The assistant assumed that because docker-compose up output showed "Container test-cluster-db-init-1 Exited," the ps command would also reflect that status. But docker-compose up and docker-compose ps have different visibility into container state. The up command shows real-time events as they happen, while ps queries the current state of managed containers—and exited one-shot containers are not "managed" in the same sense.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of several domains:

Docker Compose lifecycle management: Understanding that services with restart: "no" (the default for non-daemon services) exit and are cleaned up from the active process list. This is different from services with restart: always or restart: unless-stopped, which remain in the ps output even when exited because Docker will restart them.

The difference between docker-compose ps and docker-compose ps -a: The former shows only containers that Docker Compose considers "running" (including those that will be restarted). The latter shows all containers regardless of state. This is analogous to docker ps vs docker ps -a.

Shell scripting and pipeline debugging: Understanding how the grep -q "Exited" check works, why it fails when the input is empty, and how the 2>/dev/null redirection suppresses error messages that could have provided diagnostic clues.

The test cluster architecture: Knowing that db-init is a one-shot initialization container that must complete before Kuri nodes can start, and that the startup script's polling loop is the synchronization mechanism between these stages.

Output Knowledge Created

This message produced several valuable pieces of knowledge:

A corrected startup script: The immediate output was a fix to start.sh that replaced docker-compose ps with docker-compose ps -a in the db-init wait logic. This single-character change (adding -a) resolved the entire deadlock.

A reusable debugging pattern: The assistant demonstrated the technique of running the exact command the script uses, in the exact environment, to see what it actually returns. This is a fundamental debugging practice—reproducing the problem manually rather than reasoning about it abstractly.

Documentation of Docker Compose behavior: The message implicitly documents that docker-compose ps hides exited containers by default, which is a behavior that can surprise developers who expect it to show all containers associated with a project.

A mental model correction: The assistant's understanding shifted from "the script checks container status" to "the script checks container status but the container is invisible to the query." This corrected mental model would inform future scripting decisions.

The Thinking Process

The reasoning visible in message 297 follows a classic debugging arc:

Observation: The user reports the script is still waiting after 20 iterations. The assistant has already attempted one fix (message 292) that didn't work.

Investigation: The assistant runs docker-compose ps manually and notices the db-init container is absent from the output.

Hypothesis formation: The assistant connects the absence to the restart: "no" policy and the fact that the container already exited successfully. The container is not "running" in Docker Compose's view, so it's not listed.

Hypothesis testing (implicit): The assistant knows that docker-compose ps -a shows all containers. By contrasting what they see with ps vs what they would see with ps -a, they confirm the hypothesis.

Solution generation: The fix is straightforward—change the command to include -a. The assistant also reads the current script to see exactly what needs to change.

Verification: The assistant reads the script file to confirm the exact lines that need modification, showing a methodical approach to ensuring the fix is precise.

What's notable is what the assistant doesn't do: they don't immediately jump to a more complex solution like changing the service's restart policy, adding health checks, or restructuring the startup pipeline. Instead, they identify the minimal, correct fix that aligns the script's behavior with Docker Compose's actual semantics.

Broader Implications

This message, while small in scope, illustrates several important principles for infrastructure scripting:

Default visibility matters: Docker Compose's decision to hide exited containers by default is sensible for day-to-day operations—users typically want to see what's running. But for automation scripts that need to check completion status, this default becomes a trap. The same principle applies to many CLI tools: cron doesn't show completed jobs, systemctl hides inactive services by default, and ps itself shows only your own processes without -a.

One-shot containers require special handling: Containers designed to run once and exit (database migrations, initialization scripts, data seeding) exist in a liminal state between "running" and "gone." Automation that synchronizes around these containers must account for both states: the container may still be running, or it may have already finished and been removed from view.

The value of reproducing the problem: The assistant's decision to run docker-compose ps manually in the test environment was the key insight. No amount of staring at the script would have revealed the issue—only observing the actual command output did. This reinforces the debugging maxim: "What does the computer think is true?" rather than "What do I think is true?"

Conclusion

Message 297 is a small but perfect example of a debugging insight that transforms an intractable problem into a trivial fix. The assistant's realization that docker-compose ps hides exited containers, and that the fix is simply to add -a, resolved a startup deadlock that had persisted through multiple attempted fixes. The message captures the moment when an incorrect mental model about a tool's default behavior is corrected through direct observation, producing both a working script and a deeper understanding of Docker Compose's container lifecycle management. In the broader context of building the Filecoin Gateway's test cluster, this fix was essential—without it, the entire multi-node infrastructure could never reliably start, and the subsequent work on the horizontally scalable S3 architecture would have been blocked at the first hurdle.