The Subtle Trap of docker-compose ps: A Case Study in Debugging Shell Scripts

The Message

[assistant] ## Agent Reasoning
Fixed! Now using `docker-compose ps -a` to see exited containers.
[bash] bash -n /home/theuser/gw/test-cluster/start.sh && echo "✅ Syntax OK"
✅ Syntax OK

This brief message, the 299th in a sprawling coding session, appears deceptively simple. It is the final, quiet resolution of a debugging chain that spanned five messages, two failed attempts at a fix, and a fundamental misunderstanding about how Docker Compose reports container state. The message itself contains no code changes — the edit was applied in the previous message — only a confirmation that the syntax is valid and a one-line summary of the insight that resolved the bug. Yet within this small moment lies a rich story about assumptions, debugging methodology, and the hidden complexity of orchestration tooling.

The Context: Building a Test Cluster

To understand why this message was written, we must understand the broader context. The assistant was building a Docker Compose-based test cluster for a horizontally scalable S3-compatible storage system called the Filecoin Gateway (FGW). The architecture involved multiple services: a shared YugabyteDB instance for metadata, a db-init container that bootstraps the database schema, and two Kuri storage nodes that serve S3 API requests.

The test cluster's startup script (start.sh) had a carefully orchestrated sequence: start the services, wait for YugabyteDB to become healthy, wait for db-init to complete database initialization, and then wait for the Kuri nodes to start serving. The db-init service was designed to run once and exit — it creates the PostgreSQL database and CQL keyspace, then terminates. Docker Compose's restart: "no" policy ensures it runs exactly once during startup.

The problem emerged when the user ran the cluster for a second time, reusing the same data directory. The db-init container exited successfully (the database already existed, and the assistant had previously patched the command to suppress the "already exists" error), but the startup script kept waiting for it — 8 iterations, then 20 iterations, never breaking out of its wait loop.

The Debugging Trail

The assistant's first attempt at a fix (message 291-294) modified the wait logic to check whether db-init had already exited before entering the loop. The reasoning was sound: if docker-compose up had already waited for db-init to complete, the script should detect that and skip the wait. But the fix didn't work. The user reported that the script was still looping through all 30 iterations.

This is where the debugging becomes instructive. The assistant's initial assumption was that the grep pattern wasn't matching correctly — perhaps the output format differed from what the pattern expected. But the real issue was more fundamental. When the assistant ran docker-compose ps to inspect the state of services (message 296), the output showed only one container:

NAME                      IMAGE                                COMMAND                  SERVICE    CREATED              STATUS                        PORTS
test-cluster-yugabyte-1   yugabytedb/yugabyte:2024.2.5.0-b59   "/sbin/tini -- bin/y…"   yugabyte   About a minute ago   Up About a minute (healthy)   ...

The db-init container was completely absent from the output. It had exited, and docker-compose ps — without the -a flag — does not show exited containers. The assistant's epiphany came in message 297:

"Ah, I see! The db-init container already exited and was removed (it's not showing in docker-compose ps because it has restart: "no" and already completed). The script is checking if db-init is in the ps output, but since it already exited successfully, it's not there anymore. The fix is to check docker-compose ps -a (which shows all containers including exited ones) instead of just docker-compose ps."

The Assumption and Its Correction

The critical assumption that needed correction was that docker-compose ps shows all containers associated with a service. This is a natural assumption — the command's name and common usage imply it lists the state of everything in the compose file. But Docker Compose's default behavior is to show only running containers. Exited containers, even those that ran to completion successfully, are hidden unless you explicitly request them with -a (or --all).

This behavior mirrors Docker's own docker ps command, which also defaults to showing only running containers. But the parallel is easy to miss when you're focused on the compose abstraction layer. The assistant had already used docker-compose ps -a in a previous version of the script (line 92 of the start.sh file used docker-compose ps -a db-init to extract the exit code), yet the initial check on line 90 used plain docker-compose ps db-init. This inconsistency — using -a in one place but not the other — was the root cause of the bug.

Input Knowledge Required

To understand this message and the debugging chain it concludes, several pieces of knowledge are necessary:

Docker Compose lifecycle management: Understanding that services with restart: "no" run once and exit, and that Docker Compose tracks exited containers separately from running ones.

The docker-compose ps command's default behavior: Knowing that without -a, the command only shows running containers. This is documented behavior but is easy to overlook, especially when working under time pressure.

Shell scripting and grep patterns: The script used grep -q "Exited" to detect whether the container had exited. If the container isn't in the output at all, grep returns non-zero, and the condition fails silently.

The bash -n syntax check: The subject message runs bash -n to check syntax without executing the script. This is a best practice that prevents deploying broken shell syntax.

Output Knowledge Created

This message, and the debugging session it concludes, produced several valuable outputs:

The corrected start.sh script: The immediate output was a working startup script that properly detects when db-init has completed, even if the container has already exited and been removed from the default ps view.

A reusable debugging pattern: The assistant's method — running docker-compose ps directly to inspect actual output, rather than guessing at the format — is a model for debugging shell scripts that interact with Docker Compose.

Documentation of a subtle behavior: The insight that docker-compose ps hides exited containers is now captured in the conversation history, serving as a reference for future debugging.

The Thinking Process

The reasoning visible in the assistant's messages reveals a structured debugging approach. First, hypothesis formation: "the grep pattern isn't matching correctly." Second, empirical investigation: running the actual command to see its real output. Third, pattern recognition: noticing the absence of the db-init container entirely. Fourth, knowledge retrieval: recalling that docker-compose ps -a shows all containers. Fifth, correction: applying the fix and verifying syntax.

This process mirrors the scientific method — form a hypothesis, test it against reality, revise based on evidence. The assistant's willingness to run commands directly rather than continuing to theorize is what ultimately uncovered the real issue.

Why This Matters

This message matters because it captures a moment of genuine learning. The bug was not in the logic of waiting for db-init — that logic was sound. The bug was in the tool of observation itself. The script was using docker-compose ps to observe the state of a container, but the tool was filtering out exactly the state the script needed to see. It's a reminder that every tool has a default behavior, and those defaults are not always what we assume.

The fix — adding -a — is trivial. The insight that made the fix possible was not. And that insight, captured in the assistant's reasoning, is the real value of this message. It transforms a one-line bug fix into a lesson about the subtle ways that orchestration tools present (and hide) information about the systems they manage.