The Docker Compose Debug That Exposed a Hidden Assumption

In the trenches of distributed systems development, the most frustrating bugs are often not the complex algorithmic ones, but the simple operational gotchas that hide in plain sight. This article examines a single message from an opencode coding session where the assistant, while debugging a test cluster for a horizontally scalable S3 storage system, discovered a subtle but critical issue with Docker Compose's default behavior: exited containers don't show up in docker-compose ps without the -a flag. The message itself is brief, but the debugging journey that led to it reveals important lessons about assumptions, tooling behavior, and the nature of infrastructure debugging.

The Subject Message

The message under analysis is message 298 from the conversation, in which the assistant writes:

## Agent Reasoning The issue is that docker-compose ps db-init doesn't show exited containers by default. I need to use docker-compose ps -a to see all containers including exited ones. Let me fix this. [edit] /home/theuser/gw/test-cluster/start.sh Edit applied successfully.

This short message represents the culmination of a multi-step debugging process that had been running for several iterations. To understand why this realization matters, we need to examine the full context of what was being built and the series of failures that led to this moment.

The Broader Context: Building a Horizontally Scalable S3 Cluster

The assistant was working on a Filecoin Gateway project implementing a horizontally scalable S3-compatible storage architecture. The architecture followed a three-layer design: stateless S3 frontend proxy nodes handled request routing and load balancing, backend Kuri storage nodes maintained independent RIBS blockstore data, and a shared YugabyteDB cluster coordinated object placement metadata. This was a non-trivial distributed systems project with multiple moving parts.

To validate the architecture, the assistant had built a Docker Compose-based test cluster with two Kuri storage nodes, a shared YugabyteDB instance, and supporting infrastructure. The test cluster included a suite of shell scripts (start.sh, stop.sh, init-data.sh, etc.) parameterized with a data directory argument. These scripts orchestrated the container lifecycle, handling initialization, health checks, and graceful shutdown.

The Bug: A Wait Loop That Never Terminates

The specific bug that led to message 298 involved the start.sh script's wait logic for database initialization. The test cluster used a db-init container that ran a one-time initialization script to create the required PostgreSQL database and CQL keyspace before the Kuri nodes could connect. This container had restart: "no" in its Docker Compose configuration, meaning it would run once and exit.

The start.sh script contained a loop that waited for db-init to complete:

for i in {1..30}; do
    if docker-compose ps db-init | grep -q "Exited (0)" 2>/dev/null; then
        echo "✅ Database initialized"
        break
    fi
    ...
    sleep 2
done

The problem was that this loop never terminated. The user reported that the script was still waiting after 20 iterations, even though db-init had clearly exited (as shown in the docker-compose up output: "Container test-cluster-db-init-1 Exited").

The Initial Fix Attempt: A Partial Understanding

The assistant's first attempt to fix this issue (in messages 291-294) modified the wait logic to check if db-init had already exited before entering the loop. The fix used docker-compose ps db-init to check for "Exited" status. However, this fix still didn't work — the user reported the same waiting behavior in message 295.

This is where the debugging gets interesting. The assistant ran docker-compose ps directly (message 296) and discovered that db-init wasn't even listed in the output. The container had exited and was no longer visible. This was the critical clue.

The Root Cause: Docker Compose's Default Behavior

The insight that arrived in message 298 was precise and correct: docker-compose ps does not show exited containers by default. The -a flag (or --all) is required to include containers that have exited. This is a well-known Docker Compose behavior, but it's easy to forget or overlook, especially when you're deep in debugging mode and making assumptions about how the tools work.

The assistant's reasoning shows the moment of clarity: "The issue is that docker-compose ps db-init doesn't show exited containers by default. I need to use docker-compose ps -a to see all containers including exited ones."

This realization was the key to fixing the bug. The previous fix attempt had used docker-compose ps db-init to check for "Exited" status, but since exited containers don't appear in the default output, the check would never match. The script would then enter the wait loop, which also used docker-compose ps db-init to check for completion, and would loop forever (or until the timeout) because the container was invisible to the check.

The Fix: Applying the Insight

The fix was straightforward once the root cause was understood: change all references to docker-compose ps db-init to docker-compose ps -a db-init in the wait logic. This ensured that the exited container would be visible and its exit status could be checked.

The assistant applied this edit to start.sh and the fix was confirmed working in subsequent testing.## The Debugging Journey: A Case Study in Iterative Problem Solving

The path to message 298 is a textbook example of iterative debugging. Let's trace the full arc:

Initial symptom (message 285): The user ran start.sh and got service "db-init" didn't complete successfully: exit 1. The database already existed from a previous run, causing CREATE DATABASE to fail.
First fix (message 287): The assistant added 2>/dev/null || echo 'Database may already exist' to the SQL command, suppressing the error when the database already exists.
Second symptom (message 290): The db-init now exited successfully, but the script was waiting for it unnecessarily. The user noted "seem to incorrectly wait for db-init where it was already waited on and correctly(?) exited."
Second fix attempt (message 291-294): The assistant modified the wait logic to check if db-init had already exited before entering the loop, using docker-compose ps db-init | grep -q "Exited".
Third symptom (message 295): The fix didn't work. The script still waited through all 30 iterations. The user reported "still an issue."
Investigation (message 296): The assistant ran docker-compose ps directly and discovered that db-init wasn't listed at all. The container had exited and was removed from the default view.
Insight (message 298): The assistant realized that docker-compose ps doesn't show exited containers by default. The -a flag is required. This progression shows a classic debugging pattern: each fix addressed a surface-level symptom without fully understanding the underlying mechanism. The first fix handled the database-already-exists error. The second fix attempted to handle the exited container case but used the wrong command. Only when the assistant directly inspected the tool's output (message 296) did the real issue become apparent.

The Assumption That Failed

The critical assumption that broke the debugging process was this: the assistant assumed that docker-compose ps shows all containers, including those that have exited. This is a reasonable assumption — many command-line tools do show all relevant entities by default. docker ps, for example, shows running containers by default but exited containers only with -a. Docker Compose's ps command follows the same convention, but it's easy to forget this when you're focused on the logic of your script.

This assumption cascaded through the fix. The assistant wrote code that checked for "Exited" in the output of docker-compose ps db-init, but since exited containers don't appear in that output, the check could never succeed. The script would then enter the wait loop, which also used the same broken check, and would loop forever.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of:

Docker Compose container lifecycle: Containers with restart: "no" run once and exit. They are considered "exited" and are not shown in the default docker-compose ps output.
The docker-compose ps -a flag: The -a (or --all) flag includes exited containers in the output, showing their exit status.
Shell scripting and grep patterns: The script used grep -q to check for "Exited" status, and later used sed to extract the exit code.
The test cluster architecture: Understanding that db-init was a one-shot initialization container that needed to complete before the Kuri nodes could start.

Output Knowledge Created

This message produced several valuable outputs:

A corrected start.sh script that properly handles exited containers by using docker-compose ps -a instead of docker-compose ps.
A documented debugging insight that exited containers are hidden by default in Docker Compose.
A working test cluster that could reliably start up without getting stuck in an infinite wait loop.
A reusable pattern for checking container exit status in shell scripts that can be applied to other Docker Compose-based projects.

Broader Lessons for Infrastructure Debugging

This message illustrates several important principles for debugging distributed systems and infrastructure:

Tool behavior is not always what you expect. Even experienced developers can make incorrect assumptions about how common tools behave. Docker Compose's ps command hiding exited containers is a deliberate design choice (to avoid clutter), but it can be surprising when you're trying to check the status of a completed container.

Direct inspection beats inference. The assistant's breakthrough came when they ran docker-compose ps directly and saw that db-init wasn't listed. This direct observation contradicted their mental model and forced a re-evaluation. When debugging, running the actual command and looking at its raw output is often more productive than reasoning about what the output should be.

Iterative debugging requires questioning assumptions at each step. The assistant went through multiple fix-attempt-fix cycles. Each cycle refined the understanding of the problem. The key was that the assistant didn't just try different fixes blindly — they investigated why the previous fix didn't work, which led to the root cause.

Shell scripts are fragile. The wait logic in start.sh was a simple loop with a grep check, but it broke because of a subtle difference in tool behavior. This fragility is inherent in shell scripting, where error handling is often implicit and edge cases are easy to miss.