The Moment of Discovery: A Docker Compose PS That Revealed a Broken Restart

The Message

In the middle of a high-stakes debugging session for a horizontally scalable S3 storage system, the assistant ran a seemingly mundane command:

[assistant] [bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose ps 2>&1
NAME                      IMAGE                                                                     COMMAND                  SERVICE    CREATED             STATUS                       PORTS
test-cluster-s3-proxy-1   sha256:542637acf3efe17633dbd008c6759ff54cd610b692f7c94c05b098762e13c19c   "./s3-proxy"             s3-proxy   50 minutes ago      Up About a minute            0.0.0.0:8078->8078/tcp, [::]:8078->8078/tcp
test-cluster-webui-1      nginx:alpine                                          ...

This is message 1112 in a long conversation about building and debugging a distributed S3-compatible storage cluster. On its surface, it is nothing more than a container status check. But in context, it represents a critical turning point — the moment when the assistant discovered that a carefully orchestrated restart had silently failed, leaving the cluster in a broken state with no storage nodes running.

The Context: A Cluster on the Brink of Performance Testing

To understand why this simple diagnostic command matters, we must understand what led to it. The assistant had just completed a major optimization effort: implementing a CQLBatcher in the database layer to batch individual CQL INSERT calls, reducing database contention under high concurrency. The batcher collected writes and flushed them in batches of up to 15,000 entries, using a worker pool with exponential backoff retries, while preserving read-after-write consistency by blocking callers until the batch was durably committed.

The user had given a clear directive: "Restart with changes, test at 10/100/1000 parallel." This was a performance validation mission. The assistant needed to rebuild the Docker image with the new batcher code, restart the cluster, and run load tests at escalating concurrency levels to measure throughput improvements.

The assistant followed the steps methodically. It rebuilt the Docker image with docker build -t fgw:local . (message 1104), then attempted to restart the cluster using docker compose restart kuri-1 kuri-2 s3-proxy (message 1107). The restart command reported success: "Container test-cluster-kuri-2-1 Restarting... Container test-cluster-kuri-1-1 Restarting... Container test-cluster-s3-proxy-1 Restarting... Started." On the surface, everything looked fine.

But when the assistant ran the first load test at 10 concurrent workers (message 1109), the results were catastrophic: all write operations failed with errors. Something was deeply wrong.## The Diagnostic Chain: From Load Test Failure to Container Status

The load test failure triggered a diagnostic chain. The assistant first checked the kuri-1 logs (message 1110), which showed a suspicious pattern: "syncing group 101" repeating endlessly, along with an earlier error about "context canceled" during a PUT request. But the logs were truncated and didn't reveal the root cause clearly.

Then came message 1111, where the assistant checked the current container status. The output was incomplete — the message only shows the s3-proxy line and part of the webui line — but what it doesn't show is the critical information: the kuri-1 and kuri-2 containers are absent from the listing. The docker compose ps output, which should list all containers defined in the compose file, only shows the s3-proxy and webui. The storage nodes are missing.

This is the moment of discovery. The assistant's next message (1113) makes the realization explicit: "The kuri nodes are not showing up! The s3-proxy is using an old image (hash starts with 542637)."

Why the Restart Failed: Understanding Docker Compose Semantics

The root cause of this failure is a subtle but important detail about Docker Compose behavior. When the assistant ran docker compose restart kuri-1 kuri-2 s3-proxy, it was using the restart subcommand, which stops and starts the specified containers. However, restart does not recreate containers with a new image. If the image has changed — which it had, since the assistant just rebuilt fgw:local with the batcher code — the restart command will continue using the old container instances created from the previous image.

The s3-proxy container was restarted successfully because it already existed from a previous deployment. But the kuri nodes, which may have been in a crashed or stopped state, were not properly recreated. The restart command simply tried to start the existing container instances, which may have been using stale configurations or had underlying issues.

The assistant's assumption was that restart would pick up the new image. This is a common mistake — docker compose restart is for restarting existing containers, while docker compose up -d --force-recreate is needed to force container recreation with a new image. The distinction matters enormously in development workflows where images are iteratively rebuilt.## Assumptions Made and Their Consequences

The assistant made several assumptions during this sequence that turned out to be incorrect:

  1. That docker compose restart would pick up the new image. This is the most consequential assumption. Docker Compose's restart command operates on existing container instances; it does not check for or apply image updates. The new fgw:local image, freshly built with the batcher code, was never deployed to the kuri containers.
  2. That the restart command's success messages indicated a healthy restart. The output said "Started" for all three containers, but this only means the container processes began running, not that they were running the correct code or that their services were functional.
  3. That the S3 proxy's responsiveness (returning "Not Found" for a root GET) indicated cluster health. The assistant noted "Good, the S3 proxy is responding" in message 1108, interpreting the 404 response as evidence that the proxy was operational. In reality, the proxy was running but had no healthy backend storage nodes to route requests to.
  4. That the load test failure was a configuration or code issue rather than a deployment issue. When the first load test failed with all write errors, the assistant's initial instinct was to check logs for configuration problems, not to verify that the containers were actually running the new code. These assumptions cascaded. Each one seemed reasonable in isolation, but together they created a false picture of a functioning cluster. The docker compose ps command in message 1112 was the reality check that shattered this picture.

Input Knowledge Required to Understand This Message

To fully grasp the significance of message 1112, a reader needs several pieces of contextual knowledge:

Output Knowledge Created

Message 1112 produced concrete, actionable knowledge:

  1. The kuri-1 and kuri-2 containers were not running. The docker compose ps output only showed s3-proxy and webui. This was definitive proof that the restart had not worked as intended.
  2. The s3-proxy was running an old image. The image hash starting with "542637" was from a previous build, confirming that the new fgw:local image had not been deployed to any container.
  3. The cluster was in a degraded state. With no storage nodes, the S3 proxy was a hollow shell — it could accept requests but had no backends to process them. This explained the "Service Unavailable - No healthy backends" error that would appear in subsequent diagnostics.
  4. The correct fix was identified. The assistant immediately recognized the problem and, in the next message (1113), ran docker compose up -d --force-recreate to force container recreation with the new image.## The Thinking Process Visible in the Message Message 1112 is a diagnostic command, not a reasoning output. But the thinking process is visible in why this particular command was chosen at this particular moment. The assistant had just run a load test that failed completely (message 1109), then checked the kuri-1 logs and found them unhelpful (messages 1110-1111). The logical next step was to verify the fundamental assumption: are the containers actually running? The choice of docker compose ps over other diagnostic tools reveals a methodical debugging approach. The assistant could have: - Checked the Docker event log - Tried to curl the kuri nodes directly on their internal ports - Restarted everything from scratch - Dived deeper into the kuri configuration Instead, it went for the most basic, high-level status check: "Show me what containers are running." This is the debugging equivalent of "is it plugged in?" — a sanity check that should have been done earlier but was skipped because the restart command reported success. The incomplete output in the message is also telling. The docker compose ps output was truncated (ending with "nginx:alpine ..."), but the critical information was already visible: only two services were listed, and neither was a kuri node. The assistant didn't need to see the full output to understand the situation.

Broader Implications: The Fragility of Container Orchestration in Development

This episode highlights a recurring challenge in distributed systems development: the gap between "the container is running" and "the service is working correctly." Docker Compose reports container status based on process health — if the entrypoint process is alive, the container is "Up." But a container can be "Up" while running the wrong code, with a broken configuration, or with a service that fails to initialize properly.

In this case, the kuri containers may have been in a crash-loop state or may never have been properly created from the new image. The restart command's success messages were misleading because they only reflected Docker's attempt to start the containers, not the actual health of the services inside them.

This is why production deployments typically use health checks, readiness probes, and rolling update strategies. The test cluster, being a development environment, lacked these safeguards. The assistant's debugging process — load test fails → check logs → check container status → discover the real problem — is essentially a manual version of what a production orchestrator like Kubernetes would do automatically.

Mistakes and Lessons

The primary mistake in this sequence was using docker compose restart when docker compose up -d --force-recreate was needed. This is a common pitfall. The restart subcommand is useful for applying configuration changes or recovering from transient failures, but it is not a deployment tool. When the underlying image changes, containers must be recreated.

A secondary mistake was trusting the restart command's success output without verifying. The assistant saw "Started" and moved on to load testing, assuming the cluster was healthy. A quick docker compose ps immediately after the restart would have caught the problem before any load tests were run.

The lesson is clear: in containerized development workflows, always verify container status after any restart or deployment operation. A docker compose ps check should be as automatic as checking that a service responds to requests. The assistant learned this lesson in real-time — message 1112 was the verification that should have happened earlier, and its absence led to wasted time debugging phantom issues.

Conclusion

Message 1112 — a simple docker compose ps command — is a masterclass in diagnostic humility. It represents the moment when a developer, faced with inexplicable failures, steps back from complex theories about configuration bugs, database contention, and code errors, and asks the most basic question: "Is the thing actually running?"

The answer was no. And once that was understood, the fix was straightforward. The assistant recreated the containers with --force-recreate, the kuri nodes came up properly, and the load tests could proceed. The batcher optimization was validated, and the cluster achieved impressive throughput numbers in subsequent tests.

But the real value of this message is not in the technical fix. It is in the demonstration of a debugging mindset: when everything seems wrong, start with the simplest possible check. The containers weren't running. Everything else followed from that.