The Container That Wasn't There: A Docker Compose Debugging Epiphany

In the middle of a high-stakes debugging session for a horizontally scalable S3 storage system, a single message from the assistant captures a moment of sudden realization—the kind that every infrastructure engineer knows well. The message is brief, almost casual, but it represents a critical pivot in a troubleshooting chain that had been running for hours. The assistant writes:

The kuri nodes are not showing up! The s3-proxy is using an old image (hash starts with 542637). Let me restart properly:

This is message 1113 in a long conversation about building and debugging a distributed S3 architecture with Kuri storage nodes, a stateless S3 frontend proxy, and a YugabyteDB backend. To understand why this message matters, we need to trace the chain of events that led to it—and the assumptions that had to be broken before the real problem could be seen.

The Context: A Load Test Gone Wrong

The session had been progressing well. The assistant had just implemented a CQL batcher to optimize database writes, built a new Docker image with the changes (message 1104), and was eager to run performance benchmarks. The user had given a clear directive: "Restart with changes, test at 10/100/1000 parallel" (message 1087). The assistant dutifully built the binary, then attempted to restart the Docker Compose cluster.

The first attempt used docker compose restart (message 1107), which seemed to work—containers were restarted, the S3 proxy responded. But when the load test ran with 10 concurrent workers (message 1109), the results were catastrophic: all write operations failed. This was confusing because the system had been working perfectly before the restart, passing a 60-second stress test with 16 workers and zero corruption (message 1083).

The assistant's debugging instinct kicked in. Checking the logs of kuri-1 (message 1110) revealed a bizarre pattern: the node was stuck in an infinite loop printing "syncing group 101" over and over. This wasn't a normal startup sequence—something was fundamentally wrong with the node's initialization. The logs also showed stale error messages from a previous run ("Error handling HTTP PUT request" with timestamps from an earlier test), suggesting the container hadn't actually restarted cleanly.

The Epiphany: What `docker compose ps` Revealed

The breakthrough came when the assistant ran docker compose ps (message 1112). The output was startling:

NAME                      IMAGE                                     ...  STATUS
test-cluster-s3-proxy-1   sha256:542637acf3efe17633dbd008c6759ff54cd610b692f7c94c05b098762e13c19c   ...  Up About a minute
test-cluster-webui-1      nginx:alpine                               ...  Up

The kuri nodes were not listed at all. Not failing, not crashing—simply absent from the process table. And the s3-proxy was running an image with hash 542637..., which was the old image, not the freshly built one.

This is the moment captured in message 1113. The assistant connects two observations into a single diagnosis:

Missing containers: kuri-1 and kuri-2 aren't running, which explains why all write operations failed—there were no storage nodes to accept the data.
Stale image: The s3-proxy is using an old image hash, meaning docker compose restart didn't pick up the newly built image. The conclusion is immediate and correct: docker compose restart only restarts existing containers with their current image. It does not pull in newly built images. The assistant needed docker compose up -d --force-recreate to force Docker to create fresh containers from the latest image.

The Hidden Assumption: How `restart` Betrays the Unwary

The mistake here is subtle and extremely common. The assistant had built a new Docker image with docker build -t fgw:local . (message 1104). The Docker Compose file presumably referenced this image. When the assistant ran docker compose restart kuri-1 kuri-2 s3-proxy, Docker Compose dutifully stopped and started the existing containers—but those containers were created from the previous image build. The new image sat unused in the local registry.

The assumption was that restart would somehow "pick up" the new image. In Docker Compose, this is not the case. The restart command is designed for applying new configuration or recovering from crashes, not for deploying new images. The correct command is up -d --force-recreate, which destroys the old containers and creates new ones, resolving the image reference fresh.

But even up -d --force-recreate has a pitfall, visible in the output of message 1113: time="2026-01-31T16:18:57+01:00" level=warning msg="No services to build". This warning means Docker Compose doesn't know it needs to rebuild the image—it will use whatever is already tagged as fgw:local. The assistant had built the image manually with docker build, so the tag existed. But if the build had failed or the tag was different, this command would silently use the stale image.

The Thinking Process: A Detective's Chain of Inference

The reasoning visible in message 1113 is compressed but reveals a clear chain of inference:

Observation: Load test fails with all write errors.
Hypothesis 1: Kuri nodes crashed or failed to start. → Check logs (message 1110). Logs show "syncing group 101" loop and stale errors.
Hypothesis 2: Kuri nodes aren't running at all. → Check docker compose ps (message 1112). Confirmed: kuri-1 and kuri-2 are absent.
Observation: s3-proxy is running but with old image hash.
Synthesis: The restart didn't deploy the new image, and the kuri nodes failed to start (or never started) with the old configuration. The assistant doesn't stop to investigate why the kuri nodes disappeared—that would be a separate debugging thread. Instead, the message shows a pragmatic decision: fix the deployment first, then see if the problem recurs. The --force-recreate flag is the right tool to ensure clean state.

Input Knowledge Required

To understand this message, the reader needs:

Docker Compose fundamentals: The difference between restart, up, stop, and down, and how container images are resolved at creation time.
The architecture under test: A three-layer system with stateless S3 proxies, Kuri storage nodes, and YugabyteDB. The kuri nodes are the critical data storage layer—without them, all writes fail.
The session history: The assistant had just built a new Docker image with batcher changes, and the user had requested a restart to test performance at increasing concurrency levels.
The load test tooling: The ritool loadtest command that performs S3 PUT/GET operations with read-after-write verification.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

A corrected deployment procedure: The assistant now knows that restart is insufficient for deploying new code. The correct sequence is: build image → up -d --force-recreate (or down then up -d).
A diagnostic pattern: When containers go missing after a restart, check docker compose ps first, not just logs. Missing containers explain total failures more completely than log errors.
A warning sign: The "No services to build" message from Docker Compose is a red flag that the image might not be up-to-date.
A reproducible failure mode: Restarting containers without recreating them can leave the system in a broken state where some services are running old code and others aren't running at all.

The Broader Lesson: Infrastructure Debugging as Pattern Recognition

What makes message 1113 interesting is not the technical depth—it's a simple Docker Compose command—but the cognitive shift it represents. The assistant had been deep in code-level debugging: checking Go source files, modifying load test logic, implementing a batcher. The failure mode turned out to be purely operational: the deployment process itself was broken.

This is a classic pattern in distributed systems debugging. When a system that was working stops working after a "restart," the first question should always be: did the restart actually deploy what I think it deployed? The assistant's realization—"The kuri nodes are not showing up!"—is the moment when the debugging frame shifts from "what's wrong with the code?" to "what's wrong with the deployment?"

The message also illustrates the value of reading docker compose ps output carefully. The assistant noticed the image hash and recognized it as old. This kind of attention to detail—not just checking that containers are "Up" but verifying which image they're running—separates effective debugging from guesswork.

Conclusion

Message 1113 is a small but perfect example of infrastructure debugging in the wild. It captures the moment of insight when scattered observations coalesce into a correct diagnosis: the kuri nodes are missing, the proxy is running stale code, and docker compose restart was the wrong tool for the job. The assistant's response is immediate and practical—force recreate the containers—and the thinking process, though compressed into a single sentence, reveals a clear chain of inference from symptom to cause to remedy.

For anyone who has ever run docker compose restart and wondered why their changes didn't take effect, this message is a mirror. The assumption that "restart" means "restart with the latest code" is so natural that it takes a failure to expose it. And that's exactly what happened here: the system broke, the logs were confusing, and only by stepping back to check what was actually running did the real problem become visible.