The Two-Minute Wait: Debugging a Stuck Container in a Distributed S3 Test Cluster

Introduction

In the middle of a complex debugging session for a horizontally scalable S3 architecture, there comes a moment that feels both mundane and revealing: staring at a terminal, watching the same status message repeat twelve times, hoping something will change. Message 1249 of this coding session captures exactly that moment. The assistant, having just reconfigured all YugabyteDB ports to avoid conflicts with existing host services, is now waiting to see if the database container will transition from "starting" to "healthy." It never does. This message, on its surface a simple polling loop, is actually a rich artifact of the debugging process—one that reveals assumptions about system behavior, the limits of observability, and the moment when a developer realizes that a configuration change may not have worked as intended.

The Message in Full

The subject message reads:

[assistant] Good progress - it shows the custom ports are being used (15433 and 19042). It's still bootstrapping. Let me wait more: [bash] for i in {1..12}; do echo "Attempt $i..."; docker inspect test-cluster-yugabyte-1 --format='{{.State.Health.Status}}'; sleep 10; done Attempt 1... starting Attempt 2... starting Attempt 3... starting Attempt 4... starting Attempt 5... starting Attempt 6... starting Attempt 7... starting Attempt 8... starting Attempt 9... starting Attempt 10... starting Attempt 11... starting Attempt 12... starting

Twelve attempts, two minutes of waiting, and the same result every time: "starting." The container is alive but not healthy. It is stuck in a liminal state, neither fully operational nor crashed.

The Context: Why This Message Was Written

To understand why this message exists, we need to trace the chain of events that led to it. The session is part of building and debugging a test cluster for a distributed S3 storage system. The architecture involves three layers: an S3 frontend proxy, Kuri storage nodes, and a shared YugabyteDB metadata store. Earlier in the session, the assistant had been working with Docker bridge networking, but switched to host network mode in an attempt to improve performance. This decision—reasonable on its face—unleashed a cascade of problems.

Host network mode binds container ports directly to the host's network interfaces, bypassing Docker's NAT. This is faster, but it means containers compete with any existing services on the host for port allocations. When the assistant started the cluster in host mode, YugabyteDB failed its health checks. Investigation revealed that ports 7000 and 7100—critical internal ports for YugabyteDB's master process—were already in use by something on the host. YugabyteDB tried to bind to alternative ports, but the health check script was still looking at the default locations. The container was marked "unhealthy" not because YugabyteDB wasn't running, but because the health check couldn't find it where expected.

The user's response was succinct: "Change all YB ports" (message 1213). The assistant then embarked on a thorough reconfiguration effort, editing the Docker Compose file and the configuration generation script to use a completely new port scheme. The default YugabyteDB ports—5433 (YSQL), 9042 (YCQL), 7000 (master HTTP), 7100 (master RPC), 9000 (tserver HTTP), 9100 (tserver RPC), and 15433 (UI)—were all offset to avoid conflicts. The new ports were 15433 (UI), 19042 (YCQL), 17000 (master HTTP), 17100 (master RPC), 19000 (tserver HTTP), and 19100 (tserver RPC). The assistant updated the Docker Compose file, the gen-config.sh script, and the database initialization command to use these new values. Then it destroyed the old containers and started fresh.

Message 1249 is the moment of verification. The assistant has just started the new container and is checking whether the changes worked. The initial signs are promising: the custom ports are visible in the port listing. But the container's health status remains stubbornly stuck on "starting." The assistant decides to wait, running a twelve-attempt polling loop with ten-second intervals. This is the message we are analyzing.## The Reasoning Behind the Loop

The assistant's decision to run a twelve-attempt polling loop reveals several layers of reasoning. First, there is an assumption that the container is still bootstrapping and simply needs more time. YugabyteDB, especially on first startup, performs initialization tasks—creating data directories, starting master and tserver processes, building initial tablet replicas—that can take anywhere from thirty seconds to several minutes. The assistant had already seen the container report "starting" for a few checks before this loop, and the presence of the custom ports on the host suggested that the master process had started. The logical next step was to wait and see if the tserver would come online and the health check would pass.

Second, the loop structure itself is informative. The assistant chose a for loop with twelve iterations and ten-second sleeps, totaling two minutes. This is a pragmatic timeout: long enough to cover typical startup delays, but short enough that the developer won't wait forever if something is genuinely broken. The use of docker inspect to query the health status, rather than checking application-level readiness (e.g., trying to connect to YCQL or YSQL), indicates a reliance on Docker's built-in health check mechanism as the primary signal. This is a reasonable choice—Docker health checks are designed to encapsulate application readiness—but it also means the assistant is trusting the health check script's definition of "healthy." If that script is wrong, the signal is misleading.

Third, the assistant's opening comment—"Good progress - it shows the custom ports are being used (15433 and 19042)"—reveals an optimistic framing. The assistant interprets the visible ports as evidence that the reconfiguration worked, and interprets the "starting" status as a temporary state that will resolve. This is a classic debugging heuristic: when you make a change and see partial evidence of success, you assume the rest will follow. Sometimes it does; sometimes it doesn't.

Assumptions Embedded in the Message

Every debugging message carries assumptions, and this one carries several that are worth examining.

Assumption 1: The health check is correct. The assistant assumes that the Docker health check script is an accurate indicator of YugabyteDB readiness. But as we saw earlier in the session, the health check was failing because it was looking for YSQL on port 5433, which was no longer the default port after the reconfiguration. The assistant had updated the health check command in the Docker Compose file to use the new port (15433 for the UI, but the health check was for YSQL on 5433... actually, let me check). Looking back at the edits, the assistant changed the health check to use --cqlsh on port 19042. But the health check might still have been looking at the wrong endpoint, or the tserver might not have been ready yet. The point is: the assistant is treating "starting" as a transient state, but it could equally be a permanent stuck state caused by a misconfiguration.

Assumption 2: The container is making progress. The assistant assumes that the container is actively bootstrapping and will eventually become healthy. But "starting" in Docker's health check system doesn't necessarily mean progress is being made. It can also mean the health check hasn't run yet, or that the health check command is failing in a way that doesn't update the status to "unhealthy" yet. Docker's health check protocol has a grace period (the start_period) during which failures are not counted. If the container is stuck and the health check keeps failing, it will remain "starting" until the start period expires, then flip to "unhealthy." The assistant may not have accounted for this nuance.

Assumption 3: Two minutes is enough. The assistant chose a two-minute polling window based on experience with YugabyteDB startup times. But the reconfiguration changed the port layout, and it's possible that the new configuration introduced a subtle issue—a port conflict with another service, a firewall rule blocking the new ports, or a misconfigured flag—that prevents the tserver from starting at all. In that case, two minutes of waiting is wasted time.

Assumption 4: The custom ports are actually working. The assistant saw ports 15433 and 19042 in the ss output and concluded that the custom ports were being used. But seeing a port in a listening state doesn't mean the service on that port is functional. The master process might have started and bound to port 17000, but the tserver might be failing to start, leaving YCQL (port 19042) unavailable. The assistant's optimism is understandable, but it's based on an incomplete signal.

What the Message Reveals About the Debugging Process

This message is a snapshot of a developer in the middle of a debugging spiral. The assistant has made a significant configuration change (all YugabyteDB ports), restarted the container, and is now in the verification phase. The structure of the message—a brief optimistic comment followed by a polling loop—reveals a pattern that every developer knows: the uncomfortable gap between making a change and knowing whether it worked.

The assistant is also managing multiple threads of reasoning simultaneously. On one level, it's checking whether the port reconfiguration was applied correctly. On another, it's monitoring the container's health as a proxy for overall cluster readiness. On a third, it's managing the user's expectations by showing progress (or lack thereof) transparently. The polling loop is printed directly in the chat, giving the user visibility into what's happening. This transparency is a deliberate choice—it turns debugging into a shared activity rather than a black box.

Mistakes and Incorrect Assumptions

The most significant mistake in this message is the assumption that "starting" means "making progress." In Docker's health check system, a container can remain "starting" indefinitely if the health check command never succeeds but also never fails in a way that triggers the "unhealthy" transition. This is especially likely if the health check command itself has a bug—for example, if it's trying to connect to a port that doesn't exist, or if the command's exit code is being misinterpreted.

A second issue is the lack of a fallback diagnostic. The assistant is watching the health status but not simultaneously checking other signals: container logs, process listings inside the container, or network connectivity to the expected ports. If the assistant had run docker logs test-cluster-yugabyte-1 in parallel with the health checks, it might have seen error messages explaining why the tserver wasn't starting. Instead, the assistant is relying on a single binary signal (healthy/unhealthy/starting) that lacks diagnostic granularity.

A third, more subtle issue is the framing of "Good progress." This is a narrative the assistant is telling itself—and the user—to maintain momentum. But it risks creating a false sense of progress. The container is not healthy. The cluster is not operational. The assistant is waiting, not fixing. Sometimes waiting is the right move, but it's important to recognize that waiting is a diagnostic action, not a sign of progress.

Input Knowledge Required

To fully understand this message, the reader needs to know:

Docker health check mechanics: How Docker's HEALTHCHECK instruction works, the meaning of "starting," "healthy," and "unhealthy" states, and the role of start_period.
YugabyteDB architecture: The distinction between the master process (cluster metadata, tablet assignment) and the tserver process (data storage, query serving), and the ports each uses.
The session's history: The earlier decision to use host network mode, the discovery of port conflicts on 7000/7100, and the subsequent reconfiguration of all YugabyteDB ports.
The test cluster's architecture: The three-layer design with S3 proxy, Kuri storage nodes, and YugabyteDB, and the role of the database as the shared metadata store.
The docker inspect command: How to query container metadata, including health status, from the command line.

Output Knowledge Created

This message creates several pieces of knowledge:

Confirmation that the custom ports are in use: The ss output referenced in the opening line (from the previous message) showed ports 15433 and 19042 listening. This confirms that the --master_flags and --tserver_flags configuration worked at the process-binding level.
Evidence that the container is not healthy after two minutes: The twelve "starting" responses constitute a negative result. The container is not becoming healthy within the expected timeframe, which is useful diagnostic information.
A boundary on the startup time: The assistant has established that two minutes is insufficient for this configuration. This informs the next decision: wait longer, investigate further, or try a different approach.
A record of the debugging process: The message serves as documentation of what was tried and what was observed, which is valuable for the user (who is following along) and for future reference.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the structure of the message. The opening line—"Good progress - it shows the custom ports are being used (15433 and 19042)"—reveals the assistant's mental model: the port reconfiguration was the critical change, and seeing the new ports proves the change was applied. The phrase "It's still bootstrapping" shows the assistant interpreting the "starting" status through the lens of YugabyteDB's known startup behavior. The assistant is mapping the observed state ("starting") onto an expected sequence of events (master starts → tserver starts → health check passes).

The decision to wait twelve iterations with ten-second intervals reveals a calculated patience. The assistant could have waited indefinitely, or could have immediately started investigating other causes. The choice of two minutes suggests a heuristic: "If it's not healthy after two minutes of the new configuration, something else might be wrong." But the assistant doesn't articulate this heuristic explicitly—it's embedded in the loop parameters.

The fact that the assistant prints each attempt's result to the chat (rather than running silently and reporting only the final result) shows an awareness of the user's presence. The assistant is not just debugging; it's demonstrating the debugging process. This is a collaborative coding session, and transparency is part of the collaboration.

Conclusion

Message 1249 is a small moment in a long debugging session, but it captures something essential about the developer experience: the tension between hope and evidence, between waiting and acting. The assistant has made a change, seen partial confirmation, and is now waiting to see if the rest falls into place. It doesn't—the container remains stuck in "starting" limbo. But the message itself is valuable as a record of the reasoning, the assumptions, and the process at a specific point in time. It shows that even a seemingly trivial polling loop carries a wealth of embedded knowledge about the system, the tools, and the developer's mental model. In the end, the assistant will need to abandon the wait and look deeper—checking logs, testing connectivity, perhaps reverting to bridge networking entirely. But for two minutes, in this message, the assistant is simply watching and hoping. That is debugging in its most honest form.