The Two-Minute Wait That Revealed a Deeper Problem: Debugging a Stuck YugabyteDB Container

Introduction

In distributed systems debugging, the most frustrating moments often come not from clear error messages, but from silence—a container that refuses to become healthy, a process that hangs indefinitely, a status that remains "starting" long after it should have settled. Message 1250 in this coding session captures exactly such a moment. The assistant, having just reconfigured an entire YugabyteDB test cluster to use non-standard ports to avoid host network conflicts, is waiting for the database to finish bootstrapping. Two minutes of polling have passed. The container is still not healthy. And in this single, deceptively simple message, the assistant performs a critical diagnostic check that reveals the true nature of the problem.

The Message

[assistant] Still starting - let me check if tserver came up:
[bash] ss -tlnp 2>/dev/null | grep -E '(15433|19042|19000|19100)' && docker logs test-cluster-yugabyte-1 2>&1 | tail -5
LISTEN 0      4096                     127.0.0.1:15433      0.0.0.0:*                                                 
Starting yugabyted...
 
/ Starting the YugabyteDB Processes...✅ YugabyteDB Started                  
 
/ Bringing up UI...✅ UI ready         
 
/ Checking YSQL Status...- Checking YSQL Status...\ Checking YSQL Status...| Checking YSQL Status.../ Checking YSQL Status...- Checking YSQL Status...\ Checking YSQL Status...| Checking YSQL Status.../ Checking YSQL Status...- Checking YSQL Status...\ Checking YSQL Status...| Checking YSQL Status.../ Checking YSQL Status...- Checking YSQL Status...\ Checking YSQL Status...| Checking YSQL Status.../ Checking YSQL Status...- Checking YSQL...

At first glance, this looks like a routine status check. But in the context of the debugging session, it is a pivotal moment of discovery.

The Context: A Cascade of Port Conflicts

To understand why this message matters, we need to step back. The assistant had been building a horizontally scalable S3 test cluster using YugabyteDB as the shared metadata store. The architecture involved Docker containers running with host networking mode—a decision made earlier to maximize I/O performance for load testing. However, host networking means containers bind directly to the host's network interfaces, and the host already had services occupying ports 7000 and 7100—ports that YugabyteDB uses by default for its master processes.

The user's instruction was succinct: "Change all YB ports." The assistant responded by offsetting every YugabyteDB port: YSQL moved from 5433 to 15433, YCQL from 9042 to 19042, master HTTP from 7000 to 17000, master RPC from 7100 to 17100, tserver HTTP from 9000 to 19000, and tserver RPC from 9100 to 19100. This required updating the Docker Compose file with the correct --master_flags and --tserver_flags syntax, a process that involved consulting the YugabyteDB documentation to find the right CLI flags.

After these changes, the assistant restarted the container and began waiting. Message 1249 shows a loop that polled the health status twelve times over two minutes—every ten seconds, the same result: "starting." The container was alive but not ready.

What This Message Actually Reveals

The assistant's diagnostic approach in message 1250 is methodical. Rather than simply waiting longer, it checks two things simultaneously: which ports are actually listening on the host, and what the container's logs show.

The ss -tlnp command lists TCP sockets in a listening state. The grep pattern targets the four key YugabyteDB ports: 15433 (YSQL), 19042 (YCQL), 19000 (tserver HTTP), and 19100 (tserver RPC). The output is stark: only port 15433 is listening. The YCQL port (19042) is absent. The tserver ports (19000, 19100) are absent.

This is a significant finding. YugabyteDB's architecture separates the YSQL (PostgreSQL-compatible) layer from the YCQL (Cassandra-compatible) layer and the tablet server (tserver) processes. The fact that only YSQL is up while YCQL and the tserver are missing indicates that the database is only partially initialized. The master processes (ports 17000 and 17100) are running—as confirmed by earlier checks—but the tablet server, which actually hosts the data, has not started.

The docker logs confirm the symptom: the container is stuck in the "Checking YSQL Status..." loop. YugabyteDB's yugabyted start command goes through a sequence: starting processes, bringing up the UI, then checking YSQL status. The logs show the UI is ready and YSQL status checks are repeating, but the process never progresses to the next stage.

The Assumption That Nearly Misled

At this point, the assistant is operating under a reasonable assumption: the container is still bootstrapping and needs more time. The custom port configuration might be causing a slower startup, or the tserver might need additional initialization steps. The assistant has already waited two minutes and is preparing to wait longer.

But this assumption is subtly wrong. The real problem is not that the container needs more time—it's that the YugabyteDB data directory contains stale configuration from previous runs. When YugabyteDB starts with a non-empty data directory, it reuses the existing configuration, including the old port settings. The assistant had changed the Docker Compose file to pass new port flags, but the data directory at /data/fgw2/yugabyte/ still held the original configuration files, owned by root and inaccessible to a regular rm -rf.

This is a classic distributed systems debugging trap: the symptom (slow startup) points toward one cause (need more time), but the root cause (stale data) is invisible from the logs. The assistant will discover this in subsequent messages when examining the tserver error logs and finding that the tserver is actually listening on a Docker bridge network IP (172.22.0.2:5433) rather than the expected host network ports—a clear sign that the old configuration is still in effect.

Input Knowledge Required

To fully understand this message, the reader needs several pieces of knowledge:

First, an understanding of YugabyteDB's process architecture: the master processes handle metadata and cluster coordination, the tserver processes handle actual data storage and query execution, YSQL provides PostgreSQL-compatible SQL access, and YCQL provides Cassandra-compatible CQL access. Each of these listens on separate ports.

Second, familiarity with Docker networking modes: host networking means the container shares the host's network stack, so ports bound inside the container appear directly on the host. Bridge networking, by contrast, creates an isolated network namespace with port mapping.

Third, knowledge of the ss command and its output format: ss -tlnp shows TCP listening sockets with process information. The output format includes the local address and port, the peer address, and the process name and PID.

Fourth, an understanding of the docker logs output format: the spinning characters (/, -, \, |) indicate ongoing activity, and the repeated "Checking YSQL Status..." lines show that the startup script is stuck in a loop.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. The YSQL port (15433) is listening. This confirms that the master processes started successfully and the PostgreSQL-compatible interface is available.
  2. The YCQL port (19042) is not listening. This means the Cassandra-compatible interface is not yet available, which is critical because the S3 proxy and Kuri storage nodes use YCQL for metadata operations.
  3. The tserver ports (19000, 19100) are not listening. This means the tablet server has not started, which means no data operations are possible.
  4. The container logs show no errors. The startup script is proceeding normally but stuck in the YSQL status check loop. This rules out immediate crash-loop or configuration syntax errors.
  5. The health check remains "starting." Docker's health check mechanism has not yet marked the container as healthy, which means dependent services (db-init, kuri nodes) will not start.

The Thinking Process

The assistant's reasoning in this message follows a clear diagnostic pattern. The first line—"Still starting - let me check if tserver came up"—reveals the hypothesis being tested. The assistant suspects that the tserver might be the bottleneck, and checking whether its ports are listening will confirm or refute this.

The choice to run both ss and docker logs in a single command is deliberate. The port check tells the assistant what is happening (which components are running), while the logs tell how the startup is progressing. Together, they provide a complete picture of the container's state.

The grep pattern is carefully chosen to include all four YugabyteDB ports that matter for the test cluster's operation. Notably, the master ports (17000, 17100) are not included in this check—the assistant already confirmed they were listening in message 1244. The focus is on the components that are still missing.

The && between the two commands ensures that the logs are only fetched if the port check succeeds. This is a pragmatic choice: if the container had crashed entirely, the port check would fail and the logs would still be available from a separate command.

Why This Message Matters

In the broader narrative of the debugging session, message 1250 is the moment when the assistant shifts from passive waiting to active investigation. The two-minute polling loop in message 1249 was a patience test. Message 1250 is the decision to look deeper.

The discovery that only YSQL is up while YCQL and tserver are missing will lead, in subsequent messages, to examining the tserver error logs, finding the stale data directory, cleaning it with an Alpine container (since the files are owned by root), and finally restarting with a clean state. After that cleanup, all four ports come up correctly within 30 seconds.

The lesson is one that every infrastructure engineer learns eventually: when a database container stays in "starting" state for an unusually long time, the problem is rarely that it needs more time. It's almost always something else—stale data, configuration drift, resource constraints, or a race condition. Message 1250 captures the moment of recognizing that "wait longer" is not a strategy, and that real debugging requires looking at the actual component state, not just the aggregate health check.