The Port That Wouldn't Let Go: Debugging a Stuck YSQL in a Distributed S3 Test Cluster

Introduction

Building a horizontally scalable, S3-compatible storage system is a complex orchestration of many moving parts. When that system relies on a distributed database like YugabyteDB for metadata storage, even a seemingly simple task—getting the database container to start correctly—can become a deep debugging rabbit hole. This article examines a single, pivotal message from an opencode coding session where an engineer discovered why YugabyteDB's PostgreSQL-compatible interface (YSQL) was stuck while its Cassandra-compatible interface (YCQL) was running perfectly. The message captures a classic debugging moment: the transition from puzzlement to clarity, driven by the simple act of checking the logs.

The Message in Full

The subject message, written by the assistant (the AI coding agent), reads as follows:

[assistant] YSQL is stuck. This is odd - YCQL works but YSQL doesn't. Let me check the tserver logs:

>

[bash] docker exec test-cluster-yugabyte-1 cat /root/var/logs/tserver.err 2>&1 | tail -30

>

`` 2026-01-31 15:51:53.886 UTC [9259] LOG: pgaudit extension initialized 2026-01-31 15:51:53.887 UTC [9259] LOG: could not bind IPv4 address "127.0.0.1": Address already in use 2026-01-31 15:51:53.887 UTC [9259] HINT: Is another postmaster already running on port 15433? If not, wait a few seconds and retry. 2026-01-31 15:51:53.887 UTC [9259] WARNING: could not create listen socket for "127.0.0.1" 2026-01-31 15:51:53.887 UTC [9259] FATAL: could not create any TCP/IP sockets ``

At first glance, this is a short exchange—a statement, a command, and log output. But within these lines lies a rich story of assumptions tested, debugging methodology applied, and the kind of systems-level thinking that separates surface-level fixes from genuine understanding.

Context: The Road to This Moment

To appreciate why this message matters, we need to understand what came before it. The assistant had been building a test cluster for a horizontally scalable S3 architecture, part of a Filecoin Gateway project. The architecture followed a three-layer design: stateless S3 frontend proxies routing requests to Kuri storage nodes, which in turn stored metadata in a shared YugabyteDB cluster.

The immediate context involved a painful transition away from Docker host networking. Earlier in the session, the assistant had configured the YugabyteDB container to use host networking mode with custom ports (15433 for YSQL, 19042 for YCQL, and others for master and tserver RPC). This approach ran into port conflicts with existing services on the host machine. The assistant reverted to bridge networking, cleaned the YugabyteDB data directory thoroughly (using an Alpine container to remove root-owned files), and restarted fresh.

After the restart, the ports bound correctly—ss -tlnp showed 15433, 19042, 19000, and 19100 all listening. YCQL responded to queries immediately. But YSQL was unresponsive. The assistant ran multiple attempts with ysqlsh, each timing out. This was the puzzle: why would one interface work perfectly while its sibling was completely stuck?## The Reasoning Behind the Message

The assistant's opening statement—"YSQL is stuck. This is odd - YCQL works but YSQL doesn't"—reveals a critical reasoning pattern. The engineer is comparing two subsystems that share the same underlying infrastructure. Both YSQL (PostgreSQL-compatible) and YCQL (Cassandra-compatible) run on top of the same YugabyteDB tablet server process. They share the same data directories, the same master processes, and the same cluster metadata. If one is working and the other is not, the problem cannot be at the cluster or data layer. It must be specific to the PostgreSQL listener process.

This is a textbook application of differential diagnosis in systems debugging. By establishing that YCQL is functional, the assistant rules out a broad class of potential causes: network misconfiguration (the container can accept connections), database initialization failure (the keyspace listing worked), master process issues (the cluster is operational), and data directory corruption (YCQL reads from the same storage). The problem space narrows dramatically to something specific to the PostgreSQL-compatible frontend.

The decision to check the tserver logs is the natural next step. The assistant could have tried many other things: restarting the container, rebuilding the image, checking Docker networking, or even wiping the data directory again. But logs are the most direct source of truth. The assistant's choice reflects a debugging philosophy: before taking corrective action, gather diagnostic evidence. The command used—docker exec test-cluster-yugabyte-1 cat /root/var/logs/tserver.err 2>&1 | tail -30—is precise. It reads the error log file directly rather than relying on docker logs, which would show the yugabyted wrapper's output rather than the underlying PostgreSQL process's stderr.

The Discovery: "Address Already in Use"

The log output reveals the root cause with devastating clarity. The PostgreSQL postmaster process (the YSQL backend) attempts to bind to 127.0.0.1:15433 and receives an "Address already in use" error. The hint from PostgreSQL itself is almost comically helpful: "Is another postmaster already running on port 15433? If not, wait a few seconds and retry."

This is the smoking gun. But it also raises a deeper question: why is the port already in use? The assistant had just cleaned the data directory and started fresh. No other container should be using port 15433. The ss -tlnp output from earlier showed the port as listening, but that could be the new postmaster process itself—or it could be a zombie process from the previous container instance.

The most likely explanation is a race condition during container startup. When YugabyteDB starts, the yugabyted wrapper launches both the master and tserver processes. The tserver process in turn launches the PostgreSQL backend. If the data directory contains stale PostgreSQL state (like a postmaster.pid file), the new postmaster might incorrectly detect a running instance and refuse to start. Alternatively, the previous container's postmaster process might not have been fully terminated when Docker removed the container, leaving the port bound at the OS level.

Assumptions Made and Mistakes Revealed

This message exposes several assumptions that were either implicit or explicit in the debugging process.

Assumption 1: A clean data directory guarantees a clean start. The assistant had gone to great lengths to wipe /data/fgw2/yugabyte/* using an Alpine container with rm -rf. But the data directory might still contain files that were recreated by Docker volume mounts or that existed in subdirectories not fully cleaned. The PostgreSQL pg_data subdirectory, in particular, can contain lock files and control data that persist across container restarts if not explicitly removed.

Assumption 2: Port binding success at the OS level means the service is ready. When ss -tlnp showed port 15433 as listening, the assistant reasonably concluded that YSQL was available. But the LISTEN state in ss output only indicates that a socket bind succeeded at some point. It doesn't guarantee that the process that bound the port is healthy, or that it's the correct process. In this case, the port might have been bound by a previous postmaster instance that was orphaned during the container restart cycle.

Assumption 3: YCQL and YSQL share the same startup fate. Because both interfaces run on the same tserver, the assistant assumed that if YCQL was up, YSQL would be up too. The logs prove this assumption wrong: the tserver process itself was running (YCQL worked), but the PostgreSQL child process failed to start due to the port conflict. The two subsystems have different startup sequences and different failure modes.

Assumption 4: The container health check is reliable. The assistant had been polling docker inspect for the health status, which showed "starting" for an extended period. The health check likely tests YSQL connectivity, which was failing, so the container never transitioned to "healthy." This was actually a correct signal—the container was indeed not fully operational—but the assistant initially interpreted the long "starting" state as normal bootstrapping rather than a stuck condition.

Input Knowledge Required

To fully understand this message, a reader needs knowledge across several domains:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. A confirmed failure mode: YSQL can fail independently of YCQL even on a healthy tserver, specifically due to port binding conflicts on the PostgreSQL listener.
  2. A diagnostic procedure: When YSQL is stuck but YCQL works, check the tserver error logs at /root/var/logs/tserver.err for postmaster startup failures.
  3. A remediation path: The fix likely involves either killing the orphaned postmaster process, clearing the PostgreSQL data directory (pg_data), or waiting for the port to be released by the OS.
  4. A lesson in container lifecycle: Docker container removal does not guarantee that all child processes have released their ports, especially when using host networking or when processes are killed abruptly.

The Thinking Process in Action

The assistant's thinking process is visible in the structure of the message itself. The opening line—"YSQL is stuck. This is odd - YCQL works but YSQL doesn't"—is not just a statement of fact. It is a hypothesis framed as an observation. The word "odd" signals that the assistant recognizes a pattern violation: in a well-functioning system, both interfaces should be available together. The asymmetry is the clue.

The decision to check tserver.err rather than tserver.out or yugabyted.log shows an understanding of where PostgreSQL-specific errors would appear. The yugabyted wrapper logs general startup progress, but PostgreSQL's own stderr is the authoritative source for postmaster failures. The assistant chose the right log file for the right subsystem.

The tail -30 choice is also deliberate. The assistant could have read the entire file, but the most recent entries are most likely to contain the startup failure. The first few lines of the output confirm this: the log shows the postmaster initializing the pgaudit extension, then immediately failing on the bind call. The timestamps (15:51:53) confirm this is from the current startup attempt, not stale data.

Broader Implications

This message, while brief, illustrates a fundamental truth about distributed systems debugging: symmetry is a powerful diagnostic tool. When two components share the same infrastructure but behave differently, the difference itself points to the root cause. The assistant could have spent hours investigating network configuration, Docker Compose settings, or YugabyteDB cluster health. Instead, the simple observation that YCQL worked while YSQL didn't narrowed the search space to the PostgreSQL listener in seconds.

The message also highlights the importance of reading raw logs rather than relying on aggregated health checks. The Docker health check was stuck on "starting" for minutes, providing no useful diagnostic information. The tserver.err file, by contrast, gave an immediate, specific error with a hint. In complex systems, the most valuable debugging tool is often the simplest: read the logs.

Conclusion

The assistant's message at index 1268 is a masterclass in focused debugging. Faced with a puzzling asymmetry between a working YCQL and a stuck YSQL, the assistant applied differential diagnosis, chose the right log file, and discovered the root cause in seconds: a PostgreSQL postmaster process that couldn't bind to its port because the address was already in use. The message captures the moment of transition from confusion to understanding, driven by a methodical approach and a deep understanding of the system's architecture. It serves as a reminder that in distributed systems, the most effective debugging often comes not from trying random fixes, but from asking the right questions and looking in the right places.