The Port That Wasn't Free: A Debugging Epiphany in Distributed Systems

"The issue is clear — something else is using port 15433 on the host!"

Some debugging moments feel like a detective finally spotting the clue that was hidden in plain sight. Message 1269 in this coding session captures exactly such a moment. After nearly twenty minutes of head-scratching, repeated timeouts, and false leads, the assistant suddenly connects the dots. The message is deceptively simple — a single line of reasoning followed by a shell command — but it represents the climax of a debugging arc that reveals how distributed systems testing can go wrong in subtle, instructive ways.

The Full Message

The subject message, verbatim, reads:

[assistant] The issue is clear - something else is using port 15433 on the host! Let me check:
[bash] ss -tlnp | grep 15433
LISTEN 0      4096                     127.0.0.1:15433      0.0.0.0:*

The Debugging Journey That Preceded It

To understand why this message matters, we must reconstruct the debugging spiral that led to it. The assistant had been building a horizontally scalable S3-compatible storage cluster using a three-layer architecture: an S3 frontend proxy, Kuri storage nodes, and a shared YugabyteDB metadata database. After a series of configuration changes — including switching from host networking to bridge networking and back — the assistant had cleaned the YugabyteDB data directory and started fresh.

In message 1259, everything looked promising. A ss -tlnp check showed all four expected ports listening on the host: 15433 (YSQL), 19042 (YCQL), 19000 (master HTTP), and 19100 (tserver HTTP). The assistant declared, "All ports are now bound correctly." But this was a trap. The ports were listening on the host, but the YugabyteDB container's health status remained stuck at "starting."

Then came the first sign of trouble. In message 1261, the assistant tried to connect to YSQL:

docker exec test-cluster-yugabyte-1 bin/ysqlsh -h 127.0.0.1 -p 15433 -U yugabyte -d yugabyte -c 'select 1'

The command timed out after 120 seconds. The assistant tried again with a loop, adding 15-second pauses between attempts. Same result: timeout. Yet YCQL — the Cassandra-compatible API — worked perfectly on port 19042 (message 1266). The asymmetry was baffling. Both APIs run inside the same YugabyteDB container. Why would one work and the other hang?

The assistant checked process listings and found that both the yb-master and postgres processes were running inside the container (messages 1264–1265). The postgres process was even configured to listen on 127.0.0.1:15433. Everything looked correct from the inside. So why was the connection hanging?

The Critical Clue

The breakthrough came in message 1268, when the assistant finally checked the tserver error logs:

2026-01-31 15:51:53.887 UTC [9259] LOG:  could not bind IPv4 address "127.0.0.1": Address already in use
2026-01-31 15:51:53.887 UTC [9259] HINT:  Is another postmaster already running on port 15433? If not, wait a few seconds and retry.
2026-01-31 15:51:53.887 UTC [9259] WARNING:  could not create listen socket for "127.0.0.1"
2026-01-31 15:51:53.887 UTC [9259] FATAL:  could not create any TCP/IP sockets

There it was. The postgres process inside the container was failing to bind to port 15433 because something else had already claimed it. But the assistant had confirmed that port 15433 was listening on the host — and it was. The question was: what was listening?

The Epiphany in Message 1269

This is where message 1269 becomes the turning point. The assistant writes: "The issue is clear — something else is using port 15433 on the host!" The tone is one of sudden realization. The ss -tlnp | grep 15433 command confirms the port is in use, but critically, the -p flag (which shows the process identifier) reveals which process owns it. The output shows LISTEN 0 4096 127.0.0.1:15433 0.0.0.0:* — but the process information is truncated or not shown in the captured output.

What the assistant realized in this moment, and what becomes explicit in message 1271, is that YugabyteDB's own web UI defaults to port 15433. The yugabyted process starts a web-based administration interface on that port. When the assistant configured the YSQL port to 15433, they inadvertently created a conflict: the YugabyteDB UI grabbed port 15433 first, and when the postgres-based YSQL subsystem tried to bind the same port, it found the address already in use and crashed fatally.

This is a beautifully subtle bug. The port was listening — the ss command showed it. The YugabyteDB container appeared healthy in terms of port binding. But the port belonged to the wrong process inside the container. The YSQL listener never got a chance to start.

Assumptions and Mistakes

Several assumptions contributed to this debugging difficulty:

The "all ports are bound" assumption. When the assistant saw all four ports listening in message 1259, they assumed the system was healthy. The mistake was equating "port is open" with "service is ready." In reality, port 15433 was open but owned by the web UI, not by YSQL.

The symmetry assumption. Because YCQL worked on port 19042, the assistant assumed YSQL should work similarly on port 15433. But YCQL and YSQL have completely different implementations inside YugabyteDB — YCQL uses the tablet server's native Cassandra-compatible protocol, while YSQL runs a full postgres process. They fail independently.

The container transparency assumption. The assistant was running commands via docker exec, which executes inside the container's namespace. But the port conflict was also inside the container — the YugabyteDB UI and postgres were competing for the same port within the same container. The host's ss command showed port 15433 as bound because the container's port mapping exposed it, but it couldn't distinguish which internal process owned it without deeper inspection.

Input Knowledge Required

To understand this message, one needs:

YugabyteDB architecture knowledge: Understanding that YugabyteDB has multiple subsystems (master, tserver, YSQL/postgres, YCQL, web UI) that each bind to different ports, and that the yugabyted orchestration layer starts them in a specific order.
Docker networking basics: Knowing that docker exec runs inside the container's network namespace, and that port mapping (-p or ports: in compose) exposes container ports on the host.
Linux socket debugging: Familiarity with ss -tlnp (socket statistics with listening, TCP, numeric, processes) and the ability to interpret its output.
Postgres error interpretation: Recognizing that "could not bind IPv4 address: Address already in use" means a port conflict, and "FATAL: could not create any TCP/IP sockets" means the process cannot recover.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

A confirmed root cause: The YSQL port conflict is diagnosed, ending the "why is YSQL hanging?" mystery.
A corrective action path: The assistant immediately follows up (message 1271) by reading the docker-compose.yml and changing the YSQL port from 15433 to 25433, avoiding the YugabyteDB UI's default port.
A reusable debugging pattern: The sequence of checking process listings, then error logs, then port ownership demonstrates a methodical approach to diagnosing containerized service failures.

The Thinking Process

The reasoning visible in the surrounding messages shows a classic debugging arc:

Observation: YSQL connections hang indefinitely; YCQL works fine.
Hypothesis generation: Maybe YSQL isn't ready yet? (messages 1262–1263: waiting longer, checking health status)
Data collection: Check if postgres process is running (message 1265: yes, it is)
Refined hypothesis: Maybe postgres started but isn't accepting connections?
Log analysis: Check tserver.err (message 1268: "Address already in use")
Synthesis: The error means something else has port 15433. But what? The YugabyteDB UI!
Confirmation: ss -tlnp | grep 15433 (message 1269) confirms the port is occupied. The moment captured in message 1269 is the synthesis step — the instant when scattered observations coalesce into a coherent explanation. The assistant doesn't just say "there's an error"; they say "The issue is clear," signaling a gestalt shift in understanding. Everything that was confusing now makes sense.

Broader Implications

This debugging episode illustrates a fundamental challenge in distributed systems testing: port conflicts are often invisible until runtime. A configuration file can specify port 15433 for YSQL, and everything will look correct in static analysis. But the actual runtime behavior depends on the order in which processes start, the default ports of auxiliary services, and the namespace isolation provided by containers.

The fix — changing YSQL to port 25433 — is trivial. But the diagnosis required understanding the full YugabyteDB process model, reading error logs inside the container, and connecting the dots between a postgres bind failure and a web UI default port. It's the kind of bug that can waste hours if approached unsystematically, and minutes if approached with the right diagnostic sequence.

Message 1269, for all its brevity, captures the essence of systems debugging: the moment of clarity when the fog lifts and the root cause stands revealed.