The Port That Wasn't Free: A Case Study in YugabyteDB Port Conflict Debugging

Introduction

In the course of building a horizontally scalable S3 test cluster, an assistant encountered a perplexing problem: YugabyteDB's YSQL interface (the PostgreSQL-compatible query layer) refused to start, hanging indefinitely despite all other services working correctly. The root cause, discovered after a lengthy debugging session spanning dozens of commands and over thirty minutes of wall-clock time, was a simple but easily overlooked port conflict. The YugabyteDB web UI and the YSQL database port were both configured to use the same TCP port—15433—and the web UI won the race to bind to it. This article examines message 1271, the moment of discovery, and unpacks the reasoning, assumptions, and debugging process that led to this insight.

The Subject Message

The target message reads:

YugabyteDB UI uses 15433 by default! The webserver is on 15433. I picked that port for YSQL but YugabyteDB is using it for their web UI. Let me change the YSQL port to something else like 25433: [read] /home/theuser/gw/test-cluster/docker-compose.yml <file> 00001| # Test cluster with proper scalable S3 architecture 00002| # Usage: FGW_DATA_DIR=/data/fgw-test docker-compose up -d 00003| # 00004| # Architecture: 00005| # Client → S3 Frontend Proxy → Kuri Storage Nodes 00006| # 00007| # Port allocation (host network mode - all services bind directly to host): 00008| # 8078 - S3 API (Frontend Proxy) 00009| # 8079 - kuri-1 S3 API (internal) 00010| # 8081 - kuri-2 S3 API (internal) -- avoid 8080 conflict 00011| # 7003 - kuri-1 Local...

This is a short message—barely a paragraph—but it represents the culmination of a significant debugging effort. The assistant's tone conveys a moment of realization: the exclamation mark after "by default!" signals genuine surprise. The assistant had been operating under an incorrect assumption for the entire debugging session, and this message is where that assumption finally breaks.

The Context: A Long Debugging Session

To understand why this message matters, one must appreciate the debugging journey that preceded it. The assistant had been trying to start a YugabyteDB container with custom ports for the test cluster. The cluster uses host network mode, meaning containers bind directly to ports on the host machine rather than going through Docker's bridge networking. This approach was chosen to avoid port mapping complexity, but it also means that any port conflict on the host immediately breaks the affected service.

The debugging session began with the assistant noticing that YugabyteDB was not starting properly. After cleaning stale data directories, the assistant successfully started YugabyteDB and observed that all four custom ports appeared to be listening: 15433 (YSQL), 19042 (YCQL), 19000 (tserver), and 19100 (master). However, YSQL connections kept timing out. The assistant tried multiple approaches: waiting longer, checking process lists, verifying that the PostgreSQL backend had started, and even confirming that YCQL (the Cassandra-compatible interface) worked perfectly on port 19042. The asymmetry was baffling—why would one query interface work while the other failed?

The assistant then checked the tserver error logs and found the critical clue: "could not bind IPv4 address '127.0.0.1': Address already in use" on port 15433. Something else was already occupying that port. A quick check with ss -tlnp | grep 15433 confirmed that a process was listening on 15433, but the tool output didn't show which process owned it. This is where message 1271 enters: the assistant finally connected the dots.

The Reasoning Process

The assistant's reasoning in this message reveals a multi-step inference chain:

  1. Observation: Port 15433 is in use, preventing YSQL from binding.
  2. Domain knowledge: YugabyteDB's yugabyted process includes a built-in web UI for cluster management.
  3. Recall: The default port for the YugabyteDB web UI is 15433.
  4. Connection: The assistant had chosen 15433 as the YSQL port without realizing it collided with the UI's default port.
  5. Resolution: Change the YSQL port to something else (25433 was chosen) to avoid the conflict. The key insight is that the assistant had to know two things simultaneously: (a) that YugabyteDB has a web UI, and (b) that this UI defaults to port 15433. Without both pieces of knowledge, the port conflict would have remained mysterious. The assistant's debugging approach—checking logs, verifying process ownership, and cross-referencing with known YugabyteDB defaults—is a textbook example of systematic troubleshooting.

Assumptions Made and Mistakes Corrected

This message reveals several assumptions that were in play, some correct and some incorrect:

Incorrect assumption: The assistant assumed that port 15433 was available for YSQL because nothing else was explicitly configured to use it. The assistant had chosen 15433 as a "random" port that seemed unlikely to conflict with common services. However, this assumption failed because YugabyteDB's own web UI claims that port by default, and the assistant was unaware of this default.

Correct assumption: The assistant correctly assumed that the port conflict would be visible in the tserver error logs. This assumption guided the debugging strategy: when YSQL kept timing out, the assistant went straight to the logs rather than guessing at more exotic causes.

Implicit assumption: The assistant assumed that the YugabyteDB container was properly configured and that the issue was environmental rather than a configuration bug. This turned out to be correct—the Docker Compose configuration was sound, but the port choice was unlucky.

The mistake here is subtle and forgivable. YugabyteDB's documentation lists 15433 as the default for the yugabyted UI, but this is not a well-known fact. Many developers would assume that a database's web UI would use a port in the 8000-9000 range (like 8080 or 9000) rather than the 15000 range. The assistant's choice of 15433 was reasonable—it's far enough from common ports to avoid most conflicts—but it happened to collide with YugabyteDB's own internal default.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. YugabyteDB architecture knowledge: Understanding that YugabyteDB has multiple interfaces—YSQL (PostgreSQL-compatible), YCQL (Cassandra-compatible), and a web UI—each potentially on different ports.
  2. Docker networking basics: Knowing that host network mode means containers bind directly to host ports, making port conflicts a real possibility.
  3. Debugging methodology: Appreciating the value of checking error logs when a service fails to start, and understanding that "Address already in use" is a definitive diagnostic message.
  4. Port allocation conventions: Recognizing that 15433 is an unusual port for a web UI, and that the assistant's choice was not obviously wrong.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

  1. A specific configuration fix: The YSQL port must be changed from 15433 to a non-conflicting port (25433 was chosen).
  2. A debugging lesson: When YugabyteDB's YSQL fails to start but YCQL works, check for port conflicts on the YSQL port, especially with the YugabyteDB web UI.
  3. A documentation insight: The YugabyteDB web UI default port (15433) is not widely advertised and can easily be overlooked when configuring custom ports.
  4. A general principle: When configuring services with multiple sub-services (like YugabyteDB's master, tserver, YSQL, YCQL, and UI components), always verify that your chosen ports don't conflict with the service's own internal defaults.

The Broader Debugging Narrative

Message 1271 sits at a critical juncture in the debugging narrative. The assistant had been stuck for many iterations, trying to diagnose why YSQL wouldn't start. The problem was intermittent in appearance—the port was listening, so it seemed like the service was up, but connections would hang. This is a particularly insidious class of bug: the port is open, so network tools report success, but the wrong process owns it, so database connections fail.

The assistant's debugging approach was methodical. First, basic checks (is the container running? are ports listening?). Then, deeper investigation (check process lists, try connecting). Finally, log analysis (read tserver.err). This progression from simple to complex is standard practice, but it's worth noting how long it took—over 30 minutes of real time—because each step required waiting for commands to execute, interpreting output, and deciding the next action.

The "aha" moment in message 1271 is satisfying precisely because it explains all the symptoms at once: YCQL works because it's on a different port (19042) that doesn't conflict; the port 15433 shows as "LISTEN" because the web UI grabbed it; YSQL connections hang because the PostgreSQL process can't bind and keeps retrying. Every piece of evidence fits the port conflict hypothesis perfectly.

Conclusion

Message 1271 is a small but illuminating example of how distributed system debugging works in practice. The assistant's realization that YugabyteDB's web UI defaults to port 15433—the same port chosen for YSQL—resolved a frustrating debugging session that had consumed significant time and effort. The message demonstrates the importance of understanding a system's default port allocations, the value of systematic log analysis, and the danger of assuming that an unused-looking port is actually available. For anyone working with YugabyteDB or similar multi-service databases, this port conflict is a valuable lesson: always check your service's own defaults before assigning custom ports, because the service you're configuring might already be using the port you picked.