The Port That Wasn't There: A Debugging Epiphany in Distributed Systems

The Message

[assistant] Interesting - YugabyteDB is binding to port `15433` instead of `5433`. This might be a host network issue where port 5433 is already in use. Let me check:
[bash] ss -tlnp | grep 5433
LISTEN 0      4096                     127.0.0.1:15433      0.0.0.1:*                                                 

At first glance, this appears to be a routine diagnostic exchange — an assistant noticing a port mismatch and running a command to investigate. But this single message represents a critical turning point in a much larger debugging session, one that reveals deep truths about distributed systems, Docker networking, and the assumptions we carry when building complex infrastructure.

Context: The Road to Host Networking

To understand why this message was written, we must first understand the journey that led to it. The assistant and user had been building a horizontally scalable S3 architecture — a three-layer system consisting of stateless S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB metadata store. They had recently implemented a CQL batcher to improve write throughput and were running load tests to validate performance.

The load tests revealed a troubling pattern: at 100 and 1000 concurrent workers, the system produced hundreds of "corruption" errors. Upon closer inspection, these were not actual data corruption but connection reset by peer errors — the Docker userland proxy was becoming a bottleneck under high concurrency. The assistant summarized this clearly: "The 'corruption' is actually connection failures — the verify read errors show read: connection reset by peer, not actual data mismatches."

The user's response was decisive: "Rewrite the test-cluster to use host network." This was a reasonable architectural choice. Host networking bypasses Docker's port mapping layer entirely, allowing containers to bind directly to host ports. For a performance-sensitive distributed system undergoing load testing, this should eliminate the proxy bottleneck and reveal the true throughput ceiling of the storage nodes.

The assistant executed this rewrite across multiple files: docker-compose.yml, gen-config.sh, and README.md. Port assignments were carefully reorganized to avoid conflicts within the cluster itself — kuri-1 would use S3 port 8079, kuri-2 would use 8080, the proxy would use 8078, and so on. The cluster was stopped, configuration was regenerated, and the new host-network cluster was started.

And then YugabyteDB failed its health check.

The Moment of Discovery

This brings us to the subject message. The assistant had just observed that YugabyteDB was unhealthy. The health check logs showed ysqlsh: could not connect to server: Connection refused when trying to reach 127.0.0.1:5433. Something was wrong with the database container.

The assistant's first instinct was to check what port YugabyteDB was actually binding to. The command ss -tlnp | grep 5433 was intended to find a process listening on port 5433 — the standard PostgreSQL-compatible port that YugabyteDB uses for YSQL connections. But the result was unexpected: the grep returned 127.0.0.1:15433, not 127.0.0.1:5433.

This is the critical insight captured in the message. The assistant recognized immediately that port 15433 was YugabyteDB's alternate port — YugabyteDB's web UI and some internal services use ports in the 15000 range. The fact that ss -tlnp | grep 5433 returned a line matching 15433 (because the substring "5433" appears within "15433") was a red flag: port 5433 was simply not in use by anything. YugabyteDB had not bound to its expected port.

The assistant's reasoning, visible in the phrase "This might be a host network issue where port 5433 is already in use," reveals a working hypothesis: that host network mode had exposed a port conflict. In bridge networking mode, Docker containers have their own network namespace, so port 5433 inside the container is isolated from port 5433 on the host. But in host networking mode, the container shares the host's network stack. If something on the host was already using port 5433, YugabyteDB would fail to bind and either crash or fall back to an alternative port.

Assumptions Under the Microscope

This message exposes several assumptions that were made, some of which turned out to be incorrect or incomplete.

Assumption 1: Host networking would be a drop-in replacement. The assistant assumed that switching from bridge to host networking would be a straightforward configuration change — just remove the ports: and networks: sections, add network_mode: host, and everything would work identically but faster. This assumption ignored the fundamental difference in network isolation. Bridge networking creates a virtual network environment where container ports are mapped to host ports through Docker's proxy. Host networking eliminates that proxy but also eliminates the isolation, making the container vulnerable to port conflicts with any service running on the host.

Assumption 2: The port assignments were conflict-free. The assistant had carefully assigned distinct ports for each service within the cluster: 8078 for the S3 proxy, 8079 for kuri-1, 8080 for kuri-2, 7001 and 7002 for LocalWeb endpoints, and so on. But the assumption was that these ports were globally available on the host. In reality, the host machine had existing services — later discovered to be using ports 7000 and 7100 — that conflicted with YugabyteDB's internal ports. The assistant had not audited the host's existing port usage before making the switch.

Assumption 3: The grep command would find the right thing. The command ss -tlnp | grep 5433 was a reasonable diagnostic step, but it contained a subtle flaw: it would match any line containing the substring "5433", including port 15433. The assistant correctly interpreted this result, but a more precise grep (using grep ':5433 ' or grep -w 5433) would have avoided the ambiguity. This is a common debugging pitfall — a command that works 99% of the time can produce misleading results in edge cases.

Assumption 4: YugabyteDB's port configuration was correct. The assistant had not explicitly configured YugabyteDB's ports in the Docker Compose file for host networking. The YugabyteDB container was using its default port assignments, which include 5433 for YSQL, 9042 for YCQL, 7000 and 7100 for master processes, and 9000 and 9100 for tserver processes. In bridge mode, these defaults were fine because they were isolated within the container's network namespace. In host mode, they became liabilities.

Input Knowledge Required

To understand this message, a reader needs knowledge in several domains:

Docker networking modes: Understanding the difference between bridge networking (default, with port mapping through a proxy) and host networking (direct binding to host interfaces) is essential. The entire debugging effort hinges on this distinction.

YugabyteDB architecture: Knowledge that YugabyteDB uses multiple ports for different purposes — YSQL (PostgreSQL-compatible) on 5433, YCQL (Cassandra-compatible) on 9042, master HTTP on 7000, master RPC on 7100, tserver HTTP on 9000, tserver RPC on 9100, and web UI on 15433. The assistant's recognition that 15433 was an alternate port required familiarity with YugabyteDB's port layout.

Linux socket monitoring: The ss command (socket statistics) and its flags (-t for TCP, -l for listening sockets, -n for numeric addresses, -p for process info) are standard Linux networking tools. The grep pattern matching behavior — that grep 5433 matches any line containing "5433" anywhere, including within "15433" — is a basic but easily overlooked detail.

Distributed systems testing patterns: Understanding why one would switch from bridge to host networking — to eliminate the Docker proxy bottleneck during load testing — requires knowledge of how container networking affects performance under high concurrency.

Output Knowledge Created

This message produced several valuable pieces of knowledge:

Diagnostic evidence: The output of ss -tlnp | grep 5433 provided concrete evidence that port 5433 was not bound by YugabyteDB. The presence of port 15433 (matching due to substring) was a clue that something had gone wrong with the database's port binding.

A confirmed hypothesis direction: The assistant's suspicion that "port 5433 is already in use" set the direction for subsequent investigation. The next messages (1208-1212) would confirm that ports 7000 and 7100 were indeed occupied by existing host services, validating the hypothesis that host networking had caused port conflicts.

A decision point: This message represents the moment where the assistant realized the host networking approach had a fundamental problem. The subsequent decision — to investigate further, discover the conflicting ports, and ultimately revert to bridge networking or reconfigure ports — was shaped by this initial finding.

The Thinking Process

The assistant's thinking process, visible in the message's structure, follows a classic debugging pattern:

1. Observe symptom: YugabyteDB is unhealthy; it can't connect to port 5433.
2. Form hypothesis: Maybe port 5433 is already in use on the host (due to host networking).
3. Design experiment: Run ss -tlnp | grep 5433 to check if anything is listening on that port.
4. Interpret result: The grep matches port 15433, not 5433. This means port 5433 is not in use by anything — but YugabyteDB is also not using it. Something prevented YugabyteDB from binding to 5433.
5. Refine hypothesis: The assistant says "This might be a host network issue where port 5433 is already in use" — but the evidence doesn't fully support this. Port 5433 is free. The real issue might be that other YugabyteDB ports (7000, 7100) are in use, causing YugabyteDB to fail during initialization and never reach the point of binding to 5433. The phrase "Interesting — YugabyteDB is binding to port 15433 instead of 5433" is a bit misleading. YugabyteDB wasn't binding to 15433 as an alternative for 5433; rather, 15433 is the web UI port that happens to contain "5433" as a substring. The assistant's subsequent investigation would reveal the true nature of the problem.

The Broader Lesson

This message is a microcosm of distributed systems debugging. A seemingly simple change — switching a Docker networking mode — cascaded into a complex failure mode involving port conflicts, health check failures, and the subtle interactions between containerized services and their host environment. The assistant's methodical approach — observe, hypothesize, test, refine — is the universal debugging workflow, but it's also a reminder of how easily assumptions can lead us astray.

The most important lesson is about the hidden dependencies we create when building complex systems. Every port assignment, every networking mode, every default configuration is an assumption about the environment. When those assumptions change — as they did when switching from bridge to host networking — the entire edifice of working behavior can collapse. The assistant's discovery that port 5433 wasn't where it was supposed to be was the first crack in that edifice, and it led to a complete rethinking of the test cluster's network architecture.