When Host Networking Collides: Diagnosing Port Conflicts in a Distributed S3 Test Cluster

Introduction

In any complex distributed systems project, infrastructure decisions made for good reasons can cascade into unexpected failures. Message 1209 captures one such moment: a developer who has just converted a Docker Compose-based test cluster from bridge networking to host network mode—seeking to eliminate a performance bottleneck—discovers that the cluster's YugabyteDB database container is failing its health checks. The message is a concise diagnostic pivot: the assistant realizes that pre-existing services on the host machine are occupying ports that YugabyteDB needs, causing the database to bind to alternative ports and break internal health checks. This seemingly small observation opens a window into the realities of debugging containerized infrastructure, the trade-offs of network modes, and the importance of understanding what "host network" actually means for port allocation.

The Message

The subject message reads:

I see the problem - there are already services running on ports 7000, 7100 (YugabyteDB internal ports) and 8080 which is causing conflicts! YugabyteDB is binding to alternative ports. Let me check what the actual ports are:

>

``bash docker exec test-cluster-yugabyte-1 yugabyted status 2>/dev/null || echo "Container not running properly" ``

The message then shows the output of that command—a spinning progress indicator as yugabyted status attempts to report the database's state, apparently hanging or taking a long time.

The Chain of Reasoning: Why This Message Was Written

To understand why this message exists, we must trace the reasoning that led to it. The session leading up to message 1209 was focused on performance testing a horizontally scalable S3 storage architecture built on top of YugabyteDB and custom "Kuri" storage nodes. Earlier load tests had revealed a troubling pattern: at high concurrency (100+ workers), the system produced hundreds of "connection reset by peer" errors. The assistant and user had hypothesized that Docker's userland proxy—the default docker-proxy process that forwards ports from host to container in bridge networking mode—was becoming a bottleneck.

The decision to switch to host network mode (message 1186: "Rewrite the test-cluster to use host network") was a logical response. Host networking eliminates the proxy layer entirely: containers bind directly to the host's network interfaces, reducing latency and increasing throughput. The assistant rewrote docker-compose.yml, updated gen-config.sh to assign distinct ports for each node, and restarted the cluster.

But the restart failed. YugabyteDB came up as "unhealthy." The health check—which tries to connect to 127.0.0.1:5433 (the YSQL port)—was failing with "Connection refused." This was puzzling because the container logs showed YugabyteDB starting successfully.

The assistant's investigation (messages 1203–1208) revealed the root cause. Running ss -tlnp | grep 5433 showed that YugabyteDB was actually binding to port 15433, not 5433. The database had auto-detected that port 5433 was already in use on the host and had selected an alternative. This is a common behavior in database systems that check for port availability at startup.

Message 1209 is the moment of synthesis: the assistant connects the port mismatch to the broader problem of pre-existing services on the host. The ports 7000, 7100 (YugabyteDB's internal RPC and HTTP ports for tablet servers and master processes) and 8080 (which the new config assigned to kuri-2's S3 API) were already occupied by other processes. The host machine, which had been running various development services, was not a clean slate.

Assumptions Made

This message reveals several assumptions—some valid, some not—that shaped the debugging process.

Assumption 1: Host network mode would be a drop-in replacement. The assistant assumed that switching from bridge to host networking would be straightforward: remove the networks: and ports: directives, add network_mode: host, and everything would work identically but faster. This assumption overlooked the fundamental difference in network isolation. In bridge mode, each container has its own network namespace with its own loopback interface and port space. Port conflicts between containers and the host simply cannot occur because they live in different namespaces. In host mode, all containers share the host's network stack, making every port binding a potential conflict.

Assumption 2: The host had free ports for all container services. The assistant had assigned specific ports in gen-config.sh: 8078 for the S3 proxy, 8079 for kuri-1's S3 API, 8080 for kuri-2's S3 API, 7001 and 7002 for LocalWeb endpoints, and 5433/9042 for YugabyteDB. These assignments assumed the ports were available. In reality, port 8080 was already in use by another service, and ports 7000/7100 (YugabyteDB's internal ports) were also occupied, causing the database to shift its bindings.

Assumption 3: The health check failure indicated a problem with YugabyteDB itself. Initially, the assistant investigated whether YugabyteDB was starting correctly, checking logs and container status. The health check was failing, but the database process itself was running. The real problem was not with YugabyteDB's functionality but with its network configuration: it was listening on a non-standard port, so the health check (which targeted the standard port) couldn't reach it.

Assumption 4: The yugabyted status command would provide useful diagnostic information. The assistant attempted to run docker exec test-cluster-yugabyte-1 yugabyted status to check the database's state from inside the container. The command appeared to hang, showing a spinning progress indicator. This suggests that yugabyted status itself was having trouble connecting to the local YugabyteDB process—perhaps because it was also trying to reach the standard ports and failing.

Mistakes and Incorrect Assumptions

The primary mistake here was not anticipating port conflicts when switching to host networking. This is a classic pitfall. Bridge networking provides a convenient isolation layer that developers often take for granted. When that layer is removed, all the implicit assumptions about port availability surface as runtime failures.

A secondary mistake was in the port assignment strategy. The assistant assigned port 8080 to kuri-2's S3 API. Port 8080 is a common alternative HTTP port used by many development tools, proxy servers, and application frameworks. Choosing a less commonly occupied port range (e.g., 8081, 8082) might have avoided the conflict. Similarly, the YugabyteDB internal ports (7000, 7100) are well-known defaults for YugabyteDB's own inter-node communication—if another YugabyteDB instance or a related service was running on the host, the conflict was almost inevitable.

There was also an implicit assumption that the host development environment was "clean"—that no other services were using the ports the cluster needed. In practice, developers often have multiple services running: web servers, databases, caching layers, monitoring tools, and so on. A production server might be dedicated to a single workload, but a development machine is a shared environment.

Input Knowledge Required

To understand this message, a reader needs knowledge in several areas:

Docker networking modes. Understanding the difference between bridge networking (default, with NAT and port mapping via docker-proxy) and host networking (direct binding to host interfaces) is essential. Without this, the significance of "host network mode" and the nature of the port conflicts would be lost.

YugabyteDB architecture. Knowledge that YugabyteDB uses ports 5433 for YSQL (PostgreSQL-compatible), 9042 for YCQL (Cassandra-compatible), 7000 for intra-node RPC communication, and 7100 for HTTP monitoring helps explain why those specific ports matter. The fact that YugabyteDB can auto-select alternative ports when defaults are occupied is also relevant.

Docker Compose and container orchestration. Familiarity with docker-compose.yml structure, health checks, and the network_mode directive provides context for how the cluster was configured and why the health check failed.

Linux networking tools. The assistant uses ss -tlnp to list listening sockets, docker inspect to check container health status, and docker exec to run commands inside containers. Understanding these tools and their output is necessary to follow the diagnostic process.

The broader S3 architecture context. The cluster consists of S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB metadata store. The performance testing that motivated the host network switch—connection resets at high concurrency—explains why the assistant was willing to make such a significant infrastructure change.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

Diagnostic confirmation. The message confirms that the YugabyteDB health check failure is caused by port conflicts, not by a problem with YugabyteDB itself or with the Docker image. This narrows the debugging scope considerably.

A concrete list of conflicting ports. The assistant identifies ports 7000, 7100, and 8080 as already occupied. This is actionable information: the developer can now either free those ports, choose different ports for the cluster services, or revert to bridge networking.

A demonstration of diagnostic technique. The message shows a methodical approach to debugging container health failures: check container logs, inspect health check status, verify port bindings on the host, and correlate the two. The use of ss -tlnp to list listening sockets is a practical technique worth noting.

A cautionary example. This message serves as a case study in the hidden costs of host networking. While host mode eliminates the Docker proxy bottleneck, it introduces port management complexity that bridge mode abstracts away. The trade-off is clear: performance at the cost of isolation.

The Thinking Process

The assistant's thinking process in this message is visible in the structure of the investigation. The message begins with a synthesis statement: "I see the problem." This is the conclusion drawn from the preceding diagnostic steps (messages 1203–1208). The assistant then lists the specific ports causing conflicts, showing that they've correlated the health check failure (can't connect to 5433) with the port binding observation (YugabyteDB bound to 15433 instead) and traced that back to the occupied ports (7000, 7100, 8080).

The decision to run docker exec test-cluster-yugabyte-1 yugabyted status is an attempt to get more information from inside the container. The assistant wants to confirm what ports YugabyteDB thinks it's using and whether the database is fully operational despite the health check failure. The command's apparent hang (the spinning indicator) suggests that even internal diagnostics are affected by the port confusion—the yugabyted status tool likely tries to connect to the standard ports as well.

The message ends with the output truncated, showing the command is still running. This creates a natural pause: the assistant has identified the likely cause but needs more data to confirm. The reader can sense the debugging process in real time—the moment of insight followed by the attempt to verify.

Conclusion

Message 1209 is a small but illuminating moment in a larger debugging session. It captures the instant when a developer realizes that a seemingly straightforward infrastructure change has introduced unexpected complexity. The switch to host networking, motivated by a legitimate performance concern, collided with the messy reality of a shared development environment where ports are not cleanly available.

The message teaches a valuable lesson about network isolation in containerized systems. Bridge networking is not just a convenience feature—it provides real isolation guarantees that simplify deployment. Removing that isolation requires careful auditing of the host's port usage and a willingness to either reserve ports or handle conflicts gracefully.

For the reader, this message offers a window into the diagnostic process of a distributed systems engineer: the methodical elimination of hypotheses, the use of system tools to gather evidence, and the synthesis of multiple data points into a coherent explanation. It's a reminder that in complex systems, the most impactful bugs are often not in the code but in the assumptions we make about the environment our code runs in.