When the Fix Breaks Something Else: Debugging Host Network Port Conflicts in a Distributed S3 Test Cluster

Introduction

The message at index 1210 captures a quiet but pivotal diagnostic moment in a distributed systems debugging session. The assistant, having just converted an entire test cluster from Docker bridge networking to host network mode to eliminate proxy bottlenecks, now faces an unexpected consequence: the database container refuses to become healthy. The message is deceptively brief—a single bash command wrapped in a diagnostic observation—but it represents the moment when a well-intentioned architectural change collides with the messy reality of shared host resources. This article examines that moment in detail, unpacking the reasoning, assumptions, and knowledge flows that converge in this one message.

The Message

Here is the subject message exactly as it appeared in the conversation:

[assistant] It's still "Bootstrapping". The issue is the host network mode is causing port conflicts with existing services on the host. Let me check if yugabyte eventually becomes healthy: [bash] sleep 10 && docker exec test-cluster-yugabyte-1 bin/ysqlsh -h 127.0.0.1 -U yugabyte -d yugabyte -c '\l' 2>&1 || echo "Still not ready" ysqlsh: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5433? Still not ready

The message contains two distinct layers: the assistant's own analytical statement ("It's still 'Bootstrapping'. The issue is the host network mode is causing port conflicts...") and the output of a diagnostic command that confirms the hypothesis. The command itself is carefully constructed—a ten-second sleep to give the database time to initialize, followed by a direct PostgreSQL connection attempt via ysqlsh, with a fallback message if the connection fails.

The Road to Host Networking

To understand why this message exists, we must trace the events that led to it. Earlier in the session, the team had been running load tests against the S3 test cluster at increasing concurrency levels. At 10 concurrent workers, the system performed flawlessly: 115 MB/s write throughput with zero corruption. At 100 workers, throughput tripled to 334 MB/s, but nearly a thousand "corruption" errors appeared. At 1000 workers, the corruption count rose further. Crucially, the assistant had already determined that these were not actual data corruption but connection resets—the read: connection reset by peer errors indicated that Docker's userland proxy (docker-proxy) was becoming a bottleneck under high concurrency.

The user's response was direct and decisive: "Rewrite the test-cluster to use host network" (message 1186). This was a reasonable engineering judgment—bypassing Docker's NAT layer would eliminate the proxy bottleneck and allow direct TCP connections to the containers. The assistant immediately implemented the change, rewriting docker-compose.yml to use network_mode: host for all services, adjusting port assignments to avoid conflicts between the two Kuri storage nodes, and updating the configuration generator and README documentation.

But the transition was not smooth. When the assistant stopped the old cluster, regenerated configurations, and started the new one, YugabyteDB—the distributed SQL database serving as the metadata store—failed its health checks repeatedly. The container status showed "unhealthy" with a failing streak of 11 consecutive checks. The health check logs revealed the problem: ysqlsh: could not connect to server: Connection refused on 127.0.0.1:5433.## The Core Problem: Host Networking and Port Conflicts

The assistant's diagnosis in message 1210—"the host network mode is causing port conflicts with existing services on the host"—identifies the fundamental issue. In Docker's default bridge networking mode, each container gets its own network namespace with its own loopback interface. Port 5433 inside the container is the container's port, isolated from the host. The health check connects to 127.0.0.1:5433 inside the container, which works because YugabyteDB is listening there.

In host network mode, however, the container shares the host's network stack entirely. The container's 127.0.0.1 becomes the host's 127.0.0.1. If any process on the host is already bound to port 5433, or if a previous instance of YugabyteDB left behind a port binding, the new container cannot acquire that port. Earlier in the debugging session (message 1206), the assistant had discovered that YugabyteDB was binding to port 15433 instead of 5433—a classic symptom of port conflict where the application falls back to an alternative port. The ss -tlnp output confirmed that 127.0.0.1:15433 was listening, but the health check was still probing 5433.

This is a subtle and dangerous class of bug. The container appears to start successfully—the YugabyteDB processes launch, the UI becomes ready—but the database is inaccessible on the expected port. The health check fails, Docker marks the container as unhealthy, and any dependent service (the Kuri storage nodes, the S3 proxy) that tries to connect to YugabyteDB on port 5433 will fail with a connection refused error. The entire cluster becomes non-functional despite all containers showing as "Up."

Assumptions Made and Broken

Several assumptions underpin the host network conversion, and message 1210 reveals where they break down. The first assumption is that host ports would be available. The assistant had checked for port conflicts before the conversion, but the dynamic nature of port allocation in a development environment means that services can claim ports at any time. The earlier ss output showed ports 7000 and 7100 already in use—these are YugabyteDB's internal RPC ports. The presence of these pre-existing bindings suggests that a previous YugabyteDB instance (perhaps from an earlier test run) had left residual port claims, or that other development services were running on the host.

The second assumption is that the health check mechanism would adapt to host networking. Docker's healthcheck directive in the compose file was designed for bridge networking, where 127.0.0.1:5433 reliably reaches the container's PostgreSQL. In host mode, that same address reaches the host's loopback, which may or may not have a PostgreSQL server. The health check configuration was not updated when the network mode changed, creating a mismatch between the check's target and the actual service location.

The third assumption is that YugabyteDB's startup sequence would gracefully handle port conflicts. In practice, when YugabyteDB's yugabyted process detects that its preferred ports are occupied, it selects alternatives (like 15433 instead of 5433), but this fallback behavior is not communicated back to Docker's health check system. The container reports itself as healthy from YugabyteDB's perspective, but Docker sees a failing health check and marks it unhealthy.

The Thinking Process Visible in the Message

The assistant's reasoning is compact but reveals a clear diagnostic chain. The observation "It's still 'Bootstrapping'" refers to the output of yugabyted status from the previous message, which showed the database stuck in the bootstrapping phase. The assistant connects this symptom to the root cause: "The issue is the host network mode is causing port conflicts with existing services on the host." This is not a guess—it is an inference drawn from multiple pieces of evidence: the ss output showing alternative port bindings, the health check logs showing connection refused on 5433, and the knowledge that host networking collapses the container-host network boundary.

The diagnostic command is carefully designed. The sleep 10 gives the database time to complete its bootstrap sequence. The docker exec runs ysqlsh inside the container, bypassing any Docker networking issues. The -h 127.0.0.1 explicitly targets the loopback interface. The -c '\l' command (list databases) is a simple connectivity test. The || echo "Still not ready" provides a clear failure message. The output confirms the hypothesis: connection refused on 127.0.0.1:5433.

What is notable is what the assistant does not do in this message. There is no attempt to fix the problem yet—no fallback to bridge networking, no adjustment of health check ports, no manual port release. The message is purely diagnostic: confirm the hypothesis, gather evidence, and establish the current state. This restraint is itself a deliberate engineering choice. Before making changes, one must understand the exact failure mode.## Input Knowledge Required to Understand This Message

To fully grasp what is happening in message 1210, a reader needs several layers of context. First, they must understand Docker networking modes—specifically the difference between bridge networking (where each container has an isolated network stack with its own loopback interface) and host networking (where the container shares the host's network stack directly). Without this knowledge, the phrase "host network mode is causing port conflicts" would be opaque.

Second, the reader needs to understand the architecture of the test cluster: it consists of an S3 frontend proxy, two Kuri storage nodes, and a shared YugabyteDB metadata store. YugabyteDB is a distributed SQL database compatible with PostgreSQL, and it communicates on port 5433 for SQL connections and 9042 for CQL (Cassandra Query Language) connections. The health check mechanism in Docker Compose periodically runs ysqlsh against 127.0.0.1:5433 to verify the database is responsive.

Third, the reader needs to know the history of the session: that the cluster was originally running on bridge networking, that load tests revealed Docker proxy bottlenecks at high concurrency, and that the user explicitly requested the conversion to host networking. The assistant's earlier investigation had shown that direct connections to container IPs worked but that the Docker userland proxy was dropping connections under load.

Fourth, the reader must understand the diagnostic tools being used: docker exec to run commands inside a container, ysqlsh as the PostgreSQL/YugabyteDB command-line client, and the sleep command to introduce a timing delay. The || operator in bash provides a fallback message if the command fails, which is a common pattern for non-critical diagnostic probes.

Output Knowledge Created by This Message

Message 1210 produces several distinct pieces of knowledge. The most immediate is a confirmed diagnosis: the host network conversion has broken YugabyteDB's accessibility. The database container is running (it is not crashed), but it is not accepting connections on the expected port. This rules out several alternative explanations: the container is not failing to start, the Docker engine is not malfunctioning, and the fgw:local image is not corrupted.

The message also establishes a timing baseline. The sleep 10 followed by a failed connection attempt shows that the database does not become available within ten seconds of the check. This is useful for understanding whether the issue is a slow startup (which might resolve with more time) versus a permanent configuration problem (which will not resolve without intervention). The "Still not ready" output confirms the latter.

Furthermore, the message implicitly documents a failure mode of host networking that future developers can learn from. When converting a Docker Compose-based cluster to host networking, the health check configurations must be reviewed and potentially updated. The assumption that 127.0.0.1:5433 reaches the container's database is only valid in bridge mode; in host mode, that address reaches the host, and port conflicts become a real risk.

Mistakes and Incorrect Assumptions

While the assistant's diagnosis in this message is correct, the broader sequence reveals some incorrect assumptions made earlier. The most significant is the assumption that the host network conversion would be straightforward—that changing network_mode: host and removing port mappings would be sufficient. In practice, the conversion required updating port assignments to avoid inter-container conflicts (since containers on host network cannot both bind port 8078), adjusting the configuration generator to use distinct ports per node, and—critically—verifying that the database's health check would still function.

The assistant also assumed that host ports would be available. The earlier ss output showed ports 7000 and 7100 already in use on the host, which are YugabyteDB's internal ports. This suggests that a previous YugabyteDB instance or another service was occupying those ports, forcing the new instance to use alternatives. The assistant did not check for port availability before committing to the host network approach.

Another subtle mistake is the assumption that docker exec would reliably reach the database. The command docker exec test-cluster-yugabyte-1 bin/ysqlsh -h 127.0.0.1 runs ysqlsh inside the container, so 127.0.0.1 is the container's loopback. In host networking mode, the container's loopback is the host's loopback. If YugabyteDB inside the container is listening on 0.0.0.0:5433 (all interfaces), then 127.0.0.1:5433 from inside the container should work—unless the database process has bound to a different port due to the conflict. The 15433 port observed earlier confirms this fallback behavior.

Broader Implications for Distributed Systems Testing

Message 1210 illustrates a recurring challenge in distributed systems development: the tension between realistic testing environments and controlled isolation. Host networking provides more realistic network behavior—no proxy overhead, real TCP semantics, actual port bindings—but it sacrifices the isolation that makes Docker containers predictable. Port conflicts, resource contention, and interference from host services become real risks.

The assistant's approach to this problem is methodical: observe the symptom (unhealthy container), form a hypothesis (port conflict from host networking), and test it with a targeted diagnostic command. The message does not yet contain the solution—that will come in subsequent messages—but it establishes the foundation for the fix. Whether that fix involves releasing the conflicting ports, adjusting the health check configuration, or reverting to bridge networking with optimized proxy settings remains to be determined.

This moment also highlights the importance of understanding the full dependency chain in a containerized system. The S3 proxy depends on the Kuri nodes, which depend on YugabyteDB. If the database is unhealthy, the entire cluster is non-functional regardless of how well the other components are configured. A change that seems isolated to networking (switching to host mode) can cascade through the entire dependency tree, breaking components that were working perfectly.

Conclusion

Message 1210 is a small but revealing artifact of real-world distributed systems engineering. It captures the moment when a promising optimization—eliminating the Docker proxy bottleneck through host networking—encounters the messy reality of shared host resources. The assistant's diagnosis is correct, the diagnostic command is well-designed, and the restraint in not immediately applying a fix shows disciplined engineering judgment. The message serves as a case study in how assumptions about network isolation can break when that isolation is removed, and how a single configuration change can ripple through an entire system architecture.