When Host Networking Collides: A Case Study in Docker Port Conflicts During Distributed System Testing

The Message

In a single, deceptively brief message during an intense debugging session for a horizontally scalable S3 storage system, the assistant writes:

Ports 7000 and 7100 are used by something. This is a problem - YugabyteDB uses those ports internally. Let me see if the yugabyte managed to start on alternate ports:

>

`` [bash] docker exec test-cluster-yugabyte-1 ss -tlnp 2>/dev/null | head -20 OCI runtime exec failed: exec failed: unable to start container process: exec: "ss": executable file not found in $PATH ``

This message, appearing at index 1212 in a long conversation spanning dozens of debugging iterations, represents the moment a well-intentioned optimization collides with the messy reality of shared development environments. To understand why this message matters, we must trace the chain of decisions that led here and unpack the assumptions that silently collapsed.

The Road to Host Networking

The story begins with a performance investigation. The team had implemented a CQL batcher to improve S3 metadata write throughput, and initial load tests showed promising results: 115 MB/s with 10 workers and zero corruption. But scaling to 100 workers revealed a troubling pattern — "connection reset by peer" errors masquerading as data corruption. The loadtest tool was reporting thousands of verification failures, but closer inspection showed these were not actual checksum mismatches but TCP connections being dropped before read-after-write verification could complete.

The assistant's analysis pointed to Docker's userland proxy (docker-proxy) as the likely bottleneck. Docker's default bridge networking mode routes traffic through a user-space proxy process that translates between the host's network stack and the container's virtual Ethernet interface. At high concurrency — 100 or 1000 concurrent HTTP connections — this proxy can become saturated, leading to connection resets. The data was compelling: 10 workers produced zero errors, but 100 workers yielded 958 "corruption" events (actually connection resets), and 1000 workers produced 3,735.

The user's response was direct and decisive: "Rewrite the test-cluster to use host network" (message 1186). Host network mode bypasses Docker's network virtualization entirely, binding container processes directly to the host's network interfaces. No proxy, no NAT, no virtual bridge — just raw socket access. For a performance-sensitive distributed system like this S3 cluster, it seemed like the obvious fix.

The Implementation

The assistant dutifully rewrote the entire test cluster infrastructure. The docker-compose.yml was restructured: network_mode: host replaced the custom bridge network, port mappings were removed (since containers bind directly to host ports), and each Kuri node was assigned distinct host ports — kuri-1 on 8079/9010/7001, kuri-2 on 8080/9011/7002, and the S3 proxy on 8078. The configuration generator (gen-config.sh) and documentation (README.md) were updated accordingly.

The old cluster was stopped, configuration regenerated, and the new host-network cluster was launched. This is where the trouble began.

The Collision

YugabyteDB failed its health check. The container status showed "unhealthy" with a failing streak of 11 attempts. The health check logs revealed the problem: ysqlsh: could not connect to server: Connection refused — Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5433?

With host network mode, the container's 127.0.0.1 is the host's loopback interface. The health check was trying to connect to the host's port 5433, but something was wrong. Further investigation showed that YugabyteDB had bound to port 15433 instead of 5433 — a classic symptom of port conflict: when a process finds its desired port already occupied, it either fails or, if the application supports it, falls back to an alternative port.

The assistant checked for port conflicts and found them: ports 7000 and 7100 were already in use on the host machine. These are YugabyteDB's internal RPC ports — port 7000 for intra-node communication (the "yb-tserver" and "yb-master" RPCs) and port 7100 for the YB-master HTTP UI. With these ports occupied, YugabyteDB could not bind its standard ports and was likely starting in a degraded or misconfigured state.

The Diagnostic Attempt — and Its Failure

Message 1212 captures the assistant's first diagnostic step after discovering the port conflict. Having identified that ports 7000 and 7100 are occupied, the assistant reasons: "YugabyteDB uses those ports internally. This is a problem." The next logical step is to check whether YugabyteDB managed to start on alternate ports — perhaps the database daemon has a fallback mechanism or was configured to use different ports.

The chosen diagnostic command is revealing: docker exec test-cluster-yugabyte-1 ss -tlnp 2>/dev/null | head -20. The ss command (socket statistics) would show all listening TCP sockets inside the container, revealing which ports YugabyteDB actually bound to. The -t flag filters for TCP, -l for listening sockets, -n for numeric output, and -p to show the process. This is exactly the right tool for the job.

But the command fails: OCI runtime exec failed: exec failed: unable to start container process: exec: "ss": executable file not found in $PATH. The YugabyteDB Docker image, based on a minimal Linux distribution, does not include the ss utility. This is a mundane but frustrating failure — the tool the assistant reached for is not available in the container's environment.

What This Message Reveals About the Debugging Process

This message is interesting not because it contains a breakthrough, but because it captures a moment of friction — the point where the debugging process itself stumbles. Several layers of assumptions are visible here:

Assumption 1: Host networking would be a clean fix. The assistant and user both assumed that switching to host network mode would eliminate the Docker proxy bottleneck without introducing new problems. This assumption was reasonable given the evidence (connection resets at high concurrency), but it failed to account for the state of the host machine's port allocations. In a shared development environment, ports 7000 and 7100 may be occupied by any number of services — previous YugabyteDB instances, monitoring tools, or other development servers. The host network mode trades one set of problems (Docker proxy throughput limits) for another (direct port conflicts with the host's existing services).

Assumption 2: The standard diagnostic tools would be available. The ss command is ubiquitous on Linux distributions, but the YugabyteDB Docker image is purpose-built and minimal. It contains exactly what YugabyteDB needs to run, not a full debugging toolkit. This is a common pitfall in container debugging: the tools you rely on for diagnosis may not be present inside the container image. The assistant might have better luck with netstat -tlnp (also likely absent), cat /proc/net/tcp (a raw interface that always works), or simply checking the YugabyteDB logs for port binding messages.

Assumption 3: The port conflict was the root cause of the health check failure. While ports 7000 and 7100 being occupied is certainly a problem, it's worth noting that YugabyteDB's health check was failing on port 5433 (YSQL), not on the internal ports. The relationship between the occupied internal ports and the YSQL port failure is indirect: when YugabyteDB cannot bind its internal RPC ports, the entire process may fail to initialize properly, which in turn prevents the YSQL endpoint from becoming available. But there could also be other factors — perhaps the host already had a PostgreSQL instance on port 5433, or the YugabyteDB process was crashing for a different reason entirely.

The Broader Context: A Pattern of Infrastructure Debugging

This message sits within a larger arc of infrastructure debugging that spans the entire session. The assistant has been working through a cascade of issues: first, distinguishing real data corruption from timeout errors in the loadtest tool; then implementing a CQL batcher to improve write throughput; then investigating Docker proxy bottlenecks; then converting to host networking; and now, dealing with the port conflicts that host networking exposed.

Each step in this chain reveals something about the system's architecture and the environment it runs in. The port conflict at message 1212 is particularly instructive because it demonstrates a fundamental tension in distributed systems testing: the desire for realistic, high-performance test environments (which argues for host networking) versus the need for isolation and reproducibility (which argues for bridge networking with port mapping). There is no universal right answer — the choice depends on what you're testing and what constraints your environment imposes.

Input Knowledge Required

To fully understand this message, the reader needs several pieces of context:

  1. The host networking decision: That the test cluster was recently converted from bridge networking (with Docker's userland proxy) to host network mode, motivated by connection reset errors at high concurrency during S3 load testing.
  2. YugabyteDB's port architecture: That YugabyteDB uses port 7000 for internal RPC communication (yb-tserver and yb-master) and port 7100 for the master HTTP UI, in addition to the well-known YSQL port 5433 and YCQL port 9042.
  3. The health check failure: That YugabyteDB's container was showing "unhealthy" status because its health check (connecting to 127.0.0.1:5433) was failing, and that this was likely caused by the port conflict on ports 7000/7100 preventing proper initialization.
  4. The Docker networking models: Understanding the difference between bridge networking (where containers have isolated network stacks and ports are mapped through a proxy) and host networking (where containers share the host's network stack directly).

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. A confirmed port conflict: The assistant has definitively identified that ports 7000 and 7100 are occupied on the host, which explains why YugabyteDB is failing to initialize properly under host network mode.
  2. A failed diagnostic attempt: The ss command is not available inside the YugabyteDB container, which means alternative approaches are needed to inspect the container's listening ports.
  3. An implicit decision point: The assistant now faces a fork in the road — either resolve the port conflict on the host (by stopping whatever is using ports 7000 and 7100), or revert to bridge networking and find another way to address the connection reset problem.

The Thinking Process

The reasoning visible in this message is concise but revealing. The assistant follows a clear logical chain:

  1. Observation: Ports 7000 and 7100 are already in use on the host (established in message 1211 via ss -tlnp | grep).
  2. Inference: Since YugabyteDB uses these ports internally, this is likely the cause of the initialization failure.
  3. Hypothesis: YugabyteDB might have started on alternate ports as a fallback.
  4. Test: Use docker exec to run ss -tlnp inside the container and see what ports are actually listening.
  5. Result: The test fails because ss is not available in the container. The thinking is sound, but it reveals a gap in the assistant's mental model of container environments. The assistant assumes that a standard Linux tool like ss will be available inside any container, which is not always true — especially for production-oriented images like YugabyteDB's that are optimized for size and security. A more experienced container debugger might first check what tools are available (which ss, ls /bin, ls /usr/bin) or use a universally available interface like /proc/net/tcp.

The Aftermath

What happens after this message is instructive. The assistant does not immediately find an alternative diagnostic command. Instead, the conversation continues with the assistant eventually deciding to revert to bridge networking — acknowledging that host networking introduced more problems than it solved. The revert is not a failure; it is a pragmatic recognition that the development environment's constraints (occupied ports, shared host) make host networking impractical, and that the connection reset issue needs a different solution.

This pattern — try a promising optimization, discover unforeseen side effects, gather diagnostic data, and revert while preserving the lessons learned — is the essence of disciplined systems engineering. Message 1212 captures the diagnostic pivot point, the moment when the team learns that their chosen path has hidden costs.