When Host Networking Collides: A Case Study in Docker Port Conflicts

The Message

[assistant] YugabyteDB failed healthcheck. Let me check what's happening:
[bash] docker logs test-cluster-yugabyte-1 2>&1 | tail -50
Starting yugabyted...
...

This short, seemingly routine diagnostic message — message index 1203 in a lengthy coding session — marks a critical inflection point in the development of a horizontally scalable S3 storage cluster. On its surface, it is a simple observation: a container healthcheck failed. But beneath that surface lies a cascade of architectural decisions, unspoken assumptions, and the kind of real-world debugging that separates working prototypes from production systems. This article unpacks the single message in depth, examining the reasoning, context, assumptions, and knowledge that converge in these few lines.

The Context: Why This Message Was Written

To understand why the assistant wrote this message, we must trace the events of the preceding hour. The team had been building and load-testing a three-layer S3 architecture consisting of stateless frontend proxies, Kuri storage nodes, and a shared YugabyteDB metadata store. Load tests at 100 and 1000 concurrent workers revealed a troubling pattern: "connection reset by peer" errors that the system misidentified as data corruption. Through investigation, the team traced the bottleneck to Docker's userland proxy, which becomes a serialization point under high concurrency.

The user's instruction was decisive: "Rewrite the test-cluster to use host network" (message 1186). The assistant complied, rewriting docker-compose.yml to replace bridge networking with network_mode: host, eliminating the Docker proxy layer. The change was conceptually clean — containers would bind directly to host ports, bypassing Docker's NAT layer and its associated overhead. The assistant stopped the old cluster, regenerated configuration files, and started the new cluster. The start script appeared to succeed.

Then came message 1203. The assistant noticed that despite the apparent success, the YugabyteDB container was failing its healthcheck. This observation is the entire content of the message — a one-sentence diagnosis followed by a diagnostic command. The brevity is deceptive. The message encodes a sophisticated chain of reasoning: the assistant had just performed a major infrastructure change, observed the cluster's startup, noticed an anomaly (the healthcheck failure), and immediately pivoted to investigation mode. The message is the pivot point.

The Reasoning Process Visible in the Message

The assistant's thinking, though compressed into a single sentence, reveals a structured diagnostic approach. The phrase "YugabyteDB failed healthcheck" is not merely a statement of fact; it is a hypothesis framed as an observation. The assistant has already checked the container status (visible in the subsequent message 1204, where docker ps -a shows the container as "unhealthy"). The healthcheck mechanism is a Docker construct that periodically executes a command inside the container to verify its readiness. A failing healthcheck means the container process is running but not yet serving requests — or, more concerningly, is running but unable to serve requests due to some configuration issue.

The assistant's next action — "Let me check what's happening" followed by docker logs — demonstrates a disciplined debugging methodology: start with the most accessible diagnostic signal (container logs) before moving to more specific probes. The logs show YugabyteDB's own startup sequence, which appears to be stuck at "Checking YSQL Status..." — the database's internal readiness check is failing because it cannot connect to its own PostgreSQL-compatible endpoint on port 5433.

What is remarkable is what the assistant does not do. It does not restart the container, does not blame the healthcheck configuration, and does not assume a transient glitch. The assistant recognizes that a fundamental change was just made (host network mode) and that the healthcheck failure is likely a symptom of that change. This is systems thinking: when a single change precedes a failure, investigate the change first.

Assumptions Made and Their Consequences

The host network migration carried several implicit assumptions, and the healthcheck failure exposed them.

Assumption 1: Host ports are available. The assistant assumed that the ports used by YugabyteDB — 5433 (YSQL), 9042 (YCQL), 7000 (master HTTP), 7100 (master RPC), 9000 (tserver HTTP), 9100 (tserver RPC) — would be free on the host machine. In bridge networking mode, these ports are internal to the Docker network and can overlap with host services without conflict. In host networking mode, the container binds directly to the host's network stack, so any port conflict causes the container to either fail to bind or bind to an alternative port. The assistant's subsequent investigation (messages 1206–1211) revealed that ports 7000 and 7100 were already occupied by existing services on the host, causing YugabyteDB to shift to port 15433 for YSQL and likely other ports for its internal services.

Assumption 2: The healthcheck would adapt. Docker's HEALTHCHECK instruction in the YugabyteDB image was hardcoded to test 127.0.0.1:5433. When YugabyteDB bound to port 15433 instead (due to the 5433 conflict), the healthcheck continued probing the wrong port, perpetually failing. The assistant's later docker inspect output (message 1205) confirms this: the healthcheck log shows ysqlsh: could not connect to server: Connection refused on port 5433.

Assumption 3: Host networking is a drop-in replacement. The assistant treated the networking change as a mechanical substitution — remove networks: and ports:, add network_mode: host. But host networking changes the container's relationship to the host fundamentally. The container's 127.0.0.1 becomes the host's 127.0.0.1. Services that bind to loopback inside the container suddenly become visible (and conflict-prone) on the host loopback. The assistant's subsequent realization — "with host network mode, the container's 127.0.0.1 IS the host's loopback" (message 1206) — captures this belated understanding.

Assumption 4: The existing cluster was fully stopped. The assistant stopped the cluster via ./stop.sh before restarting, but the investigation revealed that ports 7000 and 7100 were occupied by something else on the host — not necessarily the test cluster itself. This suggests either a lingering process from a previous run or a separate service entirely. The assistant's ss -tlnp command (message 1207) shows these ports in use but does not identify the owning process, leaving the source of the conflict ambiguous.

Mistakes and Incorrect Assumptions

The most significant mistake was the failure to audit the host's port landscape before committing to host networking. A more cautious approach would have been:

  1. Check which ports are currently in use on the host (ss -tlnp or netstat)
  2. Cross-reference against the ports required by all containers in the compose file
  3. Either select unused ports or verify that the existing services can be temporarily stopped The assistant skipped this audit and proceeded directly to the rewrite. The healthcheck failure was the inevitable consequence. A secondary mistake was not anticipating the healthcheck rigidity. The assistant knew that Docker healthchecks are configured in the Dockerfile or compose file, but did not consider that the YugabyteDB image's built-in healthcheck might hardcode specific ports. When the port shifted due to conflict, the healthcheck became a false negative — reporting the container as unhealthy when the database process was actually running, just on a different port.

Input Knowledge Required

To fully understand this message, the reader needs knowledge in several domains:

Docker networking modes: Bridge networking creates an isolated network namespace per container, with explicit port mapping via -p or ports: in compose. Host networking removes the isolation, making the container share the host's network stack directly. This means no NAT overhead, but also no port conflict protection.

Docker healthchecks: The HEALTHCHECK instruction in a Dockerfile or compose file defines a command that Docker runs periodically to determine if the container is "healthy." The healthcheck runs inside the container's namespace, so in host networking mode, 127.0.0.1 refers to the host's loopback interface.

YugabyteDB startup sequence: YugabyteDB uses a multi-process architecture with yb-master and yb-tserver processes. The yugabyted orchestration tool manages startup, first launching the master and tserver processes, then bringing up the UI, then checking YSQL (PostgreSQL-compatible) and YCQL (Cassandra-compatible) endpoints. The "Checking YSQL Status..." message indicates the YSQL endpoint is not yet accepting connections.

Port conflict behavior: When a Linux process attempts to bind to a port that is already in use, the bind() system call fails with EADDRINUSE. YugabyteDB's yugabyted tool appears to handle this by selecting an alternative port (15433 instead of 5433), but this fallback is not communicated back to Docker's healthcheck mechanism.

Output Knowledge Created

This message and its surrounding investigation produced several valuable pieces of knowledge:

  1. The host has port conflicts on 7000 and 7100. This is a concrete finding that will inform all future networking decisions on this machine. Any container using host networking must avoid these ports.
  2. YugabyteDB's port fallback behavior. The database will bind to alternative ports when conflicts exist, but this creates a mismatch with healthchecks and client configurations that hardcode the standard ports.
  3. Host networking is not a drop-in replacement. The migration from bridge to host networking requires careful port planning, healthcheck auditing, and configuration updates for all dependent services.
  4. The diagnostic approach works. The assistant's method — notice the anomaly, check container logs, inspect health status, check port bindings, identify the root cause — is a replicable debugging pattern. The knowledge cascade continues in subsequent messages, where the user instructs "Change all YB ports" (message 1213) and the assistant systematically reconfigures every YugabyteDB port to avoid conflicts, ultimately getting the cluster healthy again.

The Broader Significance

This message is a microcosm of a universal engineering experience: the clean theoretical solution (host networking eliminates Docker proxy overhead) colliding with messy reality (the host already has services on those ports). The assistant's response — immediate diagnosis rather than frustration or guesswork — exemplifies the engineering mindset. The message is also a reminder that infrastructure changes have second-order effects. Changing the networking mode of one container (YugabyteDB) affects not just that container's performance characteristics, but its ability to start, its healthcheck semantics, and its interaction with every other service on the host.

In the end, the solution was not to abandon host networking but to adapt: offset all YugabyteDB ports to a range that avoids conflicts, update healthcheck commands to match, and propagate the new port numbers through the configuration chain. The cluster came back healthy, and the load tests at 100 and 1000 workers could proceed without the Docker proxy bottleneck. But the path to that solution began with a single, unadorned observation: "YugabyteDB failed healthcheck."