The Port Conflict That Unraveled Host Networking: A Single Edit in a Distributed S3 Test Cluster

The Message

[assistant] [edit] /home/theuser/gw/test-cluster/docker-compose.yml Edit applied successfully.

This is the entirety of message index 1217 in a lengthy coding session building a horizontally scalable S3 storage system. On its surface, it is the most mundane of messages—a file-editing confirmation, barely a sentence long. Yet this single edit sits at the crux of a cascading series of architectural decisions, mistaken assumptions, and hard-won debugging that ultimately forced the assistant and user to abandon a promising network optimization and retreat to safer ground. To understand why this message matters, one must trace the chain of reasoning that led to it, the assumptions it shattered, and the knowledge it produced.

The Context: Why Host Networking Seemed Like a Good Idea

The story begins with performance testing. The assistant had implemented a CQL batcher to improve write throughput to YugabyteDB, the distributed SQL database serving as the metadata store for the S3 storage cluster. Load tests showed promising results: at 10 concurrent workers, throughput reached 115 MB/s with zero corruption. But at 100 and 1000 workers, the system was plagued by "connection reset by peer" errors. The assistant correctly diagnosed these as Docker's userland proxy becoming a bottleneck under high concurrency.

The solution seemed obvious: switch the Docker Compose setup to host network mode. In host mode, containers bind directly to the host's network interfaces, bypassing Docker's port-mapping proxy entirely. This eliminates the proxy bottleneck and should allow the S3 load tests to scale without connection resets. The user agreed with a succinct instruction: "Rewrite the test-cluster to use host network."

The assistant dutifully rewrote docker-compose.yml, removing all networks: and ports: mappings, and assigned each service distinct host ports. The S3 proxy would listen on port 8078, kuri-1 on 8079, kuri-2 on 8080, and so on. The cluster was stopped, configs regenerated, and the start.sh script was run.

The Assumption That Collapsed

The assumption was simple: the host machine had no services occupying the ports that YugabyteDB needed. YugabyteDB, by default, uses a well-known set of ports: 5433 for YSQL (PostgreSQL-compatible), 9042 for YCQL (Cassandra-compatible), 7000 for the yb-master HTTP interface, 7100 for yb-master RPC, 9000 for yb-tserver HTTP, 9100 for yb-tserver RPC, and 15433 for the web UI. In bridge networking mode, these ports are internal to the Docker network and never conflict with host services. But in host networking mode, the container's ports are the host's ports—every port the container tries to bind must be available on the host.

When the assistant started the cluster with host networking, YugabyteDB failed its health check. The health check script tried to connect to 127.0.0.1:5433 and received "Connection refused." The assistant investigated and found the root cause: ports 7000 and 7100 were already in use on the host by some existing process. YugabyteDB, detecting the conflict, silently fell back to alternative ports—binding yb-master HTTP to 15433 instead of 7000, for instance—but the health check was still looking for the standard ports. The container was marked "unhealthy" and the entire cluster was stuck in a bootstrapping loop.

The Edit: A Surgical Response to a Systemic Problem

The user's response was direct and unambiguous: "Change all YB ports." This was not a request to investigate why the ports were occupied or to find a way to free them. It was an instruction to work around the conflict entirely by relocating the entire YugabyteDB port footprint to a non-conflicting range.

The assistant's reasoning, visible in message 1216, shows a clear thought process. First, it enumerated every default port that YugabyteDB uses:

The Deeper Lesson: Host Networking's Hidden Costs

The decision to change all YugabyteDB ports, while pragmatic, reveals a deeper tension in distributed systems testing. Host networking mode offers undeniable performance benefits—eliminating the Docker proxy removes a significant bottleneck under high concurrency. But it comes at a cost: the test environment becomes tightly coupled to the host's port namespace. Every service must be carefully assigned non-conflicting ports, and any pre-existing service on the host becomes a potential source of failure.

The assistant's initial assumption that the host would have free ports was reasonable but untested. The debugging sequence that followed—checking ss -tlnp, discovering ports 7000 and 7100 in use, attempting to work around the conflict—illustrates the iterative nature of infrastructure debugging. Each discovery forced a new round of edits, and each edit revealed new complexities (like the yugabyted CLI's lack of direct port flags).

Input and Output Knowledge

To understand this message, one needs input knowledge of: Docker networking modes (bridge vs. host), YugabyteDB's default port assignments, the docker-compose.yml file structure, and the general architecture of the S3 test cluster (S3 proxy → Kuri storage nodes → YugabyteDB). One also needs to understand the performance context—that the switch to host networking was motivated by connection reset errors in load tests.

The output knowledge created by this message and its surrounding edits is significant. The assistant learned (and the conversation documents) that YugabyteDB's yugabyted CLI does not support direct port flags, requiring --master_flags and --tserver_flags for port customization. The conversation also demonstrates that port conflicts in host networking mode can cause YugabyteDB to silently fall back to alternative ports, creating a mismatch between expected and actual port bindings that breaks health checks.

The Aftermath

In the messages immediately following this edit, the assistant continued to refine the port configuration, updating the db-init container, the kuri node definitions, and the gen-config.sh script. But the host networking experiment was ultimately abandoned. The complexity of managing port conflicts across all services, combined with the yugabyted CLI's limited port configurability, led the assistant to revert to bridge networking. The final state of the test cluster used bridge mode with a clean, validated configuration.

This single edit message—"[edit] /home/theuser/gw/test-cluster/docker-compose.yml / Edit applied successfully."—marks the precise moment when the assistant committed to a port-offset strategy. It is a reminder that in infrastructure engineering, the simplest instructions often conceal the most complex reasoning, and that a two-line edit can represent hours of debugging, multiple false assumptions, and the hard-won knowledge of how a distributed database behaves when its networking assumptions are violated.