The Bridge Back: Why Host Network Mode Failed and What It Taught Us About Docker Infrastructure
Introduction
In the course of building a horizontally scalable S3 storage cluster, a seemingly straightforward performance optimization—switching Docker networking from bridge to host mode—unraveled into a cascade of failures that ultimately forced a strategic retreat. This article examines a single message (index 1211) in the coding session where the assistant, after investing significant effort into converting the entire test cluster to host network mode, discovered that the host's existing port allocations made the approach untenable. The message captures a moment of technical humility: the recognition that a theoretically sound optimization can be blocked by the messy reality of a shared development environment.
The Subject Message
The host network mode is causing conflicts because ports 7000 and 7100 are already used by something on the host. Let me revert to the bridge network mode but keep the connection pool/connection reset fixes we could try. Actually, let me check what's using 7000/7100:
>
`` LISTEN 0 128 127.0.0.1:7100 0.0.0.0:* LISTEN 0 128 127.0.0.1:7000 0.0.0.0:* ``
The Reasoning and Motivation
To understand why this message was written, we must trace the chain of reasoning that led to the host network experiment. Earlier in the session, load tests at high concurrency (100 and 1000 workers) had revealed a troubling pattern: "connection reset by peer" errors that the assistant initially mistook for data corruption. Through careful investigation, the team determined these were not actual checksum mismatches but rather TCP connections being dropped under load.
The assistant hypothesized that Docker's userland proxy—the default port forwarding mechanism in bridge networking—was the bottleneck. Docker's bridge network mode routes traffic through docker-proxy, a user-space process that can become saturated under high connection counts. The user had suggested this possibility explicitly: "Might be docker-proxy issues?" (message 1178). This prompted the assistant to restructure the entire test cluster to use network_mode: host, which bypasses Docker's proxy entirely by binding containers directly to the host's network stack.
The motivation was clear: eliminate a known bottleneck in the Docker networking layer to achieve higher throughput in S3 load tests. The assistant rewrote docker-compose.yml, updated gen-config.sh to assign distinct ports to each node, modified the README documentation, and prepared to restart the cluster. This was not a casual change—it represented a significant architectural shift that touched every service in the cluster.
The Investigation: Discovering the Port Conflict
When the assistant attempted to restart the cluster with host network mode, the first sign of trouble appeared: YugabyteDB failed its health check. The health check script, which attempts to connect to 127.0.0.1:5433, was failing because YugabyteDB had bound to port 15433 instead. This was a strong indicator of a port conflict—when a container in host network mode tries to bind to a port that is already occupied on the host, the container's process silently falls back to an alternative port.
The assistant systematically investigated the port situation. Running ss -tlnp | grep -E '(5433|9042)' revealed that YugabyteDB's expected ports were not in use, but further inspection with ss -tlnp | grep -E '(7000|7100)' uncovered the root cause: ports 7000 and 7100 were already bound to 127.0.0.1. These are YugabyteDB's internal communication ports—port 7000 for the YB master HTTP server and port 7100 for the YB tablet server RPC. Since host network mode makes the container share the host's network namespace, the YugabyteDB process inside the container could not bind to these ports because they were already occupied by some other process on the host.
The assistant's final command in the message—running lsof -i :7000—was an attempt to identify exactly which process was occupying port 7000, though the output was cut off. The key discovery was already made: the host environment had pre-existing services using ports that YugabyteDB required, making host network mode impossible without either stopping those services or reconfiguring YugabyteDB's internal port assignments.
Assumptions Made and Mistakes Identified
This episode reveals several assumptions that turned out to be incorrect:
Assumption 1: The host environment is clean. The assistant assumed that ports 7000, 7100, and 5433 would be available for the test cluster. In reality, the development host had other services running—possibly a previous YugabyteDB instance, monitoring tools, or other development services. This is a classic "works on my machine" pitfall, but inverted: the machine had pre-existing state that the Docker abstraction layer normally hides.
Assumption 2: Host network mode is a drop-in replacement for bridge mode. The assistant treated the conversion as primarily a matter of removing networks: and ports: directives from the Docker Compose file. However, host network mode changes fundamental behaviors: containers lose network isolation, port conflicts become host-level conflicts, and the container's 127.0.0.1 becomes the host's 127.0.0.1. The health check scripts inside containers that reference 127.0.0.1 suddenly refer to the host's loopback interface, not a container-local loopback.
Assumption 3: Performance gains would materialize without side effects. The motivation for host networking was to eliminate Docker proxy bottlenecks. While this is a valid optimization for production deployments on dedicated hosts, the assistant underestimated the operational complexity of applying it to a shared development environment where port availability cannot be guaranteed.
The mistake was not in attempting host network mode—that was a reasonable hypothesis to test. The mistake was in the scope of the change. Rather than incrementally testing whether host networking resolved the connection reset issue (perhaps by converting just one service first), the assistant rewrote the entire cluster configuration in one pass, making it difficult to isolate which change caused which failure.
Input Knowledge Required
To fully understand this message, the reader needs knowledge in several domains:
Docker networking modes: Understanding the difference between bridge networking (where each container gets an isolated network stack and ports are mapped through docker-proxy) and host networking (where containers share the host's network namespace and bind ports directly). The key insight is that bridge mode provides port conflict isolation—two containers can both claim port 7000 internally because they exist in separate network namespaces—while host mode collapses all containers into the host's single port space.
YugabyteDB architecture: Knowledge that YugabyteDB uses ports 7000 (master HTTP server), 7100 (tablet server RPC), 5433 (YSQL), and 9042 (YCQL). Understanding that these ports are hard-coded in the yugabyted startup process and cannot be easily reconfigured through environment variables alone.
Linux network debugging: Familiarity with tools like ss (socket statistics), lsof (list open files), and netstat for inspecting port bindings. Understanding that LISTEN sockets on 127.0.0.1 are bound to the loopback interface and are only accessible from the local machine.
The S3 architecture context: Knowing that the test cluster consists of YugabyteDB (metadata store), Kuri storage nodes (data storage with LocalWeb for CAR files), and an S3 frontend proxy (routing layer). Each component has specific port requirements that must not overlap.
Output Knowledge Created
This message produced several valuable pieces of knowledge:
Negative knowledge: Host network mode is not viable in this development environment without first auditing and potentially stopping existing services on ports 7000 and 7100. This is a concrete constraint that will inform all future infrastructure decisions.
A decision point: The assistant commits to reverting to bridge networking. This is not a failure—it is an informed retreat based on empirical evidence. The phrase "keep the connection pool/connection reset fixes we could try" indicates a strategic pivot: instead of fixing the network layer, the assistant will address the connection reset problem through application-level changes (connection pooling, retry logic, or HTTP client tuning).
A debugging methodology: The systematic approach—observing the health check failure, checking port bindings, identifying the specific conflicting ports, and confirming with multiple tools—provides a template for diagnosing similar Docker networking issues in the future.
Documented environmental constraints: The host machine has pre-existing services on ports 7000 and 7100. This information, while not fully explained (the lsof output was truncated), is now part of the session's shared understanding.
The Thinking Process
The reasoning visible in this message follows a clear arc. The assistant begins with a conclusion: "The host network mode is causing conflicts because ports 7000 and 7100 are already used." This is stated with certainty, indicating that the investigation up to this point has been conclusive.
The next sentence reveals the decision: "Let me revert to the bridge network mode but keep the connection pool/connection reset fixes we could try." This is notable for its pragmatism. Rather than doubling down on host networking (by trying to kill the conflicting processes or reconfigure YugabyteDB's ports), the assistant chooses the path of least resistance. The phrase "connection pool/connection reset fixes" refers to earlier discussions about tuning Go's HTTP client, adjusting net.core.somaxconn, and other application-level mitigations that would work within the existing bridge networking setup.
The final sentence—"Actually, let me check what's using 7000/7100"—is a moment of curiosity. The assistant already knows enough to make the revert decision, but takes the extra step to identify the offending processes. This is characteristic of thorough debugging: even when you have enough information to act, gathering complete context helps prevent future surprises.
The output of the ss command confirms the conflict: both ports 7000 and 7100 show LISTEN state on 127.0.0.1. The asterisk in the "Peer Address:Port" column indicates they are accepting connections from any address, though bound to the loopback interface. The assistant does not speculate on what these services are—the priority is to revert and move forward.
Conclusion
Message 1211 is a turning point in the session. It represents the moment when a promising optimization path was abandoned not because the theory was wrong, but because the practical constraints of the environment made it impossible. The assistant's response—swift diagnosis, clear decision-making, and a fallback plan—demonstrates the kind of pragmatic engineering that distinguishes productive development from dogmatic adherence to a chosen approach.
The host network experiment was not wasted effort. It confirmed that Docker's bridge proxy was not the sole cause of connection resets (since the cluster couldn't even start in host mode to test this), and it revealed environmental constraints that will inform future deployment decisions. The decision to revert to bridge networking while pursuing application-level fixes is a classic example of working with the system you have, not the system you wish you had.
In the broader narrative of the coding session, this message marks the end of the network-layer optimization attempt and the beginning of a return to stability. The next steps—committing the CQL batcher, fixing the loadtest header issue, and achieving a clean working single-node cluster—all build on the foundation of a stable, bridge-networked test environment. Sometimes the most important optimization is knowing when to stop optimizing and start shipping.