The Directive That Reshaped an Architecture: "Rewrite the test-cluster to use host network"

Introduction

In the middle of a high-stakes debugging session focused on eliminating false corruption warnings and optimizing YCQL write throughput for a horizontally scalable S3 storage system, a single four-word user message arrived that would fundamentally alter the test infrastructure's networking architecture: "Rewrite the test-cluster to use host network." This message, brief as it was, represented a decisive inflection point in a long session of performance investigation. It was not a casual suggestion or a speculative question—it was a direct command born from a chain of empirical observations, bottleneck analysis, and a growing understanding that the Docker userland proxy had become the critical constraint preventing the system from scaling beyond modest concurrency levels.

To understand why this message was written, one must trace the investigative path that led to it. The session had begun with an apparent data corruption crisis during S3 load testing. After implementing better error classification in the load-testing tool, the team confirmed that what looked like checksum mismatches were actually context deadline timeouts—network-level failures, not data integrity issues. This discovery redirected the entire focus from corruption detection to performance optimization, specifically targeting the YCQL write path. The assistant implemented a sophisticated CQLBatcher that collected individual INSERT calls and flushed them in batches using a worker pool with exponential backoff retries, dramatically improving write throughput. But when the load tests scaled from 10 concurrent workers to 100, a new class of errors emerged: "connection reset by peer" failures that choked the system.

The Evidence That Precipitated the Command

The user's directive did not emerge from intuition alone. It was grounded in a clear sequence of empirical data that the assistant had presented moments earlier. At 10 concurrent workers, the system performed flawlessly: 115 MB/s write throughput, 117 MB/s read throughput, and zero corruption. At 100 workers, throughput tripled to 334 MB/s, but 958 "corruption" events appeared—all of which were actually connection resets, not data mismatches. At 1,000 workers, throughput plateaued at 337 MB/s while connection errors skyrocketed to 3,735. The pattern was unmistakable: the system could not sustain high concurrency because something in the network path was dropping connections under load.

The assistant's diagnosis pointed directly at Docker's userland proxy (docker-proxy), which acts as an intermediary between the host network and container ports when using default bridge networking. Every connection from the load-testing tool running on the host to the S3 proxy container at localhost:8078 had to traverse this proxy, which creates a separate user-space process for each mapped port. At high concurrency, this proxy becomes a bottleneck—it exhausts file descriptors, hits epoll limits, and eventually starts resetting connections. The assistant had even attempted to bypass the proxy by testing directly against a container's internal IP address (172.22.0.3:8078), but that approach failed because the internal IP was only reachable from within the Docker network, not from the host.## The Reasoning Behind the Directive

The user's message was therefore a strategic architectural decision disguised as a simple configuration change. It reflected a deep understanding of the system's bottlenecks and a willingness to make a fundamental infrastructure change rather than applying incremental fixes. The user correctly identified that no amount of tuning within the current networking model—no adjustments to connection pooling, TCP backlog settings, or Go HTTP server parameters—would eliminate the Docker proxy bottleneck. The proxy was a structural limitation, not a tuning parameter. Rewriting the test cluster to use host networking would remove this layer entirely, allowing containers to bind directly to host ports and communicate without any intermediary process.

This decision carried significant assumptions. The primary assumption was that the Docker host networking mode (network_mode: host) would be compatible with the existing multi-container architecture, which included two Kuri storage nodes, an S3 frontend proxy, a YugabyteDB database, and a web UI dashboard. Host networking bypasses Docker's internal DNS resolution and network isolation, meaning containers can no longer refer to each other by service name (e.g., kuri-1:8078). The user implicitly assumed that the assistant could restructure the Docker Compose configuration to handle this—either by switching to Docker Compose's built-in DNS with custom networks, by using explicit IP addresses, or by restructuring the service dependencies. The user also assumed that the performance gains from eliminating the proxy would outweigh the loss of Docker's network isolation and port management convenience.

Mistakes and Incorrect Assumptions

While the directive was correct in its diagnosis, it carried some implicit assumptions that deserve scrutiny. The first assumption was that the Docker userland proxy was the sole bottleneck. In reality, the connection resets at high concurrency could have been caused by multiple factors: the Go HTTP server's default connection handling, the S3 proxy's backend connection pool limits, or even the YugabyteDB CQL driver's connection management. The Docker proxy was the most likely culprit, but not the only possible one. The user's directive assumed a single root cause and prescribed a single solution, which is a form of premature convergence in debugging.

A second assumption was that host networking would be straightforward to implement in the test cluster. The existing docker-compose.yml relied on Docker's internal networking for service discovery—the S3 proxy connected to kuri-1:8078 and kuri-2:8078 by service name, which works seamlessly with bridge networking but not with host networking. Switching to host mode would require either hardcoding IP addresses (brittle and non-portable), implementing a custom service discovery mechanism, or restructuring the compose file to use a dedicated network with explicit IP assignments. The user's message did not specify which approach to take, leaving that architectural decision to the assistant.

Input and Output Knowledge

To understand this message, one needs significant contextual knowledge. The reader must understand Docker's networking modes—specifically the difference between bridge networking (which uses docker-proxy for port mapping) and host networking (which binds containers directly to host ports). One must also understand the load-testing results that preceded the message: the clean 10-worker run, the connection-reset pattern at 100 and 1,000 workers, and the failed attempt to bypass the proxy via internal IP addresses. The message also assumes familiarity with the test cluster's architecture: two Kuri storage nodes, an S3 frontend proxy, YugabyteDB, and the Docker Compose configuration that orchestrates them.

The output knowledge created by this message is a new infrastructure configuration. The assistant would need to produce a rewritten docker-compose.yml that uses network_mode: host for the S3 proxy and Kuri nodes, adjust port mappings accordingly (since host networking means container ports are directly exposed), remove the now-unnecessary nginx web UI proxy, and ensure that service discovery still works—likely by switching to a custom Docker network with explicit IP addresses or by using Docker Compose's depends_on and health checks. The rewrite would also need to update the start.sh and stop.sh scripts to handle the new networking model, and potentially adjust the gen-config.sh script to reflect any changes in how nodes address each other.

The Thinking Process Visible in the Message

Although the user's message is only six words long, the thinking process behind it is visible through the surrounding context. The user had been observing the assistant's debugging session in real time, watching as the load tests progressed from clean 10-worker runs to connection-reset-plagued 100-worker runs. When the assistant suggested that the Docker proxy might be the bottleneck and attempted to test directly against the container's internal IP, the user saw both the diagnosis and the failed workaround. The user's message was not a question ("Should we try host networking?") but a command ("Rewrite the test-cluster to use host network"), indicating a confident conclusion drawn from the evidence.

The brevity of the message is itself telling. It communicates complete trust in the assistant's ability to execute the change without detailed specifications. The user did not need to explain why host networking was the right approach, how to implement it, or what trade-offs to consider. The message assumes shared context, shared technical understanding, and shared goals. It is the kind of message that only works between collaborators who have been deeply engaged in the same problem space, watching the same data, and converging on the same diagnosis.

Conclusion

"Rewrite the test-cluster to use host network" is a masterclass in concise, high-leverage technical direction. It emerged from a rigorous empirical process—false corruption alarms, batcher implementation, throughput measurement, bottleneck identification, and failed workaround attempts. It cut through the complexity of connection resets and throughput plateaus to identify the root cause: Docker's userland proxy. And it prescribed a clean, architectural solution rather than a patch. The message demonstrates that sometimes the most powerful technical contribution is not a long analysis or a complex implementation, but a single, well-timed, precisely targeted directive that reshapes the infrastructure at its foundation.