Removing the Middleman: How Host Networking Eliminated the Nginx Proxy in a Distributed S3 Test Cluster

"Now update the nginx config section since we no longer need it for webui proxying (host network mode means direct access)"

This seemingly minor message — a single sentence followed by a file edit — sits at the intersection of performance debugging, infrastructure architecture, and the cascading consequences of a design decision. To understand why this message was written, we must trace the thread of reasoning that led here: a journey that began with false corruption alarms, passed through a CQL batcher optimization, and culminated in the realization that Docker's userland proxy was silently throttling an entire distributed storage cluster.

The Subject Message

The target message, delivered by the assistant in the coding session, reads as follows:

Now update the nginx config section since we no longer need it for webui proxying (host network mode means direct access): [edit] /home/theuser/gw/test-cluster/gen-config.sh Edit applied successfully.

On its surface, this is a straightforward housekeeping operation: a configuration generation script is being updated to remove an nginx section that has become obsolete. But the reasoning compressed into that single sentence encodes a chain of diagnostic insights, a user-driven architectural pivot, and a subtle but important shift in how the test cluster's networking model operates.

The Performance Crisis That Started It All

The session leading up to this message was dominated by load testing. The assistant had recently implemented a CQLBatcher — a write-path optimization that collects individual CQL INSERT calls and flushes them in batches to YugabyteDB, dramatically reducing database contention under high concurrency. Initial results were promising: at 10 concurrent workers, the system delivered ~115 MB/s write throughput with zero corruption. At 100 workers, throughput scaled to ~334 MB/s.

But there was a catch. At 100 and 1000 workers, the load test output was littered with errors: "connection reset by peer," "context canceled," and thousands of verify-read failures. The assistant initially flagged these as "corruption," but closer inspection revealed a different story. The errors were not data mismatches — they were connection failures. The server was dropping connections before verification could complete.

Identifying the Docker Userland Proxy Bottleneck

The user's observation — "Might be docker-proxy issues?" — was the key insight. Docker's default bridge networking mode uses a userland proxy process (docker-proxy) to forward traffic from the host port to the container port. This proxy is a single-threaded userspace process that can become a significant bottleneck under high concurrency. Every connection to port 8078 (the S3 API) was passing through this proxy, which had to accept, buffer, and forward each TCP connection individually.

The assistant confirmed the hypothesis by testing direct access to a Kuri node's internal Docker IP address (172.22.0.3:8078), bypassing the proxy entirely. While that particular test failed because the internal IP was unreachable from the host, the diagnostic direction was clear: the Docker bridge network's port mapping layer was the bottleneck.

The Architectural Pivot: Host Networking

The user's next instruction was decisive: "Rewrite the test-cluster to use host network." This was not a minor configuration tweak — it was a fundamental change to the test cluster's networking model. In Docker's host networking mode, containers share the host's network namespace directly. They bind to ports on the host's own IP addresses, eliminating the bridge network and the userland proxy entirely. No NAT, no port mapping, no proxy process sitting between the client and the service.

The assistant executed this pivot by rewriting the docker-compose.yml file, removing the ports: mapping directives and adding network_mode: "host" to each service. This meant that the S3 proxy on port 8078, the Kuri nodes on their internal ports, and the web UI on port 9010 would all bind directly to the host's network interfaces.

The Cascading Consequence: Why Nginx Became Redundant

This is where the subject message enters the story. The test cluster's original architecture included an nginx container that served as a reverse proxy for the web UI. The web UI — a React-based monitoring dashboard running on kuri-1's port 9010 — was previously accessed through nginx, which handled routing and proxying. This was a sensible design when using Docker's bridge network: nginx could sit on the standard HTTP port and route to the appropriate backend.

But host networking changes the equation entirely. When containers use the host network namespace, every service binds directly to a host port. The web UI on kuri-1's port 9010 becomes directly accessible at http://localhost:9010 without any proxy. Nginx, which was already a lightweight layer, becomes completely unnecessary. The gen-config.sh script, which generated nginx configuration files as part of its setup routine, needed to be updated to reflect this new reality.

The assistant's message — "Now update the nginx config section since we no longer need it for webui proxying (host network mode means direct access)" — captures this realization. The edit removes the nginx configuration generation from the script, simplifying both the codebase and the runtime architecture.

Assumptions and Decision-Making

The assistant made several assumptions in this message. First, that removing nginx was safe and desirable — that no other service depended on the nginx proxy for routing or path rewriting. Second, that the gen-config.sh script's nginx section was solely for web UI proxying and had no other function. Third, that direct access to the web UI on port 9010 was acceptable for the test cluster use case.

These assumptions were reasonable given the context. The test cluster was a development and testing environment, not a production deployment. The nginx layer had been added during an earlier phase when the architecture was still evolving, and its removal simplified the stack. However, the assumption that nginx was only used for web UI proxying could have been wrong if the configuration also handled things like CORS headers, request logging, rate limiting, or SSL termination. In a production scenario, those concerns would still require a reverse proxy layer.

Input and Output Knowledge

To understand this message, a reader needs knowledge of Docker networking modes (bridge vs. host), the role of Docker's userland proxy, the architecture of the test cluster (S3 proxy → Kuri nodes → YugabyteDB), and the function of the gen-config.sh script. They also need to understand the load testing results that motivated the change — specifically, that "connection reset by peer" errors at high concurrency were caused by the Docker proxy bottleneck.

The output knowledge created by this message is a simplified, more performant test cluster configuration. The gen-config.sh script no longer generates nginx configuration. The cluster's networking layer is flattened: clients connect directly to services without intermediate proxies. This change, combined with the host networking mode in docker-compose.yml, eliminates a known bottleneck and prepares the test cluster for higher-throughput experiments.

The Thinking Process

The assistant's reasoning, visible in the message's concise justification, follows a clear chain:

Host networking changes the access model: With network_mode: "host", containers bind directly to host ports. There is no Docker bridge network, no port mapping, and no userland proxy.
Direct access eliminates the need for routing proxies: The web UI on kuri-1's port 9010 is now reachable at localhost:9010 directly. There is no need for nginx to proxy requests to it.
The gen-config.sh script must reflect the new architecture: Since nginx is no longer part of the runtime stack, the script should not generate nginx configuration files. The nginx config section is dead code.
The edit is straightforward: Remove the nginx configuration generation block from the script. No other changes are needed because the script's other sections (Kuri node settings, environment variables) remain valid.

Broader Implications

This message, for all its brevity, illustrates an important principle in infrastructure engineering: architectural decisions cascade. The decision to switch to host networking — driven by a performance bottleneck at high concurrency — did not stop at the docker-compose.yml file. It rippled into the configuration generation scripts, the service dependencies, and the mental model of how the cluster operates. Removing nginx was not an independent choice; it was a logical consequence of a deeper change.

The message also demonstrates the value of questioning intermediate layers. The nginx proxy had been in place for the entire test cluster's lifetime, and it worked fine at low concurrency. It was only when the system was pushed to its limits — 100+ concurrent workers, 300+ MB/s throughput — that the cost of that extra hop became visible. The user's intuition about Docker proxy issues, combined with the assistant's willingness to trace the bottleneck and restructure the networking layer, turned a performance problem into a cleaner architecture.

Conclusion

The message "Now update the nginx config section since we no longer need it for webui proxying" is a small edit with a large backstory. It represents the final step in a chain of reasoning that began with false corruption warnings, passed through batcher optimization and load testing, and arrived at a fundamental networking change. The removal of nginx from the test cluster's configuration is not just cleanup — it is a signal that the architecture has evolved, that bottlenecks have been identified and eliminated, and that the system is ready for the next round of performance testing. In the world of distributed systems engineering, sometimes the most meaningful changes are the ones that remove layers, not add them.