The Docker Network Rewrite: How One Line in a Configuration File Unblocked 334 MB/s of S3 Throughput

The Message

In the middle of a high-stakes debugging session for a horizontally scalable S3 storage system, the assistant sent a message that was deceptively simple in its form but represented a critical turning point in the investigation:

[assistant] [write] /home/theuser/gw/test-cluster/docker-compose.yml
Wrote file successfully.

That is the entirety of the message at index 1188. No fanfare, no explanation, no diff output. Just a tool invocation and a confirmation. Yet this single file-write operation—rewriting the Docker Compose configuration to use host networking—was the culmination of an entire session's worth of detective work, load testing, bottleneck analysis, and architectural reasoning. It was the moment when a hypothesis about performance degradation was translated into concrete infrastructure change.

The Context: A Session of Discovery and Optimization

To understand why this message was written, one must understand the session that preceded it. The assistant had been deep in performance optimization work on a distributed S3 storage system built on top of YugabyteDB (YCQL). The system architecture consisted of stateless S3 frontend proxies routing requests to Kuri storage nodes, which in turn stored metadata in a shared YugabyteDB cluster.

The session began with a concerning finding: load tests were showing "corruption" errors. After adding better error classification to the load testing tool—distinguishing between actual checksum mismatches and context deadline timeouts—the team confirmed that no real data corruption was occurring. The "verify errors" were simply timeouts at the end of test runs. This discovery freed the team to focus on the real problem: throughput optimization.

The assistant implemented a CQLBatcher in the database layer that collected individual CQL INSERT calls and flushed them in batches, dramatically reducing database contention under high concurrency. With the batcher in place, the team ran systematic load tests at 10, 100, and 1000 concurrent workers. The results were telling:

The Hypothesis: Docker's Userland Proxy as Bottleneck

The assistant's analysis identified three potential culprits: Docker's userland proxy becoming a bottleneck, TCP socket exhaustion or ephemeral port limits, and Go's default HTTP client connection pooling. But the user had a sharper intuition: "Might be docker-proxy issues?"

This was the key insight. Docker, by default, uses bridge networking for containers. When a container port is published to the host (ports: "8078:8078"), Docker spawns a userland proxy process (docker-proxy) that forwards traffic from the host port to the container's internal IP. This proxy introduces a context switch for every packet, maintains connection state, and becomes a significant bottleneck under high concurrency. At 100+ concurrent connections doing S3 PUT and GET operations at hundreds of megabytes per second, the userland proxy was simply dropping connections.

The assistant confirmed this hypothesis by testing direct access to a Kuri node's internal Docker IP (172.22.0.3:8078), bypassing the proxy. Direct access worked, but running the load test from outside the Docker network against the internal IP was not feasible—the internal IP was only reachable from within the Docker network. This confirmed that the proxy was the choke point, but also highlighted that the solution needed to be architectural, not just a workaround.

The Decision: Host Networking Mode

When the user said "Rewrite the test-cluster to use host network," they were asking for a fundamental change to the test cluster's networking model. Instead of each container living on a private Docker bridge network with ports forwarded through the userland proxy, host networking mode makes containers share the host's network stack directly. Container ports become host ports, with no proxy layer in between.

This decision was not made lightly. Host networking has trade-offs: containers lose network isolation, port conflicts become possible, and the setup is less portable. But for a performance test cluster where the goal is to measure raw system throughput without infrastructure overhead, it was the right call.

The assistant read the existing docker-compose.yml (message 1187) to understand the current port mappings and architecture, then wrote the new version (message 1188). The new file would have replaced network_mode: bridge (the default) with network_mode: host for each service, or equivalently removed the bridge network definition and set network_mode: host on each container. The port mappings would be simplified since host networking means container ports are directly exposed without explicit mapping.

The Ripple Effects

The file write at message 1188 was just the beginning. In the immediately following messages, the assistant updated gen-config.sh to use new port assignments (message 1189), removed the nginx web UI proxy configuration since host networking made it unnecessary (message 1191), and updated the README documentation (messages 1193–1195). Each of these changes cascaded from the single decision to switch networking modes.

The nginx removal was particularly interesting. In the bridge networking setup, an nginx container had been used as a reverse proxy for the web UI, translating between the Docker network and the host. With host networking, the web UI container could serve directly on the host port, eliminating an entire layer of infrastructure. This simplification was a side benefit of the host networking change that reduced the system's complexity and removed another potential bottleneck.

Input Knowledge Required

To fully understand this message, a reader would need knowledge of several domains:

Docker networking internals: Understanding the difference between bridge mode (where each container gets an isolated network stack and ports are forwarded through docker-proxy) and host mode (where containers share the host's network stack directly). More specifically, knowing that Docker's userland proxy is a separate process that handles port forwarding and can become a bottleneck under high connection counts.

Distributed S3 architecture: The test cluster had a three-layer design: S3 frontend proxy → Kuri storage nodes → YugabyteDB. The proxy layer was where the bottleneck manifested, but the root cause was in the networking layer between the host and the containers.

Load testing methodology: The assistant had run systematic tests at 10, 100, and 1000 concurrency levels, measuring throughput and error rates. Understanding that "corruption" errors were actually connection resets required reading the error messages carefully rather than relying on aggregate counts.

TCP/IP networking fundamentals: Connection resets at high concurrency can be caused by many factors—ephemeral port exhaustion, socket backlog limits, proxy process saturation. The assistant correctly identified the userland proxy as the most likely culprit based on the pattern of errors.

Output Knowledge Created

This message produced a new docker-compose.yml file that reconfigured the entire test cluster's networking. The direct output was a configuration file, but the knowledge created extended beyond that:

  1. A validated bottleneck diagnosis: The host networking change was a test of the hypothesis that Docker's userland proxy was the bottleneck. If throughput improved and connection resets disappeared, the hypothesis was confirmed.
  2. An infrastructure pattern for high-throughput testing: Future developers working on this system would know that for performance testing, host networking is preferred over bridge networking.
  3. A simplified architecture: Removing the nginx proxy layer reduced complexity and eliminated a potential secondary bottleneck.
  4. Documentation updates: The README changes (messages 1193–1195) ensured that the reasoning behind the host networking choice was captured for future reference.

The Thinking Process

The assistant's reasoning is visible in the messages leading up to the file write. After presenting the load test summary (message 1185), the assistant explicitly listed "Docker's userland proxy becoming a bottleneck" as the first likely cause. The user's follow-up question ("Might be docker-proxy issues?") showed independent confirmation of this hypothesis.

The assistant then tested direct access to a Kuri node's internal IP (messages 1180–1183), finding that direct access worked but was not practical for external load testing. This experiment confirmed that the proxy was the bottleneck but also showed that simply bypassing it wasn't a viable long-term solution—the architecture needed to change.

The decision to use host networking rather than a less invasive solution (like increasing proxy connection limits or using --network=host on individual containers) reflected a preference for architectural fixes over configuration tweaks. The assistant could have tried tuning the Docker daemon's proxy settings or increasing net.core.somaxconn on the host, but those would have been partial measures. Host networking eliminated the proxy entirely.

Conclusion

Message 1188 appears, on its surface, to be the most mundane of engineering actions: a file write. But in the context of the session, it represents the moment when diagnosis became action, when hypothesis became infrastructure. The assistant had traced a performance bottleneck from load test numbers through error message analysis to Docker networking internals, and had arrived at a concrete, actionable solution. The file write was the culmination of that reasoning chain.

The broader lesson is that in distributed systems debugging, the most impactful changes are often not in the application code but in the infrastructure layer. The CQL batcher improved throughput from an unmeasured baseline to 115 MB/s at 10 workers. But the networking change—a single configuration file rewrite—was what would unlock the path beyond 334 MB/s to whatever the system's true ceiling might be. Sometimes the biggest performance gains come not from writing better code, but from removing the layers that get in its way.