Isolating the Docker Proxy Bottleneck: A Pivotal Diagnostic Step in S3 Load Testing

Introduction

In distributed systems debugging, the most valuable insights often come not from complex code changes but from carefully designed experiments that isolate variables. Message 1183 in this coding session represents exactly such a moment: a hypothesis-driven diagnostic step that would determine the entire direction of performance optimization for a horizontally scalable S3 storage cluster. The message itself is deceptively simple—a single command to run a load test against a container's internal IP address—but the reasoning behind it, the context that produced it, and the knowledge it would generate make it a textbook example of systematic performance debugging.

The Message

The target message reads:

Direct access works. Let me run the loadtest directly against kuri-1's internal IP:

``` cd /home/theuser/gw && echo "=== 100 WORKERS - Direct to kuri-1 internal IP ===" && go run ./integrations/ritool/... loadtest run --duration 30s --concurrency 100 --cleanup=false http://172.22.0.3:8078 2>&1 | tail -40 === 100 WORKERS - Direct to kuri-1 internal IP === ═══════════════════════════════════════════════════════════════ S3 LOAD TEST ═══════════════════════════════════════════════════════════════

Endpoint: http://172.22.0... ```

At first glance, this is merely a command invocation with its output header. But to understand its significance, we must examine the debugging journey that led to this exact moment.

The Context: A Session of Performance Troubleshooting

The broader session had been focused on optimizing the YCQL write path for the S3 storage system. Earlier, the team had implemented a CQLBatcher—a component that collects individual CQL INSERT calls and flushes them in batches to reduce database contention under high concurrency. Initial results were promising: at 10 workers, the system achieved approximately 115 MB/s with zero corruption. At 100 workers, throughput scaled to roughly 334 MB/s, but a troubling pattern emerged: "connection reset by peer" errors appeared alongside verification failures.

These errors are classic symptoms of a bottleneck somewhere in the request path. The question was where. The request path in this architecture has several layers:

The load test tool (running on the host machine)
Docker's port publishing mechanism (mapping host port 8078 to container port 8078)
The S3 frontend proxy container
The Kuri storage node containers (kuri-1 and kuri-2)
The YugabyteDB shared metadata store Any of these layers could be the limiting factor. The "connection reset by peer" errors specifically suggested that something upstream was closing connections prematurely—a classic sign of a proxy or load balancer hitting connection limits or timeouts.

The User's Insight

The turning point came in message 1178, when the user observed: "Might be docker-proxy issues?" This was a remarkably precise hypothesis. Docker's userland proxy (docker-proxy) is a small process that forwards traffic from a host port to a container port. While functional, it is known to become a bottleneck under high concurrency because each connection passes through a separate proxy process that handles the TCP proxying in userspace rather than at the kernel level. For high-throughput workloads, Docker's built-in port publishing can introduce significant overhead and connection instability.

The user's question reframed the entire debugging effort. Instead of looking for bugs in the CQLBatcher, database connection pooling, or Kuri node configuration, the team now had a specific hypothesis to test: Is the Docker userland proxy causing the connection resets at high concurrency?## The Reasoning Behind Message 1183

Message 1183 is the assistant's response to the user's hypothesis. The reasoning chain is:

Confirmation of direct access: The assistant first verified that direct access to the container's internal IP works (the preceding message 1182 shows a successful PUT request to http://172.22.0.3:8078/loadtest/direct-test). This established that the Kuri node's S3 API is functional when reached without going through Docker's port mapping.
Designing the experiment: The assistant then designed a controlled experiment: run the same load test that previously produced connection resets (100 workers, 30-second duration) but target the container's internal Docker network IP (172.22.0.3:8078) instead of the host-mapped port (localhost:8078). This is a classic A/B testing approach—keep all variables constant except the one being tested (the network path).
Preserving comparability: By keeping all other parameters identical—100 workers, 30-second duration, the same load test tool, the same cleanup settings—the assistant ensured that any difference in results could be attributed to the network path change. This is methodologically sound. The assistant's reasoning demonstrates a clear understanding of distributed systems debugging principles: isolate variables, test one hypothesis at a time, and design experiments that produce unambiguous results.

Assumptions Made

Several assumptions underpin this message:

Assumption 1: The internal Docker IP is stable and reachable. The assistant assumes that 172.22.0.3 is the correct IP for kuri-1 and that it will remain reachable throughout the test. This was verified in message 1180 by running docker compose exec kuri-1 hostname -i, which returned 172.22.0.3. However, Docker assigns IPs dynamically within the network bridge; if the container were to restart during the test, the IP could change. This is a minor risk but acceptable for a diagnostic test.

Assumption 2: The load test tool can reach the internal Docker network. The load test runs on the host machine, not inside the Docker network. For the test to work, the host must be able to route traffic to 172.22.0.3:8078. This is possible because Docker's bridge network is typically accessible from the host, but it's not guaranteed—some Docker configurations isolate the bridge network. The assistant implicitly assumes the host can reach the container's internal IP.

Assumption 3: The S3 API on port 8078 is the same service. The assistant assumes that the S3 API exposed on port 8078 inside the container is functionally identical to the S3 API exposed through the Docker proxy on the same port. This is almost certainly true—the Docker proxy is transparent at the protocol level—but it's worth verifying that no Docker-level features (like TLS termination or header manipulation) are being bypassed.

Assumption 4: 100 workers is the right concurrency level. The assistant chose 100 workers for this test because that's where the connection resets first appeared. This is a good choice—it targets the boundary where the system transitions from working to failing, which is precisely where diagnostic information is most valuable.

What the Message Reveals About the Thinking Process

The assistant's thinking process is visible in the sequence of actions leading to this message:

Observation: At 100 workers, the system produces "connection reset by peer" errors (message 1175).
Hypothesis generation: The user suggests Docker proxy as the culprit (message 1178).
Preliminary verification: The assistant tests direct access to the internal IP and confirms it works (message 1182).
Experiment design: The assistant designs a controlled comparison test (message 1183).
Execution: The assistant runs the test and begins collecting results. This is textbook scientific debugging: observe, hypothesize, predict, test. The assistant doesn't jump to conclusions or immediately start changing code. Instead, it runs a diagnostic experiment that will either confirm or refute the hypothesis with data. The choice to use tail -40 is also telling—the assistant expects the output to be long and wants to see only the final summary results (throughput, error counts, verification results). This shows an understanding of the load test tool's output format and a focus on the aggregate metrics rather than individual error messages.## Input Knowledge Required To fully understand message 1183, a reader needs familiarity with several concepts: Docker networking: Understanding the difference between Docker's bridge network mode (where containers get internal IPs like 172.22.0.x) and host port mapping (where -p 8078:8078 creates a proxy). The distinction between the Docker userland proxy and direct container access is central to the hypothesis being tested. The S3 storage architecture: The system under test has a three-layer architecture: S3 frontend proxy → Kuri storage nodes → YugabyteDB. The Kuri nodes expose an S3 API on port 8078 internally, while the frontend proxy also listens on port 8078 externally. This dual use of the same port number can be confusing but is clarified by the network context. The load test tool: The ritool loadtest run command generates synthetic S3 PUT and GET requests with verification. It reports throughput, error counts, and checksum mismatches. Understanding that "VERIFY READ ERROR" indicates a failed GET verification (data mismatch or connection error) is crucial for interpreting results. The CQLBatcher: Earlier in the session, the assistant implemented a batch write mechanism for YCQL operations. The batcher was the primary performance optimization being tested, and its interaction with network-level bottlenecks is part of the larger debugging story. Docker's userland proxy limitation: The user's hypothesis that "docker-proxy" might be the bottleneck relies on knowledge that Docker's default port publishing uses a userspace proxy process that can become a bottleneck under high concurrency, especially for workloads with many short-lived connections.

Output Knowledge Created

Message 1183, combined with its results (which appear in subsequent messages), would produce several important pieces of knowledge:

Confirmation or refutation of the Docker proxy hypothesis: If the direct-to-internal-IP test shows clean results (no connection resets, high throughput), it confirms that the Docker proxy is the bottleneck. If it shows the same errors, the bottleneck lies elsewhere—perhaps in the Kuri node itself, the database, or the load test tool.

Baseline performance data: The test establishes a performance baseline for direct container access at 100 workers. This becomes the reference point for further optimization—any improvement must be measured against this baseline.

Network path characterization: The test reveals the overhead introduced by Docker's port publishing. The difference in throughput and error rates between the two paths quantifies this overhead, informing deployment decisions (e.g., using host networking mode for production).

Debugging methodology: The session demonstrates a reproducible technique for isolating network bottlenecks in Docker-based test clusters. This methodology can be applied to other services and configurations.

Potential Mistakes and Incorrect Assumptions

While the reasoning in message 1183 is sound, several potential pitfalls deserve examination:

The internal IP might not be stable across restarts. Docker's bridge network assigns IPs dynamically. If the container restarts during the test (unlikely but possible under heavy load), the IP 172.22.0.3 could change, causing the test to fail midway. A more robust approach would be to use the container name (e.g., http://kuri-1:8078) if Docker's internal DNS is available from the host, or to use docker compose port to resolve the published port dynamically.

The test bypasses the S3 frontend proxy entirely. The architecture is designed with a stateless S3 frontend proxy that routes requests to Kuri nodes. By testing directly against a Kuri node, the assistant is testing a different configuration than the production architecture. The results will show raw Kuri node performance but won't reflect the behavior of the full system including the proxy's routing, health checking, and load balancing logic.

The host's ability to reach the Docker bridge network is not guaranteed. On some Docker configurations (particularly with custom networks or firewall rules), the host cannot directly reach container IPs on the bridge network. The assistant verified this works in message 1182, but the test itself could fail if network conditions change.

Single-node testing doesn't reveal cluster behavior. By testing against only kuri-1, the assistant is measuring single-node performance. In the full cluster, the S3 proxy distributes requests across both nodes, which could change the bottleneck profile. The proxy's health checking and backend selection logic might introduce its own overhead.

Conclusion

Message 1183 captures a pivotal moment in a performance debugging session—the transition from speculation to experimentation. The assistant's decision to run a controlled comparison test, targeting the container's internal IP instead of the Docker-proxied host port, exemplifies the scientific method applied to systems engineering. The user's hypothesis about Docker proxy bottlenecks, combined with the assistant's methodical approach to testing it, demonstrates effective collaboration between human intuition and systematic reasoning.

This message also illustrates a broader principle: when debugging complex distributed systems, the most powerful tool is not more code but better experiments. By isolating variables and designing clean comparisons, the team could determine whether to optimize the application layer (the CQLBatcher, the database connection pool) or the infrastructure layer (Docker networking, proxy configuration). The answer would determine the next steps for the entire project—whether to dive deeper into database optimization or to restructure the deployment for host networking.

In the end, the message is a testament to the value of hypothesis-driven debugging. It's not just about running a command; it's about knowing which command to run, why to run it, and what the results will mean. That is the essence of expert systems troubleshooting.