The Diagnostic Curl: How a Single Command Uncovered Docker's Userland Proxy as a Performance Bottleneck

The Message

In the middle of an intense debugging session for a horizontally scalable S3 architecture, the assistant executed a seemingly trivial command:

curl -s http://172.22.0.3:8078/loadtest/test 2>&1 | head -3

This single line—message 1181 in the conversation—is a diagnostic probe fired at a precise moment of investigative uncertainty. It is barely a sentence, yet it encapsulates an entire chain of reasoning, a hypothesis under test, and a turning point in a performance optimization effort. To understand why this command was written, we must reconstruct the context, the assumptions, the preceding failures, and the architectural knowledge that made this particular curl invocation a critical decision point.

The Context: A System Under Load

The conversation leading up to this message chronicles the development of a distributed S3-compatible storage system built on a three-layer architecture: stateless S3 frontend proxies that route requests to Kuri storage nodes, which in turn store data in a horizontally scalable backend backed by YugabyteDB. The assistant had recently implemented a CQLBatcher—a write-path optimization that collects individual CQL INSERT calls and flushes them in batches to reduce database contention under high concurrency. Initial load tests at 10 workers showed clean results: 115 MB/s throughput with zero corruption. But when the concurrency was raised to 100 workers, the system began producing alarming "connection reset by peer" errors and thousands of verification failures.

The user, observing these failures, offered a succinct hypothesis: "Might be docker-proxy issues?" (message 1178). This suggestion was grounded in deep operational experience—Docker's userland proxy (docker-proxy) is known to become a bottleneck under high connection concurrency because it operates as a userspace TCP forwarder, consuming significant CPU and kernel resources for each connection.

The Reasoning: Why This Particular Command?

The assistant's response to the user's hypothesis reveals the reasoning chain. In message 1179, the assistant first attempted to test directly against a Kuri node by targeting port 7001—the LocalWeb port, which serves CAR files rather than the S3 API. That test was misdirected. In message 1180, the assistant corrected course: "Port 7001 is LocalWeb (CAR files), not S3 API. The S3 API on kuri nodes is 8078, but Docker maps that through the proxy." The assistant then obtained the internal Docker IP address of the kuri-1 container by running docker compose exec kuri-1 hostname -i, which returned 172.22.0.3.

This brings us to message 1181. The command curl -s http://172.22.0.3:8078/loadtest/test is a targeted diagnostic with a precise logical structure:

• Target IP (172.22.0.3): The internal Docker network IP of the kuri-1 container, obtained moments earlier. This address is only reachable from within the Docker network or from containers on the same overlay network.
• Target port (8078): The S3 API port on the Kuri node itself, not the Docker-published port on the host. When accessing via localhost:8078, traffic goes through Docker's port mapping (docker-proxy), which forwards from the host's loopback interface to the container's internal port. By targeting 172.22.0.3:8078 directly, the assistant bypasses the Docker userland proxy entirely.
• Path (/loadtest/test): A simple object path used by the load testing framework. This is a lightweight probe—it tests whether the S3 endpoint is reachable and responsive without the overhead of a full load test.
• The | head -3: A pragmatic safeguard. The assistant only wants to see the first few lines of output to confirm connectivity and basic functionality, not the full response body. The underlying hypothesis being tested is: If the connection resets disappear when accessing the Kuri node directly via its internal IP, then Docker's userland proxy is the bottleneck. If they persist, the issue lies deeper—in the Kuri node's HTTP server, the Go runtime, or the YugabyteDB backend.

Assumptions Embedded in the Command

This diagnostic curl makes several implicit assumptions:

1. Network reachability: The assistant assumes that the host machine can reach the Docker container's internal IP (172.22.0.3). This is true by default on Linux when using Docker's bridge networking, because Docker adds routes to container networks on the host. However, this assumption would be false on macOS or Windows (where Docker runs in a VM), or if network segmentation policies were in place.
2. Port accessibility: The assistant assumes that port 8078 is listening on the container's internal interface and that no container-level firewall (iptables) is blocking it. The earlier netstat output from message 1168 confirmed that kuri-1 was listening on 0.0.0.0:8078, so this assumption was well-grounded.
3. Stateless probe adequacy: The assistant assumes that a simple GET request to /loadtest/test is sufficient to validate the hypothesis. This is reasonable as a first step—if the direct connection fails entirely, there's no point running a full load test. But it does not prove that the direct connection would sustain high concurrency without resets; it only proves basic connectivity.
4. The docker-proxy hypothesis: The command is built on the assumption that the user's diagnosis is correct. The assistant is not testing whether there is a bottleneck, but rather where the bottleneck is located. The curl is designed to isolate the Docker proxy layer as the suspect.

The Outcome: What the Command Revealed

The immediate result of this command is not shown in the message itself—the output was presumably a successful response or an error that informed the next steps. However, the subsequent messages reveal the outcome. In message 1182, the assistant ran a PUT request to http://172.22.0.3:8078/loadtest/direct-test and got a successful response, confirming that direct access works. In message 1183, the assistant attempted to run the full load test against the internal IP but discovered that while the host can reach the container IP, the load test tool running on the host could not sustain the connection—the writes all failed because the internal IP is only reliably reachable from inside the Docker network.

This led to a critical insight: the host could reach 172.22.0.3:8078 for individual requests, but the Docker-proxy bottleneck manifests under concurrent connections. The curl test confirmed basic connectivity but could not reproduce the concurrency issue. The assistant then pivoted to a different approach: switching the entire test cluster to Docker host networking mode, which eliminates the userland proxy entirely by binding container ports directly to the host's network interfaces.

Input Knowledge Required

To understand and execute this diagnostic command, the assistant needed:

• Docker networking internals: Knowledge that Docker uses a userland proxy (docker-proxy) for port publishing by default, and that this proxy can become a bottleneck under high connection concurrency. Also knowledge that Docker containers on a bridge network can be reached via their internal IP from the host.
• The system architecture: Understanding that the S3 API runs on port 8078 inside the Kuri container, that port 7001 is the LocalWeb (CAR) interface, and that the published port mapping on the host (localhost:8078) goes through Docker's proxy layer.
• The load test results: Awareness that at 10 workers the system performed cleanly, but at 100+ workers it produced "connection reset by peer" errors—a pattern consistent with proxy saturation rather than application-level failures.
• Go HTTP server behavior: Understanding that Go's net/http server uses goroutines per connection and can handle thousands of concurrent connections, making it an unlikely source of connection resets at only 100 concurrent requests.

Output Knowledge Created

This single curl command produced several forms of knowledge:

1. Confirmation of basic connectivity: The Kuri node's S3 API was reachable and responsive when accessed directly, ruling out application-level failures or server crashes.
2. Validation of the diagnostic approach: The assistant confirmed that the internal IP was reachable from the host, enabling further investigation.
3. A refined problem statement: The issue was not that the Kuri node was unreachable or crashing, but that the Docker proxy layer could not sustain high concurrency. This shifted the debugging focus from application code to infrastructure configuration.
4. A decision point: The success of the direct access test, combined with the failure of the proxied access at high concurrency, made the path forward clear: switch to host networking mode. This decision materialized in message 1186 when the user instructed "Rewrite the test-cluster to use host network."

Mistakes and Incorrect Assumptions

The diagnostic was not flawless. The most significant incorrect assumption was that the internal Docker IP would be a viable target for the full load test. In message 1183, the assistant discovered that "all writes failing - the internal IP is only reachable from inside the docker network." While the host can reach the container IP on a bridge network, the connection may be unreliable under high concurrency because Docker's bridge network uses NAT and iptables rules that can introduce their own overhead. The assistant initially conflated "reachable for a single curl" with "suitable for a 30-second, 100-concurrent-worker load test."

Additionally, the | head -3 pipe was a minor risk: if the curl command failed with a multi-line error message, head -3 could truncate critical diagnostic information. In practice, this was a reasonable trade-off for brevity, but it reflects an assumption that the response would be either a success (short body) or a simple failure (short error).

The Thinking Process: A Microcosm of Debugging Methodology

This message is a textbook example of the scientific method applied to systems debugging. The assistant:

1. Observed a phenomenon: Connection resets at high concurrency (messages 1174-1177).
2. Received a hypothesis: The user suggested Docker proxy issues (message 1178).
3. Designed an experiment: Test direct access to the container's internal IP, bypassing the proxy (message 1181).
4. Executed the experiment: The curl command.
5. Interpreted the results: Direct access worked, but the internal IP was not suitable for full load tests (messages 1182-1183).
6. Refined the approach: Based on the partial confirmation, switched to host networking mode (message 1186 onward). The beauty of this message lies in its economy. A single curl command, crafted with precise knowledge of the system's architecture, network topology, and failure modes, served as a diagnostic scalpel. It did not solve the problem, but it ruled out several possibilities and pointed decisively toward the correct fix. In the hands of someone who understood the system, six words—curl -s http://172.22.0.3:8078/loadtest/test—were worth a thousand lines of log analysis.