The Diagnostic Curl: How a Single PUT Request Revealed the Docker Proxy Bottleneck

The Message

[bash] echo "test" | curl -s -X PUT -d @- http://172.22.0.3:8078/loadtest/direct-test 2>&1

This is the entirety of message 1182 in a debugging session for a horizontally scalable S3 storage system built on the Filecoin Gateway. At first glance, it appears trivial—a simple curl command that pipes the word "test" into an HTTP PUT request. But in the context of the surrounding conversation, this single command represents a pivotal diagnostic pivot: the moment when the engineer stopped debugging the application layer and started interrogating the infrastructure layer.

The Context That Produced This Message

To understand why this message was written, we must reconstruct the debugging trajectory that led to it. The session had been focused on optimizing the YCQL (YugabyteDB CQL) write path through the implementation of a CQLBatcher—a component that collects individual database insert calls and flushes them in batches to reduce contention under high concurrency. Load tests had shown promising results: at 10 workers, the system achieved roughly 115 MB/s with zero corruption. At 100 workers, throughput scaled to approximately 334 MB/s. But at 100 workers and above, a new class of errors appeared: "connection reset by peer" messages during the verification phase of the load test.

The user, observing these errors, offered a pointed hypothesis in message 1178: "Might be docker-proxy issues?" This single question reframed the entire debugging effort. Until this moment, the assistant had been investigating application-level problems—the batcher configuration, database contention, the health-check logic. The user's suggestion shifted attention to the networking layer, specifically Docker's userland proxy, which is known to become a bottleneck under high concurrency due to its single-threaded connection handling and additional context-switching overhead.

The assistant's response to this hypothesis is the subject of our analysis. But before the subject message, there were two critical precursor steps. In message 1179, the assistant attempted to test directly against the kuri-1 node but mistakenly used port 7001—the LocalWeb port for CAR file access, not the S3 API port. This attempt failed to produce useful results. In message 1180, the assistant corrected course by obtaining the container's internal Docker network IP address via docker compose exec kuri-1 hostname -i, which returned 172.22.0.3. Message 1181 then verified that this IP was reachable on port 8078 (the S3 API port) with a simple GET request.

What the Command Actually Does

The subject message executes a PUT request to http://172.22.0.3:8078/loadtest/direct-test with the body "test". Let us dissect each component:

echo "test": Produces the string "test" followed by a newline, which becomes the request body.
curl -s: Runs curl in silent mode, suppressing the progress meter that would otherwise clutter diagnostic output.
-X PUT: Explicitly sets the HTTP method to PUT, which in S3 semantics creates or overwrites an object at the specified key.
-d @-: Tells curl to read the request body from stdin, which receives the piped output of echo.
http://172.22.0.3:8078/loadtest/direct-test: The target URL. The IP 172.22.0.3 is the container's address on the Docker bridge network, bypassing any port mapping or proxy. Port 8078 is the S3 API endpoint on the kuri storage node. The path /loadtest/direct-test designates the bucket as "loadtest" and the object key as "direct-test".
2>&1: Redirects stderr to stdout, ensuring any error messages from curl are captured in the same output stream. The critical architectural detail is the use of the raw container IP (172.22.0.3) rather than localhost:8078. When accessing the S3 proxy through localhost:8078, the request flows through Docker's port mapping: the Docker engine listens on the host's port 8078 and forwards traffic to the container's port 8078 via the userland proxy (or, if configured, through iptables NAT rules). By using the container's internal IP directly, the assistant bypasses this entire forwarding layer, testing the kuri node's S3 server in isolation from Docker's networking infrastructure.

The Reasoning Process: Hypothesis Isolation

The thinking visible in this message is a textbook example of the scientific method applied to systems debugging. The hypothesis, suggested by the user, is that Docker's userland proxy is the bottleneck causing connection resets at high concurrency. To test this hypothesis, one must design an experiment that isolates the variable in question.

The experiment works as follows: if connection resets occur when accessing the S3 API through localhost:8078 (which traverses Docker's port mapping) but do NOT occur when accessing the S3 API directly at 172.22.0.3:8078 (which bypasses Docker's port mapping entirely), then Docker's proxy is implicated. If connection resets occur in both cases, then the bottleneck lies within the application itself—perhaps in the batcher, the database connection pool, or the HTTP server configuration.

This is a clean experimental design because it holds the application constant while varying only the network path. The same kuri-1 container, running the same code with the same configuration, serves both requests. The only difference is the route the TCP packets take to reach it.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this diagnostic step. First, it assumes that the Docker bridge network is functioning correctly and that the container's IP address (172.22.0.3) is stable and reachable from the host. In a typical Docker setup, this is a safe assumption—containers on the same bridge network can communicate freely, and the host can reach containers via their bridge IPs.

Second, the assistant assumes that the S3 API on port 8078 is bound to all interfaces (0.0.0.0:8078) rather than only to 127.0.0.1:8078. This was verified in message 1168, where netstat -tlnp inside the kuri-1 container showed 0.0.0.0:45283 and other ports, confirming that the S3 server listens on all interfaces. If the server had been bound only to localhost, the direct IP test would have failed with a connection refused error, potentially misleading the diagnosis.

Third, the assistant assumes that the load test's connection resets are reproducible and that a single PUT request is sufficient to validate basic connectivity. This is a reasonable first step—if the direct connection fails even for a single request, there is a fundamental problem. If it succeeds, further testing with higher concurrency is warranted.

A potential mistake in this approach is the use of a single, sequential request. The "connection reset by peer" errors observed during load tests occurred under high concurrency (100+ simultaneous workers). A single curl PUT request does not reproduce the concurrency conditions that trigger the failure. The assistant appears to recognize this—the test is framed as a connectivity check ("can I reach the S3 API directly?") rather than a full concurrency test. The next logical step would be to run the load test tool against the direct IP address, which is indeed what the subsequent messages explore.

Input Knowledge Required

To fully understand this message, one must possess knowledge of several domains:

Docker networking internals: Understanding that Docker by default uses a userland proxy (docker-proxy) to forward ports from the host to containers, and that this proxy can become a bottleneck under high connection counts due to its single-threaded nature and the overhead of copying data between host and container network stacks.

S3 API semantics: Recognizing that PUT requests to /loadtest/direct-test create an object in the "loadtest" bucket with key "direct-test", and that the S3 proxy layer routes these requests to the appropriate storage backend.

The three-layer architecture: Knowing that the system has a stateless S3 frontend proxy on port 8078 (external), which routes to kuri storage nodes that also expose S3 APIs on port 8078 (internal). The direct test targets the internal S3 API on a kuri node, bypassing the frontend proxy entirely.

HTTP debugging techniques: Understanding that curl -s -X PUT -d @- is a minimal, scriptable way to issue HTTP requests without the overhead of higher-level tools, and that piping from echo produces a simple test payload.

Output Knowledge Created

This message produces several pieces of knowledge:

Direct connectivity confirmation: A successful response (which the subsequent context confirms) proves that the kuri-1 container's S3 API is reachable on its internal Docker network IP, independent of Docker's port mapping.
Baseline for comparison: The success of this single request establishes that the application is functioning correctly at the most basic level. Any future failures at higher concurrency can be attributed to resource contention or throughput limits rather than fundamental misconfiguration.
Validation of the diagnostic approach: The ability to reach the container directly confirms that the Docker bridge network is operational and that the container's IP address is correctly resolved. This opens the door to running full load tests against the direct IP to isolate the Docker proxy variable.
A refined mental model: The assistant and user now have a clearer picture of the system's behavior. The "connection reset by peer" errors observed at 100+ workers through localhost:8078 may indeed be caused by Docker's proxy, or they may persist even with direct access—either result would be informative.

The Broader Significance

This message exemplifies a fundamental pattern in distributed systems debugging: the isolation of variables through targeted, minimal experiments. When faced with a complex failure mode (connection resets under high concurrency in a three-tier system with Docker orchestration, a custom S3 proxy, storage nodes, and a distributed database), the engineer must systematically eliminate possible causes.

The user's hypothesis about Docker's proxy was not obvious. Many engineers would have continued down the application-performance rabbit hole—tuning connection pools, adjusting timeouts, profiling database queries. The user's domain knowledge about Docker's userland proxy limitations provided a crucial alternative hypothesis. The assistant's response—obtaining the container IP and issuing a direct request—was the correct experimental design to test that hypothesis.

This moment also reveals the collaborative nature of the debugging process. The assistant had been deep in the application code, implementing batchers and fixing configuration validation errors. The user, observing from a higher level, spotted a pattern that suggested an infrastructure issue. The subject message represents the intersection of these two perspectives: the user's high-level hypothesis and the assistant's ability to execute the precise technical test needed to evaluate it.

Conclusion

Message 1182 is a single curl command, eleven words of shell script. But in its context, it is a carefully designed experiment that tests a specific hypothesis about the location of a bottleneck in a distributed storage system. It demonstrates that effective debugging is not about throwing random fixes at a problem, but about forming hypotheses, designing experiments, and interpreting results. The command's brevity is a feature, not a flaw—it isolates exactly one variable (the network path) while holding everything else constant. Whether the test succeeds or fails, the result will narrow the search space and bring the team closer to the root cause. This is the essence of systematic debugging, captured in a single line of bash.