The Pivot Point: Diagnosing Docker Proxy Bottlenecks in a Distributed S3 Load Test

The Message

[assistant] All writes failing - the internal IP is only reachable from inside the docker network. Let me instead add a --network option or use docker compose exec to run the test from inside the network. Actually, the issue is the docker-proxy - let me check if we can use host network mode: [bash] cd /home/theuser/gw && echo "=== Summary ===" && echo "10 workers: 115 MB/s write, 117 MB/s read, 0 corruption" && echo "100 workers: 334 MB/s write, 333 MB/s read, 958 corruption (connection resets)" && echo "1000 workers: 337 MB/s write, 376 MB/s read, 3735 corruption (connection resets)" === Summary === 10 workers: 115 MB/s write, 117 MB/s read, 0 corruption 100 workers: 334 MB/s write, 333 MB/s read, 958 corruption (connection resets) 1000 workers: 337 MB/s write, 376 MB/s read, 3735 corruption (connection resets)

Context and Motivation

This message sits at a critical inflection point in a high-stakes debugging session. The assistant had just spent several messages implementing and testing a CQLBatcher—a performance optimization for the YCQL (YugabyteDB CQL) write path in a horizontally scalable S3 storage system built on the Filecoin Gateway platform. The batcher was designed to aggregate individual CQL INSERT calls into batches of up to 15,000 entries, flushed within 10–30 milliseconds by a pool of 8 workers with exponential backoff retries. The goal was to reduce database contention under high concurrency and push throughput beyond what the system could achieve with individual writes.

The load test results were tantalizing but puzzling. At 10 concurrent workers, the system performed flawlessly: 115 MB/s write throughput, 117 MB/s read throughput, zero corruption. At 100 workers, throughput nearly tripled to 334 MB/s—an impressive scaling curve—but 958 "corruption" events appeared, all of which were actually connection reset by peer errors rather than genuine checksum mismatches. At 1000 workers, throughput plateaued at 337 MB/s (barely above the 100-worker result) while corruption events skyrocketed to 3,735. The system was hitting a ceiling, and the bottleneck was not where anyone initially suspected.

The user had just suggested the root cause in message 1178: "Might be docker-proxy issues?" This single question reframed the entire investigation. The assistant had been assuming the bottleneck was in the application layer—the batcher configuration, the database connection pool, or the S3 proxy routing logic. But the user's intuition pointed to something much more mundane and infrastructural: Docker's userland proxy, which by default translates container ports to host ports through a process running in userspace rather than through kernel-level NAT.

The Reasoning Process Visible in the Message

The message reveals a rapid, three-stage reasoning process compressed into a single utterance.

Stage 1: Recognizing the failed experiment. The assistant had just attempted to bypass the S3 proxy layer entirely by running the load test tool directly against kuri-1's internal Docker IP address (172.22.0.3:8078). This was a clever isolation strategy: if the corruption/connection-reset pattern persisted when hitting the Kuri node directly, the bottleneck was in the Kuri application or database layer. If it disappeared, the bottleneck was in the S3 proxy or Docker networking. The experiment failed immediately—"All writes failing"—because the load test tool runs on the host machine, and Docker's internal IP addresses are only reachable from within the Docker bridge network. The assistant's first reaction was to consider workarounds: "Let me instead add a --network option or use docker compose exec to run the test from inside the network."

Stage 2: The pivot. The very next sentence—"Actually, the issue is the docker-proxy"—marks the moment of insight. The assistant connects the user's suggestion to the experimental failure and realizes that the workaround is unnecessary. The bottleneck hypothesis has shifted from application logic to infrastructure. The phrase "let me check if we can use host network mode" signals the new direction: instead of trying to route around Docker's proxy, eliminate it entirely by switching from Docker's default bridge networking (which requires the userland proxy for port mapping) to host networking (which binds containers directly to the host's network stack).

Stage 3: Synthesizing the evidence. The assistant then outputs a concise three-line summary of all load test results. This summary is not merely informational—it is an argument. The data shows a clear pattern: throughput scales from 115 MB/s to 334 MB/s when moving from 10 to 100 workers (a 2.9× improvement), but then flatlines at 337 MB/s when moving to 1000 workers. Meanwhile, corruption events appear only at the higher concurrency levels and grow superlinearly (958 at 100 workers, 3,735 at 1000 workers). The plateauing throughput combined with the explosion of connection resets is the classic signature of a proxy bottleneck: the proxy can handle a certain number of concurrent connections, and beyond that, connections are dropped or reset.

Assumptions and Their Consequences

The assistant's initial assumption was that the performance ceiling was an application-layer problem. This assumption was reasonable—the batcher was new code, the YCQL driver had known behaviors under load, and the S3 proxy routing logic was complex. The assistant had spent considerable effort tuning the batcher parameters (batch size, flush interval, worker pool count, exponential backoff) and had fixed a configuration bug (RetrievableRepairThreshold > MinimumReplicaCount) that prevented Kuri nodes from starting. All of this work assumed the bottleneck was in the software.

The user's suggestion about Docker proxy issues challenged this assumption at a deeper level. Docker's userland proxy (docker-proxy) is a small process that listens on the host port and forwards connections to the container's port. It is single-threaded by default and can become a bottleneck under high concurrency, particularly for workloads with many short-lived connections like HTTP load testing. The assistant's experiments confirmed this: the "connection reset by peer" errors are exactly what happens when the proxy's connection queue overflows and it starts rejecting new connections.

The mistake was not in the assistant's implementation—the batcher was working correctly, and the throughput numbers at 10 workers proved the write path was sound. The mistake was in the investigative priority. The assistant had jumped to the most complex possible explanation (application-layer bugs, database contention, corruption) when the simplest explanation (infrastructure bottleneck) was equally plausible. This is a classic debugging pitfall: when you've just built something new, you tend to assume the bug is in your new code.

Input Knowledge Required

To understand this message, the reader needs several pieces of context. First, the architecture of the test cluster: there are three layers—S3 proxy (port 8078), Kuri storage nodes (each with their own S3 API on port 8078 internally), and a shared YugabyteDB instance. The load test tool (ritool/loadtest) generates objects, writes them via PUT requests, then reads them back to verify integrity. Second, Docker networking basics: by default, Docker uses bridge networking with a userland proxy for port mapping, which can become a bottleneck under high concurrency. Host networking bypasses this by binding the container directly to the host's network interface. Third, the earlier investigation into false corruption warnings: the team had already confirmed that the "verify errors" were timeouts rather than actual checksum mismatches, which is why the message refers to "958 corruption (connection resets)" rather than treating them as genuine data corruption.

Output Knowledge Created

This message creates several important pieces of knowledge. The three-line summary provides a concise, quantitative characterization of the system's performance envelope under the current configuration. It establishes a baseline: 115 MB/s at low concurrency, scaling to ~334 MB/s at moderate concurrency, with a hard ceiling imposed by Docker's networking layer rather than by the application or database. The message also documents a failed experimental approach (direct internal IP testing) and the reasoning that led to abandoning it, which prevents future investigators from repeating the same dead end.

Most importantly, the message crystallizes the decision to switch to host networking. This is a non-trivial infrastructure change with implications for security (host networking reduces container isolation), port management (port conflicts become possible), and configuration (the Docker Compose file must be rewritten). The message implicitly argues that the performance gains justify these trade-offs, and it sets the agenda for the next phase of work: implement host networking, re-run the load tests, and verify that the connection reset errors disappear.

The Broader Significance

This message exemplifies a pattern that recurs throughout systems engineering: the most impactful optimizations often come not from tuning application code but from removing infrastructure bottlenecks. The batcher optimization was worthwhile—it enabled the 3× throughput scaling from 10 to 100 workers—but the Docker proxy was the true ceiling. The assistant's willingness to pivot from application debugging to infrastructure reconfiguration, prompted by a single user question, demonstrates the kind of flexible, hypothesis-driven debugging that distinguishes effective systems work.

The message also illustrates the importance of clean experimental design in performance testing. By running the same workload at three different concurrency levels (10, 100, 1000 workers), the assistant created a clear dose-response curve that revealed the bottleneck's nature. The plateauing throughput combined with the explosion of connection resets told a story that no single data point could have revealed. And the assistant's instinct to verify the hypothesis by attempting to bypass the proxy—even though that experiment failed due to network topology—was methodologically sound.

In the end, this message is about a moment of diagnostic clarity. The assistant had been deep in the weeds of batcher tuning, configuration fixes, and corruption analysis. The user's suggestion and the failed direct-IP experiment together produced a gestalt shift: the problem was not in the code but in the infrastructure. The summary line that closes the message—with its clean numbers and parenthetical "(connection resets)"—is the artifact of that shift, a concise record of what was learned and what must be done next.