The Docker Proxy Bottleneck: A Pivotal Realization in Distributed S3 Performance Tuning

The Message

[assistant] Port 7001 is LocalWeb (CAR files), not S3 API. The S3 API on kuri nodes is 8078, but Docker maps that through the proxy. Let me check the docker network setup and try host network mode or direct IP:
[bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec kuri-1 hostname -i 2>&1
172.22.0.3

Context: The Hunt for a Bottleneck

This message, seemingly a brief technical aside in a longer debugging session, represents a critical inflection point in the optimization of a horizontally scalable S3-compatible storage system built on top of YugabyteDB and IPFS. The system under test consists of a three-layer architecture: stateless S3 frontend proxies that route requests to Kuri storage nodes, which in turn store metadata in a shared YugabyteDB cluster. The team had been running increasingly aggressive load tests, and the results were puzzling.

At 10 concurrent workers, the system performed admirably, achieving roughly 115 MB/s throughput with zero corruption or errors. At 100 workers, throughput scaled to approximately 334 MB/s—a promising result—but "connection reset by peer" errors began appearing, and some verification reads failed. At 1000 workers, the system degraded severely, with widespread connection resets and timeouts. The errors looked like application-level failures, but their pattern was suspicious: they appeared as TCP-level connection resets rather than HTTP-level errors, suggesting the problem might not be in the application logic at all.

The user, observing these results, offered a prescient hypothesis: "Might be docker-proxy issues?" This single question reframed the entire investigation. Docker's userland proxy (docker-proxy) is a process that runs on the host and forwards traffic from published ports into containers. While transparent in normal operation, it becomes a bottleneck under high concurrency because every connection must pass through this proxy process, which performs a user-space accept() and connect() for each connection. At scale, the proxy's single-threaded event loop can become saturated, manifesting as connection resets, timeouts, and refused connections.

The Mistaken Assumption and Its Correction

Before this message, the assistant had attempted to isolate the bottleneck by testing directly against a Kuri node, bypassing the S3 frontend proxy entirely. The command used port 7001, which was believed to be the S3 API endpoint on the Kuri node. The results from that test were inconclusive, and the assistant needed to understand why.

The critical realization in this message is the correction of a fundamental architectural misunderstanding: port 7001 is not the S3 API. It is the LocalWeb port, which serves CAR (Content Addressable aRchive) files—a completely different protocol used for IPFS-related data retrieval. The actual S3 API on each Kuri node listens on port 8078, the same port that the S3 frontend proxy exposes externally. However, Docker maps this port through its userland proxy, meaning that even direct tests against localhost:8078 are passing through the same potential bottleneck.

This distinction matters enormously. If the assistant had continued testing against port 7001, the results would have been misleading—they would have measured LocalWeb performance, not S3 API performance. The correction demonstrates a deep understanding of the system's architecture and the importance of testing the right interface.

The Reasoning Process: From Observation to Action

The message reveals a clear chain of reasoning:

  1. Observation: Load tests at high concurrency produce "connection reset by peer" errors.
  2. Hypothesis: The Docker userland proxy might be the bottleneck.
  3. Initial test: Attempt to bypass the proxy by testing directly against a Kuri node on port 7001.
  4. Correction: Port 7001 is LocalWeb, not S3 API. The S3 API (port 8078) goes through the Docker proxy.
  5. New investigation: Check the Docker network setup to understand how containers communicate.
  6. Action: Execute docker compose exec kuri-1 hostname -i to get the container's internal IP address. The command hostname -i returns 172.22.0.3, which is the container's IP address on the Docker bridge network. This information is crucial because it enables a new testing strategy: instead of going through the published port (which traverses the userland proxy), the load test tool could be configured to connect directly to the container's internal IP address on the Docker bridge network, bypassing the proxy entirely.

Assumptions and Their Implications

Several assumptions underpin this message, some explicit and some implicit:

Explicit assumption: The Docker userland proxy is the bottleneck. This is the working hypothesis, and the assistant is gathering evidence to test it. The assumption is reasonable given the pattern of errors (connection resets at high concurrency) and Docker's known behavior.

Implicit assumption: The container's internal IP address is reachable from the host. This is true for Docker bridge networks by default, but it depends on the network configuration. The 172.22.0.3 address confirms this.

Implicit assumption: The S3 API on the Kuri node (port 8078) is functional and would perform better without the Docker proxy. This assumption drives the entire investigation but has not yet been verified.

Implicit assumption: The bottleneck is in the network layer, not in the application or database. This is a narrowing of the investigation space—if the Docker proxy is the problem, then the CQL batcher implementation, the YugabyteDB configuration, and the Kuri node logic may all be adequate.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. Architectural clarification: Port 7001 is LocalWeb (CAR files), not S3 API. This prevents future testing mistakes and clarifies the system's port layout.
  2. Container IP address: 172.22.0.3 for kuri-1. This enables direct testing against the container without going through the Docker proxy, which is the next logical step in the investigation.
  3. A refined testing strategy: Instead of testing through the published port (which traverses the Docker proxy), the team can now test against the container's internal IP address directly. If the connection resets disappear, the Docker proxy is confirmed as the bottleneck.
  4. A decision point: The assistant has committed to investigating the Docker network setup and trying host network mode. This decision shapes the subsequent work, which will involve modifying the Docker Compose configuration to use host networking instead of the default bridge networking with published ports.

The Broader Significance

This message exemplifies a pattern that recurs throughout systems engineering: the most impactful optimizations often come not from tuning application code but from understanding and eliminating infrastructure bottlenecks. The team had already implemented a sophisticated CQL batcher to optimize database writes, achieving impressive throughput at moderate concurrency. But the batcher's benefits were being masked by a lower-level bottleneck in the network stack.

The Docker userland proxy is a particularly insidious bottleneck because it is invisible in normal operation. It only manifests under high concurrency, and its symptoms (connection resets, timeouts) can easily be mistaken for application-level failures. The team had initially suspected data corruption, then database contention, and only through systematic investigation arrived at the network layer.

This message also demonstrates the importance of domain knowledge in debugging. The user's suggestion of "docker-proxy issues" came from experience with Docker's behavior under load—knowledge that cannot be derived from reading logs or source code alone. The assistant's response shows the value of taking such suggestions seriously and investigating them methodically, even when they require correcting one's own assumptions.

What Follows

The container IP address 172.22.0.3 is the key that unlocks the next phase of investigation. With this address, the assistant can run load tests that bypass the Docker proxy entirely, comparing results against the proxy-mediated tests. If the connection resets disappear, the Docker proxy is confirmed as the bottleneck, and the solution is to switch to host networking mode, which eliminates the proxy entirely by binding containers directly to host network interfaces.

The broader lesson is that performance optimization is a process of elimination. Each layer of the system—application logic, database access, network infrastructure—must be tested in isolation to identify where the true bottleneck lies. This message represents the moment when the investigation pivots from the application layer to the infrastructure layer, setting the stage for a significant architectural change that will unlock the system's full throughput potential.