The Moment of Reckoning: Restarting a Distributed Storage Cluster with Host Network Mode

The Message

cd /home/theuser/gw/test-cluster && ./start.sh /data/fgw2
========================================
FGW Test Cluster (2 Storage Nodes)
========================================

Data directory: /data/fgw2

Architecture:
  - S3 Proxy: S3 API (:8078) - routes to Kuri nodes
  - kuri-1: LocalWeb (:7001) + Web UI (:9010)
  - kuri-2: LocalWeb (:7002)
  - YugabyteDB: Shared metadata

✅ Docker image fgw:local exists

📁 Initializing data directories...
Initializing test cluster data directories in /data/fgw2
Setting permissions (may require sudo for existing...

On its surface, this message is mundane: an assistant running a shell script to start a Docker Compose-based test cluster. But in the context of the broader conversation, it represents a critical inflection point—the moment when a multi-hour debugging session reached its climax and a fundamental architectural decision was executed. The assistant had spent the preceding messages investigating performance bottlenecks, implementing a CQL write batcher, running load tests at escalating concurrency levels, and diagnosing a frustrating pattern of "connection reset by peer" errors. Now, with a single command, the entire test infrastructure was being torn down and rebuilt on a different networking foundation.

The Debugging Journey That Led Here

To understand why this message matters, we must trace the debugging arc that preceded it. The assistant had been building a horizontally scalable S3-compatible storage system with a three-layer architecture: stateless S3 frontend proxies routing requests to Kuri storage nodes, which in turn stored metadata in a shared YugabyteDB cluster. After implementing a high-throughput CQL batcher to optimize write performance, the assistant ran load tests at three concurrency levels: 10, 100, and 1000 parallel workers.

The results were revealing and frustrating. At 10 workers, the system performed flawlessly: 115 MB/s write throughput, 117 MB/s read throughput, and zero corruption. At 100 workers, throughput tripled to 334 MB/s—but 958 "corruption" errors appeared. At 1000 workers, throughput plateaued at 337 MB/s with 3,735 errors. Crucially, the assistant had already fixed the load test to distinguish between genuine data corruption (checksum mismatches) and connection failures, and these errors were all the latter: read: connection reset by peer.

The pattern was unmistakable. The system could handle moderate concurrency beautifully, but beyond a certain threshold, connections were being violently terminated. The throughput scaling from 10 to 100 workers showed the batcher was working—the system could handle the load—but something in the networking layer was breaking under pressure.

The Docker Proxy Hypothesis

When the user suggested "Might be docker-proxy issues?", they identified the likely culprit. Docker's default bridge networking mode uses a userland proxy process (docker-proxy) to forward traffic from the host's network namespace into the container. This proxy is a single-threaded userspace process that can become a bottleneck under high connection counts. Every HTTP request from the load test client to the S3 proxy at localhost:8078 was passing through this proxy, which had to accept the connection, inspect the packet, and forward it to the container's internal IP.

The assistant confirmed this hypothesis through experimentation. When they ran the load test directly against a Kuri node's internal Docker IP (172.22.0.3:8078), all writes failed—because that IP was only reachable from within the Docker network. The test client running on the host had no route to that address. This confirmed that all external traffic must go through the Docker proxy, making it an unavoidable bottleneck.

The user's directive was clear and decisive: "Rewrite the test-cluster to use host network." This was not a suggestion to experiment further or to tune parameters—it was an architectural mandate. The assistant executed it across multiple files: the docker-compose.yml was rewritten to remove bridge networks and port mappings, replacing them with network_mode: host; the gen-config.sh script was updated to assign distinct ports to each Kuri node (kuri-1 on 8079, kuri-2 on 8080, the S3 proxy on 8078); and the README was updated to document the new architecture.## The Assumptions Embedded in the Start Command

The message ./start.sh /data/fgw2 is deceptively simple. It encodes several critical assumptions that deserve examination.

Assumption 1: The Docker image exists and is current. The start script checks for fgw:local and confirms "✅ Docker image fgw:local exists." This assumes that the most recent code changes—including the CQL batcher implementation, the load test fixes, and any other modifications—have been baked into a Docker image. If the assistant had forgotten to rebuild the image after making source changes, the cluster would start with stale code, and the host network test would be meaningless. The assistant had previously run docker build -t fgw:local . to ensure the image was up to date.

Assumption 2: The data directory is clean and compatible. The script passes /data/fgw2 as the data directory. This assumes that the previous cluster's data (from the bridge networking era) is either compatible with the new configuration or will be overwritten. In practice, the gen-config.sh script regenerates settings.env files for each Kuri node, and the start script initializes data directories. But if the YugabyteDB keyspaces or CQL schema had changed between runs, there could be subtle incompatibilities.

Assumption 3: Port conflicts are resolved. Host network mode means containers bind directly to host ports. The new configuration assigns kuri-1 to port 8079, kuri-2 to port 8080, and the S3 proxy to port 8078. If any of these ports were occupied by lingering processes from the previous cluster (which used different port mappings), the start would fail. The assistant had just run ./stop.sh /data/fgw2, which should have stopped all containers, but there's always the risk of orphaned processes or port conflicts with non-Docker services.

Assumption 4: The host networking stack can handle the load. The entire motivation for switching to host networking was to eliminate the Docker proxy bottleneck. But this assumes that the host's own TCP/IP stack, socket limits, and network configuration are adequate for the intended concurrency levels. If the real bottleneck was something else—ephemeral port exhaustion, somaxconn limits, or the Go HTTP server's connection handling—then host networking would only partially address the problem.

What the Message Does Not Show

The output in this message is truncated. We see "Setting permissions (may require sudo for existing..." and then the message ends. The reader is left hanging—did the start succeed? Did the cluster come up cleanly? This truncation is itself meaningful: it reflects the real-time, iterative nature of the coding session. The assistant was not writing a polished report; they were executing commands and observing output as it streamed in. The conversation continues with subsequent messages that show the cluster starting successfully, the S3 proxy becoming healthy, and load tests running with dramatically improved results.

The Deeper Significance: From Investigation to Action

This message marks the transition from diagnosis to treatment. The preceding messages were about understanding why the system was failing at high concurrency. The assistant had run load tests, examined logs, compared direct vs. proxied access, and built a strong circumstantial case against Docker's bridge networking. But until this moment, no corrective action had been taken on the infrastructure itself. The batcher was a software optimization; the host network switch was an infrastructure overhaul.

There is also a subtle but important dynamic between the user and the assistant. The user's intervention—"Rewrite the test-cluster to use host network"—was not a suggestion but a directive. The assistant had been exploring the problem space, presenting findings, and offering recommendations ("For production, use host networking or native port mapping"). The user cut through the analysis and commanded the fix. This is a pattern common in pair debugging: the technical lead identifies the root cause and delegates the implementation, trusting the engineer (or AI assistant) to execute across all the affected files.

The Knowledge Flow

Input knowledge required to understand this message: The reader needs to know that Docker has two primary networking modes—bridge (default, with a userspace proxy) and host (direct binding to host interfaces). They need to understand that the "connection reset by peer" errors in the load tests were caused by the Docker proxy becoming a bottleneck under high concurrent connections. They need to know the architecture of the system: S3 frontend proxy routing to Kuri storage nodes with shared YugabyteDB metadata storage. And they need to know that the test cluster had been running with bridge networking, with port mappings like 8078:8078 that forced all traffic through docker-proxy.

Output knowledge created by this message: The message creates a new baseline state for the test cluster. After this start completes, the cluster will be running with host networking, each Kuri node will have distinct ports directly bound to the host, and the Docker proxy bottleneck will be eliminated. This new state enables the next round of load testing to determine whether the connection reset errors are resolved, or whether additional tuning is needed at the application or OS level.