The Moment the Proxy Broke: Debugging TCP Port Exhaustion in a Distributed S3 Architecture

Introduction

In the midst of an intensive debugging session for a horizontally scalable S3 storage system, one message stands out as a critical turning point. Message 749 in the conversation captures the moment when the assistant, having just fixed a cascade of JSON serialization bugs, container configuration issues, and frontend rendering problems, turns to verify that the S3 frontend proxy is properly tracking metrics—and instead discovers a fundamental networking failure. The message is deceptively brief, consisting of a single bash command and its output, but it reveals a serious infrastructure problem: the S3 proxy cannot connect to one of the backend storage nodes because the Linux kernel has exhausted its pool of ephemeral ports. This article examines the reasoning, context, assumptions, and implications of this pivotal debugging moment.

The Message in Full

The assistant writes:

The timestamp is valid. The frontend should work now with the camelCase JSON. Let me also check that we're tracking proper metrics from the S3 frontend proxy as well (since that's where traffic actually hits): `` [bash] docker logs test-cluster-s3-proxy-1 2>&1 | tail -5 2026-01-31T13:56:51.650Z ERROR gw/s3frontend/backend s3frontend/server.go:319 Failed to proxy request {"error": "Put \"http://kuri-2:8078/mybucket/data32/2vnj_khnhwfydu/opvzp2/rjhs/a2r-c_rdch-m/n7zky3w0fe?x-id=PutObject\": dial tcp 172.22.0.4:8078: connect: cannot assign requested address", "backend": "kuri-2"} 2026-01-31T13:56:51.656Z ERROR gw/s3frontend/backend s3frontend/server.go:319 Failed to proxy request {"error": "Head \"http://kuri-2:8078/mybucket/data32/2vnj_khnhwfydu/opvzp2/rjhs/a2r-c... ``

At first glance, this looks like a routine verification check. The assistant has just spent several messages fixing the JSON field name mismatch between Go's PascalCase default serialization and the React frontend's camelCase expectations. The ClusterTopology, ThroughputHistory, and ClusterEvent structs all received JSON tags. The Go build was fixed after a type mismatch in anonymous structs within cluster_metrics.go. Docker images were rebuilt, containers were restarted, and test traffic was generated. The web UI now returns properly formatted camelCase data. Everything appears to be working.

But the assistant knows that the real test is whether the S3 frontend proxy—the actual entry point for client requests—is functioning correctly. This is the moment of truth.

Why This Message Was Written: The Reasoning and Motivation

The assistant's motivation for running this particular command stems from a methodical debugging philosophy: verify each layer independently, then verify the integration. The architecture under construction has three distinct layers:

1. S3 Frontend Proxy (port 8078) — a stateless HTTP server that accepts S3 API requests from clients
2. Kuri Storage Nodes (kuri-1, kuri-2) — the actual storage backends that hold data
3. YugabyteDB — the distributed SQL database for metadata The assistant had already verified that the Kuri nodes themselves were running and returning data through the web UI's RPC interface. The ClusterTopology RPC showed both storage nodes as healthy. The RequestThroughput RPC showed traffic metrics being recorded. The frontend React components were rendering data correctly after the JSON tag fix. However, there was a crucial gap in this verification: all the RPC queries were going through the web UI's Nginx proxy, not through the actual S3 endpoint. The assistant recognized this and explicitly stated the motivation: "Let me also check that we're tracking proper metrics from the S3 frontend proxy as well (since that's where traffic actually hits)." This is the reasoning of someone who understands that a system is only as reliable as its weakest link, and that the S3 proxy is the most critical path in the architecture.

The Discovery: "Cannot Assign Requested Address"

The Docker logs reveal a repeating error pattern. Every request to kuri-2:8078 fails with the error: dial tcp 172.22.0.4:8078: connect: cannot assign requested address. This is a classic Linux networking error that occurs when the operating system cannot allocate a local ephemeral port for an outgoing TCP connection.

The phrase "cannot assign requested address" is distinct from "connection refused" or "connection timed out." It indicates that the problem is at the socket allocation level, not at the remote endpoint. The Linux kernel maintains a pool of ephemeral ports (typically in the range 32768–60999 on many systems) for outgoing TCP connections. When a client application makes many rapid outbound connections, each connection consumes one ephemeral port. If connections are not closed properly, or if they are closed and reopened too quickly, the pool can be exhausted, or the TIME_WAIT state can prevent port reuse.

In the context of an S3 proxy that handles many concurrent requests, this error suggests one of several possible root causes:

• Too many concurrent connections: The proxy is opening connections to kuri-2 faster than they can be closed and recycled.
• Missing connection pooling: The proxy might be creating a new TCP connection for every S3 request instead of reusing existing connections.
• TIME_WAIT accumulation: Each closed connection enters the TIME_WAIT state (typically 60 seconds on Linux), during which the ephemeral port cannot be reused. If the proxy handles thousands of requests per second, it can exhaust the available port range within seconds.
• Container networking limits: Docker's default networking may impose additional constraints on port allocation, especially if the proxy container is configured with a limited connection table size.## Assumptions Made and Their Implications The assistant's actions leading up to this message reveal several assumptions, some of which proved incorrect. Assumption 1: The S3 proxy is working if the web UI shows data. The assistant had spent considerable effort fixing the web UI and verifying that RPC endpoints return correct data through the Nginx proxy on port 9010. The assumption was that if the monitoring layer works, the data path must also work. This message proves otherwise: the web UI was returning metrics from the Kuri storage nodes directly, not from the S3 proxy. The proxy itself was silently failing on every proxied request. Assumption 2: Metrics are being tracked from the proxy layer. The assistant's stated intent was to "check that we're tracking proper metrics from the S3 frontend proxy." This assumes that the metrics collection system (ClusterMetrics in cluster_metrics.go) is integrated with the S3 proxy's request handling. In reality, the metrics being displayed in the web UI were coming from the Kuri storage nodes' internal RPC, not from the S3 proxy. The proxy's failures were invisible to the monitoring dashboard. Assumption 3: The test traffic actually reached the backend. Earlier in the session, the assistant ran a loop of PUT and GET requests against localhost:8078 and then verified that throughput metrics increased. The metrics did increase, which seemed to confirm that the traffic was flowing. However, the Docker logs now reveal that while the proxy received the requests, it failed to forward them to kuri-2. The metrics that increased were likely from the requests that did succeed (going to kuri-1) or from the proxy's own internal counters, not from the complete request path.

Mistakes and Incorrect Assumptions

The most significant mistake revealed by this message is not a coding error but a verification gap. The assistant had been testing the system primarily through the web UI's RPC interface (port 9010) and through direct curl commands to the S3 proxy (port 8078). The curl commands appeared to succeed because they returned HTTP responses—but the responses may have been error pages or partial successes that didn't raise red flags.

The error cannot assign requested address is particularly insidious because it can be intermittent. Some requests might succeed (those routed to kuri-1), while others fail (those routed to kuri-2). If the test script only checked HTTP status codes or response bodies, it could easily miss the fact that half the requests were failing.

Another mistake is the assumption that the Docker Compose networking configuration was correct. The error message shows the proxy trying to connect to 172.22.0.4:8078, which is kuri-2's container IP. The fact that the connection fails at the socket level—not at the HTTP level—suggests a networking configuration issue. Possible causes include:

• The proxy container and kuri-2 container are on different Docker networks
• The docker-compose.yml is missing a network directive connecting the proxy to the storage nodes
• The proxy's FGW_BACKEND_NODES environment variable contains incorrect URLs
• The proxy is using the wrong network interface or has a restricted net.ipv4.ip_local_port_range

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs:

1. Understanding of the three-layer architecture: The system consists of S3 frontend proxies (stateless, handling client connections), Kuri storage nodes (stateful, storing data), and YugabyteDB (metadata storage). The proxy forwards S3 requests to the appropriate storage node.
2. Knowledge of Linux TCP networking: The error "cannot assign requested address" is a socket-level error distinct from "connection refused" or "connection timed out." It indicates local ephemeral port exhaustion or a routing issue.
3. Familiarity with Docker container networking: Docker assigns each container an internal IP address (e.g., 172.22.0.4). Containers on different Docker networks cannot communicate unless explicitly connected.
4. Context from the debugging session: The assistant had just fixed JSON serialization issues, rebuilt Docker images, and verified that the web UI returns correct data. This message represents the next logical verification step.
5. Understanding of the S3 proxy implementation: The proxy uses Go's net/http library to forward requests. The error originates from s3frontend/server.go:319, indicating the proxy's backend request function.

Output Knowledge Created by This Message

This message creates several important pieces of knowledge:

1. The S3 proxy has a connectivity problem to kuri-2: This is the primary finding. The proxy can reach kuri-1 (otherwise the earlier test traffic would have failed entirely) but cannot establish TCP connections to kuri-2.
2. The monitoring dashboard is not representative of actual system health: The web UI showed both storage nodes as "healthy" and displayed throughput metrics, yet the proxy was failing on every request to one of them. This reveals a critical gap in the monitoring system: it monitors the storage nodes themselves, not the proxy's ability to reach them.
3. The test methodology needs revision: The assistant's approach of testing through the web UI and through curl commands was insufficient. Direct verification of the proxy's logs—which this message finally performs—is essential.
4. The error pattern suggests a systemic issue: The repeated errors for every request to kuri-2 (both PUT and HEAD operations) indicate that this is not a transient glitch but a persistent connectivity failure that will affect all traffic.

The Thinking Process Visible in the Message

The assistant's reasoning is evident in the structure of the message. The opening sentence—"The timestamp is valid. The frontend should work now with the camelCase JSON"—serves as a status update, closing the loop on the previous debugging effort. The assistant is mentally checking off items on a verification checklist.

The phrase "Let me also check that we're tracking proper metrics from the S3 frontend proxy as well" reveals the assistant's systematic approach. The word "also" is telling: it acknowledges that the web UI verification was necessary but insufficient. The parenthetical "(since that's where traffic actually hits)" shows the assistant's understanding of the architecture's critical path.

The choice to check Docker logs rather than make another RPC call or curl request is significant. The assistant has exhausted the higher-level verification methods and is now drilling down to the raw container output. This is the debugging equivalent of "going to the source."

The fact that the message ends with a truncated error log—cut off mid-line with "..."—is itself meaningful. The assistant likely saw enough in the first few lines to understand the pattern and didn't need to see more. The repeated errors for both PUT and HEAD requests to kuri-2 establish that the problem is systematic and affects all HTTP methods.

Conclusion

Message 749 is a classic debugging pivot point. The assistant had been working up through the layers of abstraction—fixing JSON serialization, verifying RPC endpoints, checking frontend rendering—and finally reached the bottom layer: the actual request path. The discovery of the "cannot assign requested address" error transforms the debugging session from a frontend/API integration issue into a networking infrastructure problem.

This message also serves as a cautionary tale about verification completeness. The assistant's earlier tests (curl commands showing success, web UI showing metrics) were not lying—they were simply testing different paths through the system. The curl commands succeeded because they tested the proxy's ability to receive requests, not its ability to forward them. The web UI showed metrics because the Kuri nodes reported themselves as healthy, not because the proxy could reach them.

The true value of this message lies in what it prompts next: a deep investigation into Docker networking, TCP connection management, and the proxy's connection pooling strategy. The assistant must now diagnose why the proxy can connect to kuri-1 but not kuri-2, whether the issue is in the Docker Compose configuration, the kernel's TCP settings, or the proxy's own connection handling code. This single log line opens an entirely new debugging chapter.