The Connection Refused That Revealed a Misunderstood Architecture

Message: [assistant] [bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec kuri-1 wget -q -O- http://localhost:8078/ 2>&1 Output: wget: can't connect to remote host: Connection refused

Introduction

In the middle of an intense debugging session for a horizontally scalable S3 architecture, a single command was issued that, on its surface, appears trivial: an assistant used docker compose exec to run wget inside a Kuri storage node container, targeting localhost:8078. The response was a flat "Connection refused." This message, brief as it is, sits at a critical inflection point in the session—a moment where the assistant's mental model of the system architecture collided with reality, and where a simple network diagnostic revealed a subtle misunderstanding of which service owns which port. To understand why this message matters, we must trace the debugging thread that led to it and examine the assumptions baked into that single wget invocation.

The Context: A System Under Test

The assistant had been building and iteratively debugging a test cluster for a horizontally scalable S3 storage system. The architecture, as documented in the project's Docker Compose configuration and README, follows a three-layer design:

S3 Frontend Proxy (stateless, port 8078) — routes S3 API requests to backend storage nodes
Kuri Storage Nodes (kuri-1, kuri-2) — the actual storage backends that hold object data
YugabyteDB — shared database for coordination and object placement tracking The S3 proxy is the entry point for all client traffic. It maintains a pool of backend Kuri nodes, performs health checks against them, and routes requests accordingly. The Kuri nodes themselves expose internal APIs on port 9090 for inter-node communication and CAR file serving, and they also expose an S3-compatible API internally. Moments before this message was sent, the assistant had restarted the entire test cluster with new code that included a CQL batcher optimization. The restart appeared successful—all containers showed "Up" status. But when the assistant ran load tests against the S3 proxy at http://localhost:8078, every write operation failed. A quick curl to the proxy returned a telling error: "Service Unavailable - No healthy backends." This was the crisis that needed solving. The proxy was running, but it considered both Kuri nodes unhealthy. Without healthy backends, the entire S3 service was effectively dead.

The Diagnostic Trail

The assistant's debugging followed a logical progression. First, it checked the proxy logs, which confirmed that both backends had been registered:

Added backend {"id": "kuri-1", "url": "http://kuri-1:8078"}
Added backend {"id": "kuri-2", "url": "http://kuri-2:8078"}

The backends were known to the proxy but were being marked unhealthy. The next logical question was: are the Kuri nodes actually serving traffic? To answer this, the assistant needed to check whether a Kuri node's S3 endpoint was responsive.

Here, the assistant made a reasonable but incorrect assumption: that the Kuri nodes listen on port 8078 for S3 traffic. After all, the proxy routes to http://kuri-1:8078 and http://kuri-2:8078. If the proxy expects the backends to be on port 8078, surely the Kuri nodes must be listening there.

The assistant then executed the command that is the subject of this article: using docker compose exec kuri-1 to run wget -q -O- http://localhost:8078/ from inside the kuri-1 container. The result was "Connection refused."

The Hidden Assumption

The critical assumption embedded in this command is that kuri-1 listens on port 8078. But the Docker Compose configuration tells a different story. The port allocation in the architecture is:

8078 — S3 API (Frontend Proxy) — this is the proxy's port, exposed on the host
9010 — Web UI (kuri-1) — cluster-wide monitoring dashboard
9090 — Internal API (kuri nodes) — for part fetching and inter-node communication The Kuri nodes do not expose port 8078 at all. The proxy's backend URL http://kuri-1:8078 refers to port 8078 inside the Docker network, but that port belongs to the s3-proxy container, not to kuri-1. The proxy routes to itself? No—the proxy's backend pool configuration points to kuri-1:8078 as a logical endpoint. The Kuri nodes may indeed serve S3 traffic on port 8078 internally (within the container), but the port is not mapped to the host. When the assistant execs into the kuri-1 container and tries localhost:8078, it's connecting to the loopback interface inside the container. If the Kuri node's S3 server isn't running on that port, or if it's bound to a different interface (e.g., 0.0.0.0:9090), the connection will be refused. The "Connection refused" was therefore not necessarily a sign that the Kuri node was broken—it was a sign that the assistant was probing the wrong port from the wrong context. The Kuri node's S3 endpoint might be on port 9090 (the internal API port), or the S3 server inside the Kuri container might not be listening on localhost at all.

Why This Matters

This single message is a microcosm of a larger pattern in distributed systems debugging: the ease with which we conflate logical service boundaries with physical network boundaries. The assistant's mental model had the Kuri nodes as S3 endpoints on port 8078, but the actual architecture separates the stateless proxy (port 8078) from the storage nodes (internal API on port 9090). The proxy's backend URLs use port 8078 because that's the port the proxy exposes to clients—but internally, the proxy may translate that to a different port when communicating with backends, or the backends may indeed listen on 8078 inside the container but not on the loopback interface.

The debugging session that followed this message would need to pivot: instead of checking if the Kuri node is listening on 8078, the assistant needed to check:

What port the Kuri node's S3 server actually binds to
Whether the health check endpoint is correctly configured
Whether the proxy's health check logic is compatible with the Kuri node's API

Output Knowledge and Lessons

The output of this message—"Connection refused"—is a negative result, but it is valuable negative knowledge. It tells us that the Kuri node is not serving S3 traffic on port 8078 on its loopback interface. This eliminates one hypothesis (that the Kuri node is fully operational on the expected port) and forces the investigation toward other possibilities: the port mapping is wrong, the health check endpoint is different, or the Kuri node's S3 server failed to start.

For someone reading this session log, the message teaches an important lesson about debugging distributed systems: when a proxy reports "no healthy backends," the first instinct should be to verify the health check mechanism itself, not just the backend's raw availability. The assistant's command was a step in that direction, but it was aimed at the wrong port. A more targeted diagnostic would have been to check the Kuri node's actual configured listen port (from its settings file or logs) and to probe the health check endpoint specifically.

Conclusion

The message wget: can't connect to remote host: Connection refused is deceptively simple. It represents a moment where a developer's mental model of a multi-service architecture was tested against reality and found to have a subtle flaw. The assistant assumed that because the proxy routes to kuri-1:8078, the Kuri node must be listening on port 8078. But in a layered architecture with separate stateless proxies and storage nodes, port ownership is not always what it seems. This message is a reminder that every "Connection refused" is an invitation to re-examine our assumptions about which service runs where, on which port, and on which interface.