"Only kuri1 getting traffic seems wrong": A User's Observation That Exposed an Architectural Gap

Subject message (index 2055): [user] Only kuri1 getting traffic seems wrong

Introduction

In the middle of a complex QA cluster deployment session for the Filecoin Gateway's distributed S3 storage system, a single sentence from the user cut through the technical noise: "Only kuri1 getting traffic seems wrong." This seemingly simple observation, delivered after a load test had just completed against one of the cluster's storage nodes, was anything but trivial. It exposed a fundamental architectural misunderstanding in the current deployment—one that had been lurking beneath the surface of an otherwise successful infrastructure build. The message was a catalyst, triggering a chain of investigation, architectural correction, and ultimately the deployment of a proper stateless S3 proxy frontend that enabled true multi-node, cross-node object access. To understand why this message carried such weight, one must understand the context in which it was delivered, the assumptions it challenged, and the architectural insight it represented.

The Context: A Load Test That Revealed Too Little

Moments before the user's message, the assistant had successfully built and executed the ritool loadtest binary against the QA cluster. The load test ran for 30 seconds with 5 concurrent workers, writing and reading objects of sizes between 1KB and 100KB against the S3 endpoint at http://10.1.232.83:8079. The test completed, but the assistant had not yet checked whether traffic was distributed across both kuri nodes. The assistant's focus had been on getting the load test to run at all—there had been earlier struggles with CLI argument ordering, and the successful execution felt like a win. But the user, watching from a higher vantage point, immediately spotted the problem: the load test had been pointed at kuri1's direct S3 endpoint (10.1.232.83:8079), meaning kuri2 received zero traffic. In a properly functioning distributed storage cluster, both nodes should be serving requests.

The user's observation was not merely about load distribution. It was a statement about architectural correctness. The system was supposed to be a horizontally scalable S3-compatible storage cluster with multiple storage nodes. If all traffic went to a single node, the system was not scaling—it was a single-node system wearing a cluster's clothing.

The Reasoning Behind the Message

The user's message reflected several layers of reasoning. First, there was the operational intuition that a load test hitting only one node indicated a configuration problem rather than a load test limitation. The user understood that the load test tool could be pointed at any endpoint, but the architecture should have provided a single entry point that distributed traffic. Second, the user recognized that the current setup—where each kuri node exposed its own S3 endpoint directly—was a deviation from the roadmap's architecture, which specified stateless S3 frontend proxies that route requests to backend storage nodes. Third, the user was thinking about the system as it would behave in production: if a node goes down, traffic should be rerouted; if a node is overloaded, new nodes should share the load. None of this was possible with direct node endpoints.

The user's message also implicitly challenged an assumption the assistant had been operating under: that running kuri nodes as direct S3 endpoints was acceptable for a QA deployment. The assistant had previously configured FGW_BACKEND_NODES on each kuri node to make the topology API work, but this was a cosmetic fix—it made the web UI render correctly without actually enabling cross-node data access. The user saw through this.

Assumptions Exposed

The assistant had made several incorrect assumptions that the user's message helped surface. The first was that the cluster topology API returning healthy nodes meant the cluster was functioning correctly. In reality, the topology data was purely informational; it did not enable any routing or load distribution. The second assumption was that the load test hitting a single node was acceptable because the test was "against the cluster." But there was no cluster-level endpoint—each node was an island. The third assumption was that bitswap (IPFS peer-to-peer block exchange) would handle cross-node retrieval automatically. When the assistant later tested this by connecting the two kuri nodes via their IPFS swarm APIs, cross-node reads still failed because the S3 layer was not designed to fetch blocks from peer nodes—it served only from the local blockstore.

The most significant incorrect assumption was architectural: the assistant had deployed kuri nodes as combined storage-and-S3 endpoints, when the roadmap explicitly called for separate stateless S3 frontend proxies. This distinction matters because a stateless proxy can route requests based on metadata stored in the shared S3 CQL keyspace, whereas a kuri node's built-in S3 server can only serve data it holds locally. The user's message was the catalyst that forced this realization.

The Investigation: Following the Thread

The assistant's response to the user's message began with investigation. Checking kuri2's logs revealed the root cause: when kuri2 received a GET request for an object written to kuri1, it found the object metadata in the shared filecoingw_s3 keyspace (which recorded node_id='kuri_01'), but it could not retrieve the actual block data. The error message was telling: "block was not found locally (offline): ipld: could not find bafkreicq..." The S3 server on kuri2 was looking for the block in its local blockstore, not asking kuri1 for it.

The assistant then attempted a workaround—connecting the two nodes via IPFS swarm to enable bitswap-based block exchange. This failed because the S3 layer does not use bitswap for retrieval; it uses a direct local blockstore lookup. The architecture simply did not support cross-node reads at the S3 level. This was the moment the assistant realized that the user's observation was not about load balancing but about a missing architectural layer.

The Solution: Deploying the S3 Proxy

The correct fix was to deploy the s3-proxy frontend on the head node (10.1.232.82), configured with the list of backend kuri nodes and access to the shared S3 CQL keyspace. The proxy would receive all S3 requests, look up the object's node_id in the shared metadata, and route the request to the correct backend node. This is the architecture specified in the roadmap: a stateless routing layer in front of storage nodes.

The assistant deployed the s3-proxy binary, created a proper systemd service, and configured it with FGW_BACKEND_NODES and RIBS_S3_CQL_HOSTS. After a brief struggle with environment variable naming (the proxy expected FGW_YCQL_HOSTS rather than RIBS_S3_CQL_HOSTS), the proxy came online. A subsequent load test through the proxy showed both kuri nodes receiving traffic—3 PUT/DELETE operations each—confirming that cross-node routing was working.

Knowledge Created

This message and its aftermath created several important pieces of knowledge. First, it established that the QA cluster needed a proper s3-proxy frontend to function as a multi-node system—direct node endpoints are only useful for debugging. Second, it revealed that the S3 metadata in the shared CQL keyspace records the node_id of the storage node holding each object, which is the key piece of information the proxy uses for routing. Third, it demonstrated that bitswap connectivity between nodes is insufficient for S3 cross-node reads because the S3 server does not use bitswap for block retrieval. Fourth, it validated that the ansible deployment roles for the s3_frontend were functional and could be used for future deployments.

Conclusion

The user's seven-word message—"Only kuri1 getting traffic seems wrong"—was a masterclass in concise, high-impact feedback. It was not a complaint about the load test tool or a request for a configuration tweak. It was an architectural observation that revealed a fundamental gap between the deployed system and the intended design. The message forced a correction that transformed the QA cluster from a collection of independent storage nodes into a true distributed system with a proper routing layer. In doing so, it demonstrated that sometimes the most valuable contributions in a technical conversation are not lengthy explanations or complex code changes, but a single, well-timed observation that reframes the entire problem.