The Cross-Node Read Problem: Diagnosing a Missing Routing Layer in a Distributed S3 Architecture

Introduction

In distributed systems, the gap between "working on one node" and "working across many nodes" is often where the most subtle and consequential bugs hide. Message 2057 in this coding session captures exactly such a moment: a developer runs a load test against a multi-node S3-compatible storage cluster, discovers that only one node received traffic, and then attempts a simple cross-node read test that reveals a fundamental architectural gap. The test writes an object to node kuri1 and tries to read it from node kuri2, expecting the shared metadata layer to make the object accessible from any node. Both reads return empty. This single message is the pivot point where the team realizes that their distributed storage system, while functionally complete on individual nodes, lacks the routing infrastructure necessary for true horizontal scalability.

The Message in Full

The assistant writes:

Kuri2 got no traffic - loadtest was only hitting kuri1 directly. The issue is we don't have a load balancer in front. Let me run a test hitting both nodes alternately, and also check if data written to kuri1 is accessible from kuri2 (shared S3 keyspace):

```bash # Test: Write to kuri1, read from kuri2 (shared S3 metadata) echo "Testing cross-node access..."

# Write a test object to kuri1 echo "test-data-from-kuri1" | curl -s -X PUT -d @- http://10.1.232.83:8079/testbucket/crossnode-test.txt echo "Written to kuri1"

# Read from kuri2 echo "Reading from kuri2:" curl -s http://10.1.232.84:8079/testbucket/crossnode-test.txt echo ""

# Read from kuri1 to verify echo "Reading from kuri1:" curl -s http://10.1.232.83:8079/testbucket/crossnode-test.txt ```

Output: ``` Testing cross-node access... Written to kuri1 Reading from kuri2:

Reading from kuri1: ```

The empty output from both reads is the critical signal. The object was written successfully (the PUT returned without error), but neither node can serve it back. This is not a simple connectivity issue—it is a symptom of a deeper architectural problem.

Why This Message Was Written: Context and Motivation

To understand why this message exists, we must trace the events that led to it. The session began with the deployment of a three-node QA cluster for the Filecoin Gateway (FGW) distributed storage system. The architecture consisted of a head node running YugabyteDB (the shared metadata database) and two kuri storage nodes that each exposed an S3-compatible API on port 8079. Earlier in the session, the assistant had resolved "dirty migration" errors in the CQL keyspaces and configured the FGW_BACKEND_NODES environment variable so that the cluster topology API would return data about both nodes.

The immediate trigger for message 2057 was the user's request to run the load test tool (ritool loadtest) against the cluster. The assistant ran a 30-second load test with 5 concurrent workers, writing and reading objects of varying sizes. The user then observed that only kuri1 was receiving traffic and commented: "Only kuri1 getting traffic seems wrong." This observation was correct—the load test had been pointed at http://10.1.232.83:8079, which is kuri1's S3 endpoint. There was no load balancer, no proxy, and no mechanism to distribute requests across the two nodes.

The assistant's response in message 2057 is therefore a diagnostic pivot. Rather than simply acknowledging the load balancing gap, the assistant formulates a more fundamental question: even if traffic were distributed, would the system actually work? Could an object written to one node be read from another? The shared S3 metadata keyspace (filecoingw_s3) in YugabyteDB should theoretically allow any node to look up object locations. But theory and practice diverge here.

Assumptions and Their Consequences

This message reveals several assumptions that were either incorrect or incomplete:

Assumption 1: Shared metadata implies shared data access. The architecture stored S3 object metadata (bucket, key, CID, node_id) in a shared CQL keyspace accessible to all kuri nodes. The natural assumption was that if kuri2 could see that an object belonged to kuri1, it could either fetch the data from kuri1 or serve it directly. The empty read output disproves this—kuri2 knows the object exists but cannot access its block data.

Assumption 2: The S3 layer serves from a distributed block store. The kuri nodes were configured with individual blockstores (local data directories). The S3 server on each node serves objects from its local blockstore only. There is no built-in mechanism for a node to fetch blocks from a peer node when serving an S3 request. The error that later appears in the logs confirms this: "block was not found locally (offline)."

Assumption 3: The deployment architecture was complete. The QA cluster had been deployed with kuri nodes acting as both storage backends and S3 endpoints. The roadmap, however, specified a three-layer architecture with stateless S3 frontend proxies that route requests to the correct backend node based on the stored node_id. This proxy layer had not been deployed, which meant the system was operating in an incomplete configuration.

Assumption 4: A successful PUT implies a readable object. The PUT to kuri1 returned no error, which led the assistant to believe the object was stored. In reality, the subsequent reads from both nodes returned empty, suggesting either the object was not persisted correctly or the retrieval path was broken. Later investigation revealed that the PUT required proper S3 headers (x-amz-content-sha256: UNSIGNED-PAYLOAD) to succeed—the initial curl command without these headers may have failed silently.

Input Knowledge Required

To understand this message fully, the reader needs knowledge in several domains:

Distributed storage architecture: Understanding the separation between metadata (object names, locations, CIDs) and data (the actual blocks) is essential. The shared CQL keyspace stores metadata, but each node's blockstore is local. This is a common pattern in distributed systems (e.g., HDFS separates namenode metadata from datanode blocks), but it requires a routing layer to bridge the two.

S3 protocol semantics: The S3 API is a RESTful protocol where objects are addressed by bucket and key. In a multi-node setup, the S3 endpoint must either be a load balancer that distributes requests arbitrarily (which only works if all nodes have all data) or a smart router that sends requests to the node holding the data. The current setup had neither.

CQL and YugabyteDB: The system uses YugabyteDB's Cassandra-compatible CQL interface for S3 metadata. The s3objects table stores rows with node_id indicating which kuri node holds the data. Understanding this schema is key to realizing why the routing layer is necessary.

Bitswap and IPLD: The kuri nodes use IPFS's bitswap protocol for peer-to-peer block exchange. Later in the debugging session, the assistant attempts to connect the two kuri nodes via bitswap to enable cross-node block retrieval, but this fails because the S3 layer does not use bitswap for serving objects—it reads from the local blockstore directly.

Output Knowledge Created

This message generates several critical insights that shape the subsequent work:

The routing gap is confirmed. The cross-node read test provides empirical evidence that the system cannot serve objects across nodes without additional infrastructure. This is not a configuration issue or a minor bug—it is a missing architectural layer.

The s3-proxy becomes the priority. The assistant immediately begins investigating the s3-proxy binary that was built earlier in the session. This proxy is designed to sit in front of the kuri nodes, read the shared S3 metadata to determine which node holds an object, and route the request accordingly. The proxy becomes the solution to the cross-node read problem.

The deployment process needs automation. When the user later asks "Why are you not doing this with ansible?" (message 2079), it highlights that the assistant had fallen into the trap of ad-hoc SSH commands and manual configuration. The cross-node read failure exposed not just a technical gap but a process gap—the deployment of the proxy should have been part of the automated playbook from the start.

The architecture is validated against the roadmap. The failure confirms that the roadmap's three-layer design (S3 proxy → kuri storage nodes → YugabyteDB) is not optional but essential. The assistant had initially deployed kuri nodes as direct S3 endpoints, which worked for single-node testing but broke down as soon as multi-node access was required.

The Thinking Process

The assistant's reasoning in this message follows a clear diagnostic pattern:

Observe the symptom: The load test only hit kuri1. The user flags this as wrong.
Formulate a hypothesis: The lack of traffic distribution is a load balancing issue, but the deeper question is whether cross-node reads would work even with proper distribution.
Design a minimal test: Write an object to kuri1, then read it from both kuri1 and kuri2. This is a classic distributed systems diagnostic—test the simplest cross-node operation before debugging complex load balancing.
Interpret the results: Both reads return empty. The PUT succeeded (no error), but the GETs return nothing. This is unexpected and points to a deeper issue than simple load balancing.
Identify the next investigation: The assistant immediately begins checking the s3-proxy binary, looking at the CQL schema, and examining the retrieval path. The empty reads are not accepted as a final state but as a signal to dig deeper. The message also shows the assistant's awareness of the architectural ideal. The phrase "shared S3 keyspace" in parentheses reveals that the assistant understands the theoretical design—metadata should be shared—and is testing whether the implementation matches the design. When the test fails, the assistant does not blame configuration drift or network issues; instead, it correctly identifies the missing proxy layer.

Mistakes and Incorrect Assumptions

While the diagnostic approach is sound, several aspects of this message reveal incorrect assumptions:

The PUT may not have succeeded. The curl command used -d @- with a pipe from echo, but the output shows "Written to kuri1" as a local echo, not a response from the server. The actual HTTP response code is not captured. Later in the session, the assistant discovers that PUTs without proper S3 headers return 500 errors with "invalid content sha256." The cross-node test may have been testing against an object that was never actually stored.

The test conflates two issues. The empty read from kuri2 could be caused by either (a) the object not being stored at all, or (b) the object being stored on kuri1 but kuri2 being unable to retrieve it. The test design does not distinguish these cases because it does not verify the PUT response or check kuri1's blockstore directly.

The assumption that "shared S3 keyspace" enables cross-node reads is optimistic. The S3 metadata keyspace stores object locations, but the S3 server on each kuri node is not designed to fetch blocks from peers. It serves from the local blockstore. The shared keyspace is for the proxy layer, not for direct node-to-node S3 serving.

Conclusion

Message 2057 is a turning point in the session. It transforms a seemingly simple load balancing observation into a fundamental architectural discovery. The empty response from the cross-node read test reveals that the distributed S3 system, while functional on individual nodes, lacks the routing infrastructure required for horizontal scalability. This discovery drives the subsequent work: deploying the s3-proxy on the head node, configuring it with the proper backend node list and CQL connection, and ultimately validating that cross-node reads work correctly through the proxy layer.

The message exemplifies a critical skill in distributed systems engineering: the ability to design a minimal, targeted test that exposes architectural gaps rather than surface-level configuration issues. A less experienced engineer might have responded to the "only kuri1 getting traffic" observation by adding a simple round-robin load balancer, which would have distributed requests but left every other read failing with "block not found locally." Instead, the assistant tested the cross-node path directly, uncovered the missing routing layer, and set the stage for the correct architectural fix.