The Moment of Discovery: Cross-Node Read Failure in a Distributed S3 Cluster

Introduction

In the complex process of deploying and validating a distributed storage system, there are moments when a single diagnostic command reveals a fundamental architectural gap. Message 2061 in this coding session is exactly such a moment. It is a brief, almost throwaway test — two curl commands and their output — yet it exposes the critical difference between a system that appears to be working and one that actually functions as designed. This message marks the turning point where the assistant discovers that the two-node Kuri cluster, despite both nodes running healthily and sharing a common database, cannot serve objects across nodes. The data written to node kuri_01 is invisible to node kuri_02, and the entire premise of a horizontally scalable S3 architecture collapses into a single-node island.

The Message

Here is the exact content of the subject message, as it appeared in the conversation:

Now let's test cross-node read:

>

```bash # Read from kuri2 (object written to kuri1) echo "Reading from kuri2:" curl -s http://10.1.232.84:8079/testbucket/test.txt echo ""

>

# Read from kuri1 echo "Reading from kuri1:" curl -s http://10.1.232.83:8079/testbucket/test.txt ```

>

Reading from kuri2:

>

Reading from kuri1: test-data-123

The asymmetry is stark. The first curl returns nothing — an empty line where the content test-data-123 should appear. The second curl returns the expected payload. The two nodes, kuri_01 (10.1.232.83) and kuri_02 (10.1.232.84), behave differently even though they are supposedly part of the same distributed cluster. The empty response from kuri_02 is not a network error or a crash; it is a silent failure of the system's core promise: that any node can serve any object.

The Road to This Test

To understand why this message was written, we must trace the events that led to it. Earlier in the session, the assistant had successfully deployed two Kuri storage nodes across two physical machines (10.1.232.83 and 10.1.232.84), with a single-node YugabyteDB instance running on a head node (10.1.232.82). The Kuri daemons were running, the Cluster Topology API was returning healthy status for both nodes, and the user had just run a load test against kuri_01's S3 endpoint on port 8079. The load test succeeded, but the user immediately noticed a problem: "Only kuri1 getting traffic seems wrong" (message 2055). This observation was correct — the load test had been pointed at a single node, and there was no load balancer or routing layer distributing requests.

The assistant's first instinct was to test whether the system could handle cross-node access at all. If data written to kuri_01 could be read from kuri_02, then the cluster had some form of shared storage or peer-to-peer retrieval. If not, then each node was operating as an independent silo, and the "cluster" was merely a collection of unrelated servers sharing a database schema.

The initial attempt to write a test object failed with a 500 error, revealing that the S3 endpoint required proper AWS signing headers. The assistant corrected this by adding x-amz-content-sha256: UNSIGNED-PAYLOAD and a Content-Type header, which produced a successful 200 response from kuri_01. The response even included a custom X-Node-Id: kuri_01 header, confirming which node had handled the write. With a known object now stored on kuri_01, the assistant issued the two curl commands that constitute message 2061.## The Diagnostic Value of Silence

The empty response from kuri_02 is more informative than any error message. A crash would have produced a connection refused or a timeout. A 404 would have indicated that the object metadata was missing. A 500 would have pointed to an internal error. But an empty 200 response — a successful HTTP request with no body — tells a specific story: the S3 server on kuri_02 received the request, looked up the object metadata in the shared filecoingw_s3 keyspace, found the entry (because the S3 metadata is stored in the shared YugabyteDB), but then failed to retrieve the actual block data. The S3 layer on kuri_02 attempted to read the content-addressed block from its local blockstore, did not find it, and returned nothing — or more precisely, returned an empty response body without signaling an error.

This behavior is a design choice, not a bug. The Kuri S3 server is intentionally configured to serve data from the local blockstore only. It does not attempt peer-to-peer block retrieval for S3 requests. This makes sense from a performance perspective: S3 is supposed to be fast, and cross-node bitswap retrieval over the network would introduce latency. But it also means that the cluster cannot function as a distributed object store without an additional routing layer. The system's architecture, as documented in the roadmap, calls for separate stateless S3 frontend proxies that sit between clients and the Kuri storage nodes. These proxies read the node_id field from the shared S3 metadata and route each request to the correct backend node. The assistant had not yet deployed this proxy layer, so each Kuri node was acting as an independent S3 endpoint serving only its local data.

Assumptions and Misconceptions

The assistant made several assumptions that this test disproved. First, there was an implicit assumption that the shared CQL keyspace for S3 metadata would somehow enable cross-node reads. The metadata was indeed shared — both nodes could see the same s3objects table entries — but the metadata alone is not the data. The actual block content is stored in each node's local blockstore, keyed by content identifier (CID). The shared database tells the system where an object lives (which node ID), but the S3 server on each node does not use that information to fetch from the peer.

Second, the assistant assumed that the bitswap peer-to-peer network, which connects the Kuri nodes to each other and to the broader Filecoin network, would handle cross-node block retrieval automatically. Earlier in the session, the assistant had connected kuri_01 to kuri_02 via the IPFS swarm API, and the connection succeeded. But bitswap is a protocol for content discovery and retrieval in the IPFS ecosystem, not a transparent backend for S3 reads. The S3 server does not fall back to bitswap when a block is missing locally — at least not in the current configuration. The error logged by kuri_02 (visible in subsequent messages) confirms this: "block was not found locally (offline): ipld: could not find [CID]". The phrase "offline" is telling — it suggests the retrieval system attempted a local lookup and declared the block unavailable rather than attempting a network fetch.

Third, there was an assumption about the completeness of the deployment. The assistant had deployed the Kuri binaries, configured the systemd services, set up the database schemas, and verified that both nodes started successfully. From an operations perspective, the cluster looked healthy. But health checks only verify that a process is running, not that the system fulfills its functional requirements. The cross-node read test exposed a gap that no monitoring dashboard or status API would have caught.

The Thinking Process Revealed

The structure of message 2061 reveals the assistant's reasoning process. The message begins with "Now let's test cross-node read" — a deliberate transition from the previous activity (running a load test against a single node) to a targeted diagnostic. The assistant is thinking: "We know kuri_01 works. We know both nodes are running. But does the cluster actually function as a distributed system?" The two curl commands are designed as a controlled experiment: write to one node, read from both. The symmetry of the test (same URL path, same object key, different hosts) isolates the variable of interest — node identity.

The fact that the assistant did not immediately interpret the empty response as an error is also revealing. The output is presented without commentary: "Reading from kuri2:" followed by a blank line, then "Reading from kuri1:" followed by "test-data-123". The assistant is showing the raw evidence to the user, letting the asymmetry speak for itself. This is a hallmark of good diagnostic practice: present the data, not the interpretation, and let the reader draw the same conclusion. The subsequent messages (2062 onward) show the assistant investigating the root cause — querying the CQL database, checking logs, and ultimately discovering that the S3 layer lacks cross-node retrieval capabilities.## Input Knowledge Required

To fully understand message 2061, the reader needs to grasp several layers of context. One must understand that the S3 API is an HTTP-based object storage protocol, and that curl commands like PUT and GET correspond to object writes and reads. One must know that the two IP addresses (10.1.232.83 and 10.1.232.84) represent distinct physical servers, and that port 8079 is the S3-compatible endpoint exposed by each Kuri node. One must understand the concept of a "content-addressed" storage system where objects are identified by cryptographic hashes (CIDs) rather than by location — and that this design means the system knows what data it needs but not necessarily where to find it. One must also understand the architecture of the FGW system: the shared YugabyteDB cluster that holds S3 metadata (bucket names, object keys, CIDs, and node IDs) alongside the per-node local blockstores that hold the actual data. Without this context, the asymmetry between the two curl responses is just a curiosity; with it, the empty response becomes a flashing red indicator of an architectural gap.

Output Knowledge Created

This message produced several concrete pieces of knowledge. First, it established definitively that the two-node cluster did not support cross-node S3 reads in its current configuration. This was not a theoretical concern — it was a measured, reproducible fact. Second, it generated a specific error signature: the S3 server returns a successful HTTP status code with an empty body when the requested block is not in the local blockstore. Third, it narrowed the search space for the root cause: the problem was not in the shared metadata layer (which was working correctly) but in the data retrieval path. Fourth, it created a reusable test procedure — write to node A, read from node B — that could be used to verify any future fix.

The downstream effects of this discovery were significant. In the messages that follow, the assistant investigates the CQL database to confirm that the S3 metadata is indeed shared (message 2062), checks the kuri_02 logs to find the specific error message (message 2063), discovers that the bitswap peer connection does not help (messages 2065-2067), and ultimately pivots to deploying the s3-proxy frontend on the head node (messages 2073-2076). The user's intervention at message 2079 — "Why are you not doing this with ansible?" — redirects the assistant from ad-hoc SSH commands to proper infrastructure automation, leading to a clean, reproducible deployment of the proxy layer via the existing Ansible playbooks. The final verification shows both nodes receiving traffic during the load test, with cross-node reads working correctly through the proxy.

Mistakes and Incorrect Assumptions

The most significant mistake was the assumption that a "running cluster" is equivalent to a "working cluster." Both Kuri nodes reported healthy status, the database was connected, and the S3 API responded to requests. Yet the system failed at its primary function: serving objects regardless of which node received the request. This is a classic distributed systems pitfall — the difference between component health and system behavior.

A subtler mistake was the assumption that the shared CQL keyspace for S3 metadata would suffice for cross-node reads. The assistant had correctly configured the database connection for both nodes, and the metadata was indeed visible from both. But metadata visibility does not imply data accessibility. The S3 server on each node reads the metadata to find the CID and node ID, but then attempts to retrieve the block from its local blockstore only. The node ID field in the metadata is written but not read by the retrieval path — it is informational rather than functional. This is a design issue in the Kuri S3 implementation itself, not a configuration error.

The assistant also initially assumed that bitswap peering would solve the problem. After connecting the two nodes via the IPFS swarm API, the cross-node read still failed. This revealed that the S3 layer does not integrate with bitswap for block retrieval — or at least, not without additional configuration. The error message "block was not found locally (offline)" suggests that the retrieval system explicitly avoids network fetches for S3 requests, perhaps for performance or security reasons.

Conclusion

Message 2061 is a masterclass in diagnostic minimalism. Two curl commands, four lines of output, and a world of insight. The empty line between "Reading from kuri2:" and "Reading from kuri1:" is not just whitespace — it is the sound of a distributed system failing silently. The message captures the moment when theory meets reality, when the architecture diagram on paper confronts the actual behavior of running code. It is a reminder that in distributed systems, the hardest problems are not the crashes but the silences — the responses that should contain data but do not, the features that should work but are not wired up, the clusters that look healthy but cannot serve their purpose. The assistant's response to this discovery — methodical investigation, log analysis, database queries, and ultimately architectural correction — is a model of disciplined debugging. And the user's timely intervention to enforce proper automation practices ensures that the fix is not just a one-off SSH command but a repeatable, version-controlled deployment. The cross-node read test was a failure that taught more than any success could have.