The Moment of Discovery: Diagnosing Cross-Node S3 Retrieval in a Distributed Storage Cluster

Introduction

In the complex process of deploying a distributed storage system across three physical nodes, there comes a pivotal moment when theory meets reality. Message 2063 in this coding session captures exactly such a moment—a diagnostic breakthrough where the assistant identifies why cross-node S3 reads are silently failing. This message is the fulcrum upon which the entire subsequent architecture pivots, leading to the deployment of an S3 proxy frontend that properly routes requests across the cluster.

The message itself is deceptively simple: the assistant observes that S3 object metadata stores a node_id field indicating which node holds the data blocks, and realizes that kuri2—despite having shared metadata access—cannot serve objects whose blocks reside on kuri1. But the reasoning behind this observation, the chain of evidence leading to it, and the architectural implications ripple through the rest of the session.

The Context: A Cluster That Almost Works

To understand why this message matters, we must trace the events that led to it. The assistant had just deployed a three-node QA cluster for the Filecoin Gateway (FGW) distributed storage system. The architecture consisted of two kuri storage nodes (kuri1 at 10.1.232.83 and kuri2 at 10.1.232.84) backed by a shared YugabyteDB instance on a head node (10.1.232.82). The YugabyteDB hosted both SQL and CQL (Cassandra Query Language) keyspaces, including a shared filecoingw_s3 keyspace that was supposed to provide unified S3 object metadata across the cluster.

Initial testing looked promising. The loadtest tool ran successfully against kuri1, writing and reading objects. The cluster topology API returned healthy status for both nodes. But the user noticed something suspicious: "Only kuri1 getting traffic seems wrong" (message 2055). This observation triggered a deeper investigation.

The assistant first attempted a simple cross-node test: write an object to kuri1, then read it from kuri2. The write succeeded (with proper S3 headers including x-amz-content-sha256: UNSIGNED-PAYLOAD), and reading from kuri1 returned the data correctly. But reading from kuri2 returned an empty response—no data, no error message visible to curl.

The Diagnostic Chain: Tracing the Empty Response

Message 2063 is the moment the assistant connects the dots. The preceding messages had already revealed critical pieces of evidence:

  1. The S3 metadata is shared: Querying the filecoingw_s3 keyspace's s3objects table showed that objects written to kuri1 were visible from any node. The row included node_id='kuri_01', bucket='testbucket', key='test.txt', and the content identifier (CID) bafkreicq725qu5x2tcre6ff6jb7cbuu4msnmmdceannrzsk6spdmom3gb4.
  2. Kuri2 knows the object exists: The shared keyspace means kuri2 can see the same metadata. When an S3 GET request arrives, kuri2 looks up the object in the shared index and finds it.
  3. But kuri2 cannot serve the data: The actual block data referenced by the CID is stored in kuri1's local blockstore. Kuri2's S3 server implementation only serves blocks from its own local blockstore—it has no mechanism to fetch blocks from peer nodes. The assistant's insight in message 2063 is precisely this: "the S3 object metadata stores node_id='kuri_01' - so kuri2 knows the object exists but the actual data blocks are on kuri1. The system needs to route the request to kuri1 or fetch from it." This is followed by a diagnostic command that confirms the hypothesis by examining kuri2's logs during the failed read attempt. The log entry is unambiguous:
ERROR gw/s3 s3/server.go:65 error handling HTTP GET request
{"URL": "/testbucket/test.txt", "error": "error getting object reader:
failed to get node for cid bafkreicq725qu5x2tcre6ff6jb7cbuu4msnmmdceannrzsk6spdmom3gb4
and key testbucket/test.txt: block was not found locally (offline):
ipld: could not find bafkreicq725qu5x2tcre6ff6jb7cbuu4msnmmdceannrzsk6spdmom3gb4"}

The critical phrase is "block was not found locally (offline)". The S3 server on kuri2 attempts to retrieve the block from its local blockstore, fails, and returns an empty response. There is no fallback to peer nodes, no network retrieval, no routing logic.

Assumptions and Misconceptions

This message reveals several assumptions that were implicitly held—and which turned out to be incorrect:

The shared metadata assumption: The architecture assumed that because S3 object metadata was stored in a shared CQL keyspace (YugabyteDB), any node could serve any object. The metadata layer was correctly shared, but the data layer was not. Each kuri node maintained an independent local blockstore, and the S3 server implementation had no cross-node block retrieval capability.

The "just works" assumption about distributed storage: There was an implicit expectation that a multi-node deployment would automatically distribute and serve data across nodes. In reality, the system required explicit routing logic—either a proxy layer that dispatches requests to the correct backend node, or a peer-to-peer block exchange protocol like bitswap.

The completeness assumption: The assistant had been iterating on this deployment for some time, fixing dirty migration states, configuring environment variables, and verifying individual node health. The assumption was that if each node was healthy and the shared database was working, cross-node operations would follow. Message 2063 shattered that assumption.

The Thinking Process: From Symptom to Root Cause

The reasoning visible in this message follows a classic diagnostic pattern:

  1. Observe the symptom: Cross-node reads return empty responses.
  2. Examine the data path: Check what metadata is available to each node.
  3. Identify the disconnect: The metadata is shared, but the data blocks are not.
  4. Formulate the root cause: Each node's S3 server only serves from its local blockstore.
  5. Propose the solution space: Either route requests to the correct node (proxy) or enable peer-to-peer block exchange (bitswap). The assistant then immediately proceeds to test the bitswap hypothesis (messages 2064-2068), connecting the two kuri nodes via their IPFS swarm APIs. When even that fails to enable cross-node reads, the assistant correctly concludes that the S3 layer itself lacks cross-node retrieval logic and pivots to deploying the s3-proxy frontend.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several critical insights:

  1. The architecture gap: The current deployment has a shared metadata layer but isolated data layers. Cross-node S3 reads require either a routing proxy or peer-to-peer block exchange.
  2. The diagnostic method: Checking the error logs on the target node reveals the exact failure mode ("block was not found locally").
  3. The remediation path: The s3-proxy frontend, which was already part of the roadmap's architecture, is the correct solution. It reads the node_id from the shared S3 metadata and routes the request to the appropriate backend node.

The Broader Significance

This message represents a classic distributed systems debugging moment. The system appeared to work—both nodes were healthy, the shared database was operational, the topology API showed correct information—but a fundamental architectural property (data locality) was silently breaking a core use case (cross-node reads). The assistant's ability to trace the empty HTTP response back through the metadata layer to the blockstore layer, and to correctly identify the missing routing mechanism, demonstrates a systematic approach to distributed system debugging.

The insight in message 2063 directly leads to the deployment of the s3-proxy on the head node, configured with the backend node list and CQL connection parameters. After resolving some initial startup issues (the proxy needed to connect to YugabyteDB, which required proper environment variable formatting), the proxy became operational and successfully routed cross-node requests. The final verification showed both nodes receiving traffic during load testing, with objects written to one node being readable from the other through the proxy layer.

Conclusion

Message 2063 is the diagnostic turning point in a complex multi-node deployment. It captures the moment when the assistant moves from "the cluster is running" to "the cluster is correctly architected." The observation that shared metadata alone is insufficient for cross-node data access—that the node_id field in the S3 objects table is not just informative metadata but a routing imperative—transforms the deployment strategy. Without this insight, the cluster would have remained a collection of isolated single-node systems sharing a database, rather than a true distributed storage cluster. The message exemplifies the kind of architectural debugging that separates a working demo from a production-ready system.