The Diagnostic Pivot: How a Single Error Log Revealed the Missing Proxy Layer in a Distributed S3 Architecture

The Message

[assistant] Still empty. The issue is the S3 layer isn't configured for cross-node retrieval. Let me check the retrieval config:
[bash] ssh 10.1.232.84 "sudo journalctl -u kuri --since '30 sec ago' --no-pager 2>&1 | grep -i test.txt"
Jan 31 22:34:50 fgw-ribs2 kuri[17592]: 2026-01-31T22:34:50.411Z        ERROR        gw/s3        s3/server.go:65        error handling HTTP GET request        {"URL": "/testbucket/test.txt", "error": "error getting object reader: failed to get node for cid bafkreicq725qu5x2tcre6ff6jb7cbuu4msnmmdceannrzsk6spdmom3gb4 and key testbucket/test.txt: block was not found locally (offline): ipld: could not find bafkreicq725qu5x2tcre6ff6jb7cbuu4msnmmdceannrzsk6spdmom3gb4"}

Context: The State of Play

This message arrives at a critical inflection point in a multi-hour session deploying a distributed S3-compatible storage system across three physical nodes. The architecture under construction is the Filecoin Gateway (FGW), a horizontally scalable S3 storage system designed to eventually serve as an enterprise-grade retrieval layer for the Filecoin network. The deployment spans a head node (10.1.232.82) running a single-node YugabyteDB cluster, and two storage nodes (kuri1 at 10.1.232.83, kuri2 at 10.1.232.84) running the "kuri" daemon—the core storage engine that manages block data and exposes an S3-compatible API on port 8079.

In the moments leading up to this message, the assistant has been methodically working through a deployment checklist. The kuri daemons on both nodes are running. The cluster topology API, which initially returned empty results, has been fixed by setting the FGW_BACKEND_NODES environment variable so each node knows about its peers. A load test has been run against kuri1 directly, successfully writing and reading objects. But a critical question remains: can data written to one node be read from the other?

The user has just observed that the load test only hit kuri1, which "seems wrong" in a distributed system. The assistant's investigation has revealed that the shared S3 metadata keyspace (stored in YugabyteDB's CQL interface) correctly records which node owns each object—the s3objects table shows node_id='kuri_01' for objects written to kuri1. But when kuri2 receives a GET request for that same object, it fails. The assistant attempted to solve this by connecting the two nodes via IPFS/libp2p bitswap, establishing a direct peer-to-peer link. Yet the cross-node read still returns empty.

This is the moment captured in the target message: the assistant confronts the failure, checks the logs, and discovers the root cause.

The Reasoning Process

The message opens with a stark two-word assessment: "Still empty." This is not merely a status update—it is a hypothesis falsified. The assistant had just connected kuri1 to kuri2 via the IPFS swarm API, establishing a direct libp2p connection between the two storage nodes. The implicit assumption was that bitswap, IPFS's block-exchange protocol, would handle cross-node retrieval transparently. If node A needs a block that node B has, bitswap should fetch it. But the test proves otherwise.

The assistant's next sentence reveals the analytical pivot: "The issue is the S3 layer isn't configured for cross-node retrieval." This is a crucial diagnostic insight. The assistant is distinguishing between the network layer (bitswap, which handles raw block exchange between IPFS nodes) and the application layer (the S3 server, which handles HTTP requests for objects). Even though the underlying block-exchange protocol might work, the S3 server's code path for serving a GET request doesn't leverage it. The S3 layer looks for blocks locally and, failing that, returns an error rather than attempting to fetch from peers.

To confirm this hypothesis, the assistant retrieves the relevant log entry from kuri2. The error message is precise and revealing: "block was not found locally (offline): ipld: could not find [CID]". The phrase "not found locally (offline)" is particularly telling—it indicates that the retrieval code path explicitly distinguishes between local and remote blocks, and the S3 handler is only consulting the local blockstore. The error originates from s3/server.go:65, the HTTP GET handler, confirming that the failure is at the S3 application layer, not the networking layer.

Assumptions and Their Consequences

Several assumptions underpin the work leading to this message, and the error log exposes their limitations.

The first assumption was that bitswap connectivity between nodes would be sufficient for cross-node S3 reads. This is a reasonable assumption in a pure IPFS context, where bitswap is the primary mechanism for content discovery and retrieval. However, the FGW's S3 server is not a generic IPFS gateway—it's a purpose-built S3-compatible frontend that maintains its own index of objects and their locations. When a GET request arrives, the server looks up the object in the shared S3 metadata (which tells it the object exists and belongs to kuri_01), then attempts to retrieve the content by its content identifier (CID). The retrieval path apparently only checks the local blockstore before giving up, rather than consulting the bitswap routing table or attempting a network fetch.

The second assumption was that the FGW_BACKEND_NODES environment variable, which was just added to enable the cluster topology display, would also enable cross-node data access. In reality, this variable is used for the diagnostics API (the cluster topology endpoint) and for the proxy routing layer—not for the kuri daemon's internal block retrieval logic. Each kuri node remains fundamentally standalone in its S3 serving capability.

The third, more subtle assumption was that the architecture was complete as deployed. The deployment had kuri nodes acting as both storage backends and S3 endpoints, a design that the project's roadmap explicitly rejects. The roadmap calls for a three-layer hierarchy: stateless S3 frontend proxies on port 8078, kuri storage nodes on internal ports, and YugabyteDB as the shared metadata store. By running kuri nodes with their S3 ports exposed directly, the assistant had inadvertently created a system where each node could only serve its own data.

Input Knowledge Required

To fully understand this message, one needs familiarity with several layers of the system. The S3 protocol and its HTTP API are foundational—the message deals with PUT and GET requests for objects stored in buckets. The IPFS ecosystem is equally important: bitswap is the peer-to-peer block exchange protocol, CIDs (Content Identifiers) are the hash-based addressing scheme, and the swarm API manages peer connections. The YugabyteDB CQL interface provides the shared metadata store, using a Cassandra-compatible query language to maintain the s3objects table that records which node owns each object. The systemd journal is the logging mechanism, and the error format follows Go's structured logging conventions with JSON payloads.

The concept of a "blockstore" is central: in IPFS-based systems, content is split into blocks identified by CIDs, and a blockstore is the local storage for these blocks. The error "block was not found locally (offline)" indicates that the retrieval code path distinguishes between local and remote block availability, and the S3 handler is only checking the former.

Output Knowledge Created

This message creates several important pieces of knowledge. First, it definitively establishes that bitswap connectivity alone is insufficient for cross-node S3 reads in the current architecture. The nodes can peer at the IPFS level, but the S3 server doesn't use bitswap for object retrieval. Second, it identifies the exact error path and location (s3/server.go:65) where the failure occurs, giving developers a precise target for debugging or enhancement. Third, it surfaces the architectural gap between the cluster metadata (which knows where objects live) and the data retrieval mechanism (which can only access local blocks).

Most importantly, this message sets the stage for the solution that follows in subsequent messages: the deployment of the s3-proxy frontend. The proxy is designed precisely to solve this problem—it reads the shared S3 metadata to determine which backend node owns an object, then routes the request to that node's internal S3 endpoint. The assistant's recognition that "the S3 layer isn't configured for cross-node retrieval" is the key insight that leads to this architectural correction.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message follows a clear diagnostic pattern. The first step is observation: the cross-node read returned empty despite the bitswap connection. The second step is hypothesis formation: the problem is at the S3 layer, not the network layer. The third step is evidence gathering: retrieving the error log to confirm the hypothesis. The fourth step is interpretation: the error message reveals that the block retrieval path is purely local.

The phrase "Let me check the retrieval config" is particularly interesting because it reveals an expectation about the system's design. The assistant expects to find a configuration parameter that controls cross-node retrieval behavior—perhaps a flag that enables remote block fetching, or a list of peer nodes to consult. What the assistant actually finds is that no such configuration exists for the kuri S3 server. The cross-node retrieval capability is not a configurable feature that was accidentally left disabled; it's a capability that doesn't exist in the kuri daemon's S3 handler at all. This discovery forces a architectural rethink: instead of configuring kuri to fetch from peers, the system needs a separate routing layer that directs requests to the correct node.

The Broader Significance

This message, though brief, represents a classic moment in distributed systems engineering: the discovery that a system's components, while individually functional, do not compose into the desired behavior without additional infrastructure. The bitswap connection between nodes works perfectly for IPFS-native use cases, but the S3 compatibility layer has different assumptions about data locality. The shared metadata store correctly tracks object ownership, but the retrieval path doesn't consult it for routing decisions.

The solution—deploying the s3-proxy frontend on the head node, configuring it with the backend node list and CQL connection parameters, and routing all S3 traffic through it—is the direct consequence of the insight gained in this message. The proxy becomes the intelligent routing layer that the architecture was missing: it reads the metadata to determine the owning node, forwards the request accordingly, and returns the response to the client. This is the architecture the roadmap always specified, but the assistant had to discover through failure why it was necessary.

In the end, this message is a testament to the value of precise error messages and methodical debugging. A single log line—"block was not found locally (offline)"—provided the key to understanding a fundamental architectural limitation. The assistant's willingness to question assumptions, check the evidence, and pivot to a different architectural approach turned a frustrating dead end into a clear path forward.