The Diagnostic Probe: Tracing Cross-Node Block Retrieval in a Distributed S3 Cluster

Introduction

In the course of debugging a multi-node Filecoin Gateway (FGW) distributed storage cluster deployed across three physical machines, a seemingly simple command was issued that would reveal a critical gap in the system's architecture. The message in question—message index 2065 in the conversation—is a single bash command that queries the IPFS/libp2p swarm peer list on one of the storage nodes. On its surface, it is unremarkable: an SSH invocation running curl against a local API endpoint, piped through jq for JSON extraction. But in the context of the debugging session, this probe represents a pivotal diagnostic moment—a hypothesis being tested about how blocks flow between nodes in a horizontally scalable S3 storage system.

The Exact Message

The message reads:

[assistant] [bash] ssh 10.1.232.83 "curl -s -X POST http://127.0.0.1:5001/api/v0/swarm/peers" 2>&1 | jq -r '.Peers[]?.Peer' 2>/dev/null | head -10
12D3KooWPDgEFRktYgkSuR5yxEnpDvpJRAQnegHTCKcnqrPzBLsJ
12D3KooWPDgEFRktYgkSuR5yxEnpDvpJRAQnegHTCKcnqrPzBLsJ
12D3KooWMN6i3hd2WKE8JrFP8Khm2RF8T339X99HMxseo1DW2Ehk
12D3KooWLphi7GpbCCbo9u3kMsb3uZJdJ48buJ4bVMsNCgXy1Umn
12D3KooWT3Ye1Fo2bXoA4FrTc9B8TTQjw21RdTHLcT8NvZUjifXc
12D3KooWMsS5eTRmMUh5zezU9Jhk5zwa8n8XvWUv9mBLJHmvtf9m
Qmdfjtk6hPoyrH1zVD9PEH4zfWLo38dP2mDvvKXfh3tnEv
12D3KooWNG9wuK4fDx3sQb9eFtewg4sCj5wpXEYm1oWY3FwGTkq4
12D3KooWDWrNae9G3D444SZNX3MTCgWYkg777P3afWWwSK1nJTzn
12D3KooWPvh2s4sabDqPVygakjk4p...

The command connects to the kuri node at 10.1.232.83 (designated as kuri_01 in the cluster topology) and queries its local IPFS swarm API endpoint at http://127.0.0.1:5001/api/v0/swarm/peers. This returns a JSON list of all peers currently connected to this node via the libp2p networking layer. The jq filter extracts the Peer field from each entry, and head -10 limits output to the first ten lines. The output reveals a collection of multiaddr peer IDs—cryptographic identifiers in the form of 12D3KooW... strings—representing other nodes in the broader IPFS/Filecoin network that this kuri node is connected to.

The Debugging Context: Why This Message Was Written

To understand why this particular probe was necessary, we must trace the debugging session that led to it. The session began with the user deploying a QA cluster across three physical nodes: a head node running YugabyteDB, and two kuri storage nodes (kuri_01 at 10.1.232.83 and kuri_02 at 10.1.232.84). After resolving initial "dirty migration" state errors in the CQL keyspaces and configuring the FGW_BACKEND_NODES environment variable to enable cluster topology discovery, the user asked the assistant to run the load test tool against the cluster.

The load test was executed against kuri_01's S3 endpoint at http://10.1.232.83:8079. It ran successfully, writing and reading objects. But the user immediately spotted a problem: "Only kuri1 getting traffic seems wrong." This observation was astute—it revealed that the test was targeting a single node directly, bypassing any load distribution mechanism. More importantly, it raised the question of whether data written to one node could be read from another.

The assistant then conducted a cross-node access test. A test object was written to kuri_01 via a PUT request with proper S3 headers (x-amz-content-sha256: UNSIGNED-PAYLOAD). The write succeeded with a 200 OK response and returned an ETag (a CID: bafkreicq725qu5x2tcre6ff6jb7cbuu4msnmmdceannrzsk6spdmom3gb4). Reading from kuri_01 returned the data correctly. But reading from kuri_02 returned an empty response.

The assistant investigated the database layer, querying the shared filecoingw_s3 CQL keyspace on the YugabyteDB cluster. The s3objects table contained entries for both the load test objects and the manually written test file. Critically, each row stored a node_id field—the test file had node_id='kuri_01', indicating which node held the actual block data. The shared metadata was working correctly: both nodes could see the object index. But when kuri_02 tried to serve a GET request for an object whose blocks resided on kuri_01, it failed with the error: "block was not found locally (offline): ipld: could not find bafkreicq725qu5x2tcre6ff6jb7cbuu4msnmmdceannrzsk6spdmom3gb4."

This error reveals the fundamental storage architecture: each kuri node maintains a local blockstore. The S3 metadata is stored in a shared database, but the actual content-addressed blocks (IPLD blocks identified by CIDs) are stored locally on the node that received the write. For cross-node reads to work, either a proxy layer must route requests to the correct backend node, or the nodes must be able to fetch blocks from each other via peer-to-peer exchange.

The assistant's reasoning at this point, visible in the commentary, considered two options: "Either a proxy/frontend layer that routes requests to the correct node based on node_id, or peer-to-peer block exchange between nodes (bitswap)." The assistant then checked whether bitswap—the IPFS block exchange protocol—was configured between the nodes. This is the direct motivation for message 2065.## The Thinking Process: From Symptom to Hypothesis

The assistant's reasoning chain is clearly visible in the messages leading up to and following this probe. The chain goes: (1) cross-node reads fail with "block not found locally" → (2) the shared S3 metadata correctly points to kuri_01 as the node holding the blocks → (3) kuri_02 cannot serve blocks it doesn't have locally → (4) perhaps the nodes can fetch blocks from each other via the IPFS/libp2p bitswap protocol → (5) let's check if the nodes are peered with each other.

This is a classic diagnostic pattern: identify the symptom, trace it to its root cause in the data flow, form a hypothesis about the missing mechanism, and test that hypothesis with a targeted probe. The probe in message 2065 is the hypothesis test. The assistant is asking: "Are the kuri nodes connected to each other via the libp2p swarm, such that bitswap could serve as the cross-node block retrieval mechanism?"

The choice of the swarm peers API endpoint is deliberate. In IPFS/libp2p architecture, the swarm represents the set of directly connected peers. If two kuri nodes are peered, they can exchange blocks via bitswap without any additional routing infrastructure. If they are not peered, blocks cannot flow between them at the IPFS layer, and a different solution (such as an S3 proxy frontend) would be needed.

The Output: What the Probe Revealed

The output of the command is a list of ten peer IDs. Critically, the assistant's subsequent message (index 2066) interprets this output: "Nodes have peers but they're external Filecoin peers, not each other." This interpretation is the key insight. The peer IDs in the list belong to other Filecoin network participants—storage miners, bootstrap nodes, or other IPFS peers on the public network—but none of them is the peer ID of kuri_02 (the other node in the cluster). The assistant then proceeds to explicitly connect the two nodes by fetching kuri_02's peer ID via its own API and issuing a swarm/connect command to establish the direct libp2p link.

However, the subsequent test (message 2067) shows that even after establishing the swarm connection, cross-node reads still fail: "Still empty." The assistant then checks the logs again and sees the same "block was not found locally" error. This negative result is just as informative as a positive one would have been. It tells the assistant that bitswap peering alone is insufficient—the S3 layer has its own routing logic that does not automatically fall back to bitswap for cross-node block retrieval. The S3 handler checks the local blockstore first and fails if the block is not present, regardless of whether the node has peered with the block's owner.

Assumptions and Their Consequences

The assistant made several assumptions in this diagnostic sequence. First, it assumed that the IPFS/libp2p stack underlying the kuri nodes would naturally handle cross-node block exchange if the nodes were peered. This assumption was reasonable given that bitswap is designed precisely for this purpose—it is a content-addressed block exchange protocol that allows nodes to fetch blocks from peers that have them. However, the assumption overlooked a critical detail: the S3 handler in the kuri node may not be integrated with the bitswap layer for serving S3 GET requests. The S3 layer appears to have its own retrieval path that checks the local blockstore directly, bypassing the bitswap exchange mechanism.

Second, the assistant assumed that the swarm API at port 5001 was the correct interface for checking and managing peer connections. This is correct for standard IPFS nodes, but the kuri nodes might have custom networking configurations. The fact that the API responded with a valid peer list confirmed that the standard IPFS API was available.

Third, there was an implicit assumption that the two kuri nodes were not already connected to each other. The output confirmed this—none of the peer IDs matched the other node. This was a correct reading of the data.

Input Knowledge Required

To understand this message, one needs knowledge of several domains. First, the IPFS/libp2p networking model: the concept of peer IDs (cryptographic node identifiers), the swarm (the set of directly connected peers), and the bitswap protocol for block exchange. Second, the S3 storage model: the separation between metadata (stored in a shared database) and data blocks (stored in local blockstores), and the routing problem this creates in a multi-node deployment. Third, the specific architecture of the FGW system: kuri nodes run an embedded IPFS node with a local blockstore, and the S3 handler is a plugin that stores object metadata in YugabyteDB while keeping the actual content-addressed blocks locally. Fourth, operational knowledge of the deployment: the IP addresses of the nodes, the port numbers for the S3 API (8079) and the IPFS API (5001), and the SSH access configuration.

Output Knowledge Created

This message produced several pieces of knowledge. It confirmed that the kuri nodes have functioning IPFS swarm endpoints and are connected to the broader Filecoin network. It revealed that the two kuri nodes are NOT directly peered with each other, establishing the need for either explicit peering or an alternative routing mechanism. The subsequent test after peering (message 2067) further revealed that even with peering, cross-node S3 reads still fail, indicating that the S3 layer does not use bitswap for block retrieval. This negative result was crucial—it forced the assistant to pivot to the alternative solution: deploying an S3 proxy frontend that routes requests to the correct backend node based on the node_id stored in the shared metadata.

The Broader Significance

Message 2065 is a small but revealing moment in a larger debugging narrative. It represents the point at which the assistant tested the hypothesis that the IPFS networking layer could solve the cross-node retrieval problem, and found that hypothesis to be incorrect. This led directly to the architectural decision to deploy the s3-proxy frontend—a stateless routing layer that reads the node_id from the shared S3 metadata and forwards the request to the appropriate kuri node. This is the architecture described in the project's roadmap: stateless S3 frontend proxies that route to backend storage nodes.

The probe also illustrates a broader principle in distributed systems debugging: when a system has multiple layers (IPFS networking, S3 API, database metadata, local blockstore), failures can occur at any layer, and diagnostic probes must be designed to isolate which layer is responsible. The assistant systematically ruled out the database layer (shared metadata was correct), then tested the networking layer (peering was missing, then added), and finally concluded that the S3 application layer needed its own routing mechanism. This layered diagnostic approach is a model for debugging complex distributed storage systems.