The Bitswap Assumption: A Debugging Detour in Distributed S3 Architecture

Introduction

In the course of deploying and validating a QA test cluster for the Filecoin Gateway's (FGW) distributed S3 storage system across three physical nodes, a single message captures a pivotal moment of debugging—one that reveals how even experienced engineers can follow a plausible but ultimately incorrect path. The message, sent by an AI assistant during an opencode coding session, represents an attempt to solve cross-node block retrieval by manually connecting two storage nodes via libp2p swarm peering. While the action succeeded technically (the connection was established), it failed to solve the underlying problem, forcing a fundamental reassessment of the architecture and leading to the correct solution: a dedicated S3 proxy frontend.

The Message

Here is the message in its entirety:

Nodes have peers but they're external Filecoin peers, not each other. Let me connect the two kuri nodes:

>

```bash # Get kuri2's peer ID KURI2_ID=$(ssh 10.1.232.84 "curl -s -X POST http://127.0.0.1:5001/api/v0/id" | jq -r '.ID') echo "Kuri2 PeerID: $KURI2_ID"

>

# Connect kuri1 to kuri2 ssh 10.1.232.83 "curl -s -X POST 'http://127.0.0.1:5001/api/v0/swarm/connect?arg=/ip4/10.1.232.84/tcp/4001/p2p/${KURI2_ID}'" ```

>

Output: `` Kuri2 PeerID: 12D3KooWPDgEFRktYgkSuR5yxEnpDvpJRAQnegHTCKcnqrPzBLsJ {"Strings":["connect 12D3KooWPDgEFRktYgkSuR5yxEnpDvpJRAQnegHTCKcnqrPzBLsJ success"]} ``

At first glance, this appears to be a straightforward operational command: discover a peer's identity and establish a network connection. But the reasoning behind it, the assumptions it encodes, and its ultimate failure make it a rich subject for analysis.

The Context: A Cluster That Almost Works

To understand why this message was written, we must trace the debugging journey that preceded it. The QA cluster consisted of three physical nodes: a head node (10.1.232.82) running a single-node YugabyteDB instance, and two kuri storage nodes (kuri1 at 10.1.232.83, kuri2 at 10.1.232.84). After resolving "dirty migration" states in the CQL keyspaces that prevented the kuri daemons from starting, and configuring the FGW_BACKEND_NODES environment variable to populate the Cluster Topology API, the cluster appeared healthy. Both nodes reported as "healthy" in the topology view.

The user then ran a load test, but only kuri1 received traffic—because the test was pointed directly at kuri1's endpoint. This prompted a critical question: could data written to one node be read from the other? The assistant tested this by writing an object to kuri1 and attempting to read it from kuri2. The result was revealing: kuri1 could serve the object, but kuri2 returned an empty response.

Investigation into the shared S3 CQL keyspace (hosted on YugabyteDB) showed that the object metadata was indeed visible to both nodes. The s3objects table contained entries with node_id='kuri_01', indicating which node held the actual block data. When kuri2 received a GET request for an object whose blocks lived on kuri1, it failed with a specific error: "block was not found locally (offline): ipld: could not find [CID]". The metadata was shared, but the blocks were not.

The Reasoning: Two Possible Solutions

At this point, the assistant correctly identified two architectural approaches to solve cross-node block retrieval:

  1. A proxy/frontend layer that routes requests to the correct backend node based on the stored node_id metadata.
  2. Peer-to-peer block exchange between nodes, using the IPFS/libp2p bitswap protocol that the kuri nodes already supported. The assistant chose to investigate option 2 first. This was a reasonable choice: the kuri nodes are built on IPFS/libp2p technology, they already had a swarm of peers (visible via the /api/v0/swarm/peers endpoint), and bitswap is the standard mechanism for exchanging blocks between IPFS nodes. If two kuri nodes could fetch blocks from each other via bitswap, the S3 layer on each node would be able to serve any object regardless of which node originally stored it. The blocks would simply be retrieved on demand from the peer. This brings us to message 2066. The assistant checked the existing peers on kuri1 and found many connections—but they were all external Filecoin peers, not the other kuri node. The logical next step was to explicitly connect the two kuri nodes to each other, establishing a direct libp2p link that would enable bitswap exchange between them.## The Assumption: Bitswap Will Solve Cross-Node Retrieval The decision to connect the nodes via swarm peering rested on several implicit assumptions, each of which seemed reasonable in isolation but collectively led the assistant down a garden path. Assumption 1: Bitswap is the retrieval path for S3 objects. The kuri nodes run an IPFS stack with bitswap enabled, and the S3 server sits on top of this stack. When the S3 layer needs a block identified by a CID, it queries the local blockstore. If the block is missing, bitswap should theoretically fetch it from peers. This is how IPFS works: content-addressed retrieval across a peer-to-peer network. The assistant assumed that the S3 layer would transparently benefit from bitswap's network retrieval capabilities. Assumption 2: The nodes simply weren't connected. The swarm peer list showed many external peers but no cross-kuri connection. The assistant assumed that connecting them was the missing piece—that once a direct libp2p link existed, bitswap would automatically discover and fetch blocks from the peer node. Assumption 3: The connection would be sufficient. The assistant did not check whether bitswap was configured to fetch blocks from specific peers, whether the S3 layer had a timeout or fallback mechanism for network retrieval, or whether the blockstore was configured to request blocks from the network at all. The assumption was that "connected peers" equals "automatic block exchange."

The Mistake: A Plausible but Incorrect Path

The connection succeeded. The output was unambiguous: {"Strings":["connect 12D3KooWPDgEFRktYgkSuR5yxEnpDvpJRAQnegHTCKcnqrPzBLsJ success"]}. But the very next test (message 2067) showed that reading from kuri2 still returned empty. The bitswap connection did not solve the problem.

Why? The fundamental issue was architectural. The S3 server on each kuri node is designed to serve data from its local blockstore. While bitswap exists in the stack, the S3 read path does not necessarily trigger bitswap retrieval for every request—especially when the blockstore returns a definitive "not found" status. The S3 layer's error message was explicit: "block was not found locally (offline)." The "offline" qualifier is significant—it suggests the system knew the block was not local and did not attempt network retrieval, or that network retrieval was not configured for the S3 path.

The assistant's mistake was not in trying bitswap peering—that was a legitimate diagnostic step—but in assuming that bitswap would automatically bridge the gap between the shared metadata layer and the distributed block storage layer. In reality, the architecture separates these concerns deliberately: metadata lives in a shared CQL keyspace (YugabyteDB), while block data lives in each node's local blockstore. The S3 proxy frontend, not bitswap, is the component designed to route requests to the correct node based on the metadata's node_id field.

Input Knowledge Required

To understand this message fully, one needs knowledge of several domains:

Output Knowledge Created

This message produced several forms of knowledge:

  1. Negative knowledge: Bitswap peering alone does not enable cross-node S3 reads in this architecture. This is a valuable finding—it confirms that the S3 layer's block retrieval path does not automatically fall back to bitswap for blocks stored on peer nodes.
  2. Operational knowledge: The kuri nodes' IPFS swarm APIs are accessible and functional. The assistant demonstrated the exact commands to retrieve a peer ID and establish a swarm connection between two nodes.
  3. Diagnostic confirmation: The successful connection proved that the nodes could communicate at the libp2p level, ruling out network connectivity issues between them. This narrowed the problem space to the application-layer routing logic.## The Thinking Process Visible in the Message The message reveals a clear chain of reasoning compressed into a few lines. The opening sentence—"Nodes have peers but they're external Filecoin peers, not each other"—shows the assistant processing the output of the previous swarm peers query and identifying the gap. The "Let me connect the two kuri nodes" statement is the conclusion drawn from that observation: if the nodes are not connected, connecting them should enable block exchange. The two-command sequence is structured as a miniature diagnostic script: first discover the peer ID (a prerequisite for connection), then establish the connection using that ID. The output is captured inline, showing both the discovered peer ID and the success message. This is characteristic of the assistant's debugging style throughout the session: short, focused commands with immediate output verification. What is not visible in the message is any hedging or uncertainty. The assistant does not say "let me try this and see if it works" or "this might not be sufficient." It presents the action as the logical next step. This confidence is understandable given the context—bitswap is the standard mechanism for IPFS block exchange, and connecting two nodes that need to exchange blocks is a textbook operation. But the confidence also reflects a subtle over-reliance on the IPFS abstraction layer: the assistant assumed that because the nodes could exchange blocks via bitswap, the S3 layer would use that capability.

The Resolution: Proxy Frontend as the Correct Solution

The story does not end with this failed attempt. After the bitswap connection proved insufficient, the assistant pivoted to the correct solution: deploying the s3-proxy frontend on the head node. The proxy was configured with the list of backend nodes and the CQL connection information, allowing it to route each S3 request to the correct kuri node based on the node_id stored in the shared metadata. This approach cleanly separates the routing logic from the storage logic: the proxy handles request distribution, while each kuri node focuses on serving its local blockstore.

The proxy deployment was done properly through the existing Ansible infrastructure, updating the QA inventory and running the playbooks rather than manually configuring systemd files. The final verification showed both nodes receiving traffic during the load test, with cross-node read/write operations working correctly.

Lessons in Distributed Systems Debugging

This episode illustrates several important principles for debugging distributed systems:

Test the simplest hypothesis first, but be prepared to discard it. The bitswap connection was a reasonable first attempt—it required minimal effort and could have solved the problem if the architecture worked as the assistant assumed. When it failed, the assistant had more information about the system's actual behavior.

Understand the layers of abstraction. The kuri nodes have an IPFS stack with bitswap, but the S3 layer sits above it and may have different retrieval semantics. The error message "block was not found locally (offline)" was the crucial clue that the S3 layer was not attempting network retrieval. The assistant initially focused on the network layer (are the nodes connected?) when the real issue was at the application layer (how does the S3 server resolve object reads?).

Shared metadata does not imply shared data. The YugabyteDB-backed CQL keyspace provided a unified view of object metadata across both nodes, but the actual block data remained partitioned by node. This is a common pattern in distributed storage systems—metadata and data have different distribution strategies—and it requires a routing layer to bridge them.

Conclusion

Message 2066 is a small but instructive moment in a larger debugging session. It captures the moment when an engineer, faced with a distributed system that almost works, makes a reasonable assumption about how its components should interact, tests that assumption with a precise operational command, and discovers that the system's actual behavior diverges from the expected model. The bitswap connection succeeded technically but failed functionally, forcing a reassessment that led to the correct architectural solution: a dedicated proxy frontend that routes requests based on metadata rather than relying on peer-to-peer block exchange.

The message is a testament to the value of empirical debugging in distributed systems. Rather than theorizing endlessly about why cross-node reads failed, the assistant executed a concrete test—connect the peers, try the read—and let the system reveal its true behavior. That revelation, though initially disappointing, pointed directly to the correct solution and ultimately produced a working multi-node S3 cluster.