The Moment of Discovery: Debugging Cross-Node S3 Reads in a Distributed Storage Cluster

The Subject Message

[assistant] Interesting - kuri1 can read it, but kuri2 returns empty. This means the shared S3 metadata isn't working as expected. Let me check the S3 CQL keyspace:
[bash] ssh 10.1.232.82 "python3 -c \"
from cassandra.cluster import Cluster
cluster = Cluster(['10.1.232.82'], port=9042)
session = cluster.connect('filecoingw_s3')

# Check tables
rows = session.execute('SELECT table_name FROM system_schema.tables WHERE keyspace_name = %s', ['filecoingw_s3'])
print('Tables in filecoingw_s3:')
for row in rows:
    print(f'  - {row.table_name}')

# Check S3Objects table if it exists
try:
    rows = session.execute('SELECT * FROM s3objects LIMIT 5')
    print('\\nS3Objects entries:')
    for row in rows:
        print(f'  {row}')
except Exception as e:
    print(f'\\nError querying s3objects: {e}')

cluster.shutdown()
\""
Tables in filecoingw_s3:
  - multipartuploads
  - cidgroups
  - s3objects
  - multihashtogroup
  - schema_migrations

S3Objects entries:
  Row(bucket='loadtest', key='loadtest/w0/obj796-1769898791576787757', cid='bafkreie42btymaun6rfgvfnlydtdkk3xk6knram3ripclmim76sayigvce', size=689283, updated=datetime.datetime(2026, 1, 31, 22, 33, 11, 589000), node_id='kuri_01', expires_at=None)
  Row(bucket='testbucket', key='test.txt', cid='bafkreicq725qu5x2tcre6ff6jb7cbuu4msnmmdceannrzsk6spdmom3gb4', size=1...

Introduction: The Architecture of a Distributed S3 System

The Filecoin Gateway (FGW) project implements a horizontally scalable, distributed S3-compatible storage system built on a sophisticated multi-layer architecture. At its core, the system separates stateless S3 frontend proxies from stateful storage nodes (called "kuri" nodes), which in turn store data blocks in a content-addressable blockstore backed by YugabyteDB for metadata. This architecture promises the holy grail of distributed storage: any node in the cluster can serve any object, regardless of which node originally stored it. The S3 metadata lives in a shared CQL keyspace (filecoingw_s3) accessible to all nodes, while the actual block data is distributed across nodes based on content-addressed storage policies.

The subject message captures a critical debugging moment in the deployment of a three-node QA test cluster. The assistant had just successfully deployed kuri daemons on two physical nodes (10.1.232.83 and 10.1.232.84) backed by a single-node YugabyteDB on a head node (10.1.232.82). After resolving "dirty migration" state errors that prevented the kuri services from starting, and configuring the FGW_BACKEND_NODES environment variable to enable cluster topology discovery, the assistant ran a load test against one node. The test succeeded, but when the user observed that only one node was receiving traffic, the assistant began investigating cross-node data access.

The Reasoning: Why This Message Was Written

This message represents the moment of empirical discovery—the point at which the assistant confronts the gap between architectural theory and operational reality. The assistant had just demonstrated that a single kuri node could serve objects it had stored locally. The next logical question was whether the distributed architecture was actually working: could a different node serve the same object by reading from the shared metadata store and fetching the block data from the peer node?

The assistant's reasoning chain is visible in the preceding context. In message 2057, the assistant writes a test object to kuri1 and attempts to read it from kuri2, with the explicit comment: "Test: Write to kuri1, read from kuri2 (shared S3 metadata)." The initial test fails—kuri2 returns empty. The assistant then fixes the PUT request by adding proper S3 headers (x-amz-content-sha256: UNSIGNED-PAYLOAD), successfully writes the object, and confirms kuri1 can read it. But kuri2 still returns nothing.

This is the moment captured in the subject message. The assistant's opening line—"Interesting—kuri1 can read it, but kuri2 returns empty. This means the shared S3 metadata isn't working as expected."—reveals the core assumption being tested. The assistant had assumed that because both kuri nodes connect to the same YugabyteDB instance and share the filecoingw_s3 keyspace, any node could resolve an object's metadata and serve it. The empty response from kuri2 disproves this assumption, prompting a deeper investigation.

The Investigation: Querying the Shared Database

The assistant's response is methodical. Rather than guessing at the cause, it goes directly to the source of truth: the YugabyteDB CQL keyspace. The Python script executed on the head node connects to the Cassandra-compatible CQL interface, lists the tables in the filecoingw_s3 keyspace, and queries the s3objects table for entries.

The results are revealing. The database contains the expected tables—s3objects, multipartuploads, cidgroups, multihashtogroup, and schema_migrations—confirming that the schema migration completed successfully. The s3objects table contains entries for both the loadtest objects and the test object, each with a node_id field indicating which node stored the data. Crucially, the test object written to kuri1 has node_id='kuri_01', confirming that the metadata was properly recorded.

This query produces a critical piece of output knowledge: the metadata is present and correct in the shared database. The problem is not that kuri2 cannot find the object's metadata—the problem is that kuri2 cannot serve the object even when it knows where the data lives. This narrows the debugging search space considerably. The issue must lie in how kuri2 attempts to retrieve the block data from kuri1, not in the metadata lookup itself.

Assumptions and Their Consequences

The subject message reveals several assumptions that shaped the assistant's debugging approach:

Assumption 1: Shared metadata implies cross-node readability. The assistant assumed that because both nodes share a CQL keyspace, any node could serve any object. This is architecturally correct in theory—the S3 proxy layer is designed to route requests to the correct backend node based on metadata. However, the kuri nodes in this deployment were running as direct S3 endpoints (not behind the s3-proxy), meaning each kuri node's S3 handler only looks up metadata locally and attempts to serve blocks from its own blockstore. Without the proxy layer to perform the routing decision, each node is effectively a standalone S3 server that happens to share a database.

Assumption 2: The FGW_BACKEND_NODES environment variable would enable cross-node communication. The assistant had configured this variable earlier (message 2037) to enable the cluster topology API. However, this variable is used by the diagnostic/topology system, not by the S3 request handling path. The S3 server code doesn't use FGW_BACKEND_NODES to route requests—it only knows about its own local blockstore.

Assumption 3: The S3 implementation would automatically fall back to peer nodes. The assistant may have assumed that the S3 server would check peer nodes when a local blockstore lookup fails. The architecture documentation suggests this is the role of the s3-proxy frontend, not the kuri nodes themselves.

Input Knowledge Required

To understand this message, one needs knowledge of:

  1. The FGW distributed architecture: The separation between stateless S3 proxies and stateful kuri storage nodes, with shared YugabyteDB for metadata.
  2. CQL and YugabyteDB: The Cassandra Query Language interface that YugabyteDB provides, and the concept of keyspaces and tables.
  3. The s3objects table schema: Particularly the node_id field that records which node stored each object.
  4. The deployment topology: Three physical nodes (head node for database, two kuri nodes for storage) and the network configuration.
  5. The S3 protocol: The requirement for proper headers like x-amz-content-sha256 for PUT operations.
  6. The debugging context: The prior resolution of dirty migration states, the configuration of FGW_BACKEND_NODES, and the successful load test against kuri1.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. Confirmation that the shared CQL keyspace is operational: The filecoingw_s3 keyspace contains all expected tables and data.
  2. Evidence that metadata is correctly recorded: The s3objects table contains entries with accurate node_id fields.
  3. Narrowing of the problem space: The issue is not metadata availability but block data retrieval across nodes.
  4. A clear debugging direction: The next step is to investigate how kuri2 attempts to fetch block data from kuri1, which will likely lead to the realization that the s3-proxy frontend is needed.
  5. Validation of the database schema migration: The presence of all expected tables confirms that the migration process completed correctly.

The Thinking Process

The assistant's thinking process in this message is a textbook example of systematic debugging. The sequence is:

  1. Observe the symptom: kuri1 can read, kuri2 cannot.
  2. Form a hypothesis: "The shared S3 metadata isn't working as expected."
  3. Test the hypothesis directly: Query the shared database to verify metadata presence.
  4. Interpret the results: The metadata is present and correct.
  5. Refine the hypothesis: The problem must be in the data retrieval path, not the metadata lookup.
  6. Set the stage for the next investigation: The assistant has now gathered enough information to pivot to the real solution—deploying the s3-proxy frontend. The decision to query the database via Python on the head node rather than through any kuri node's API is significant. It demonstrates an understanding of the system's data flow: the database is the shared source of truth, and querying it directly eliminates any node-specific caching or routing issues that might obscure the results.

The Broader Significance

This message is a turning point in the deployment session. It marks the transition from "the cluster is running" to "the cluster is actually working as a distributed system." The assistant had successfully deployed the kuri daemons, resolved startup issues, and demonstrated single-node functionality. But a distributed storage system that cannot serve data from peer nodes is not a distributed system at all—it's a collection of independent storage silos that happen to share a database.

The recognition that cross-node reads fail leads directly to the next major action: deploying the s3-proxy frontend on the head node with proper routing configuration. This completes the architectural vision of a three-layer hierarchy (S3 proxy → kuri nodes → YugabyteDB) and enables the load test to distribute traffic across both storage nodes through a single endpoint.

In the broader narrative of the coding session, this message represents the moment when theoretical architecture meets operational reality. The system was designed with clean abstractions and separation of concerns, but the deployment process had skipped a critical layer. The debugging process revealed this gap not through code review or documentation reading, but through empirical testing—writing an object to one node and trying to read it from another. This is the essence of distributed systems debugging: trust nothing, verify everything, and let the data guide you to the truth.