The Keyspace That Almost Wasn't: A Pivot Point in Distributed S3 Architecture

Introduction

In the course of building a horizontally scalable S3-compatible storage system atop a Filecoin gateway, a single, seemingly minor edit to a Docker Compose file represented a fundamental architectural about-face. The message in question—message index 516 in the conversation—is deceptively brief:

[assistant] ## Agent Reasoning I need to update the S3 proxy to use the shared S3 keyspace filecoingw_s3. [edit] /home/theuser/gw/test-cluster/docker-compose.yml Edit applied successfully.

Five lines. One file edit. Yet this message encapsulates the resolution of a deep architectural crisis that had consumed the preceding dozen exchanges. It marks the moment when the assistant abandoned a flawed, partial approach to multi-node isolation and committed to a clean keyspace-segregation strategy. To understand why this edit matters, one must trace the chain of reasoning—and the series of failures—that led to it.

The Crisis: Two Nodes, One Keyspace, No Isolation

The conversation's immediate context was a test cluster for a three-layer S3 architecture: stateless S3 frontend proxies on port 8078, Kuri storage nodes handling deals and blockstore data, and a shared YugabyteDB backend. The assistant had successfully built the cluster infrastructure, Docker images, and monitoring dashboard, but when both Kuri nodes were enabled, the second node (kuri-2) consistently failed at startup.

The root cause was architectural: both nodes shared the same YugabyteDB keyspace for all their data—groups, deals, blockstore metadata, everything. When kuri-2 started, it tried to resume groups from the database and found groups created by kuri-1. It attempted to operate on those groups as if they were its own, leading to conflicts, deadlocks, and crashes. The assistant's initial fix was to add a node_id field to database queries, filtering results so each node only saw its own groups. This approach required modifying the RbsDB struct to carry a nodeID string and threading it through all group and deal queries.

The user's response to this effort was blunt and decisive. In message 512, they wrote:

"This makes no sense to me at all whatsoever. Groups are entirely separate to nodes, owned by nodes. There is one global table tracking them that is supposed to have a nodeid. All SQL/CQL APIs / calls related to groups/deals/etc MUST contain node ID. Or maybe easier we should seggregate db/keyspace for kuri nodes at the RIBS layer and only share at S3 layer (per node ipfs-y blockstore but s3 tables shareb between all)"

The user presented two paths: (1) add node_id to every single query across the entire RIBS layer, or (2) segregate keyspaces—each Kuri node gets its own database keyspace for RIBS data, while S3 object routing metadata lives in a shared keyspace accessible to all nodes and proxies. The second approach was simpler, cleaner, and avoided touching dozens of database queries.

The Pivot: From Per-Query Filtering to Keyspace Segregation

The assistant's reasoning in message 513 shows the pivot happening in real time. The agent analyzed the user's suggestion, recognized that keyspace segregation was "actually simpler and cleaner," and immediately began implementing it. The first step was updating gen-config.sh to generate per-node keyspace configurations. Each Kuri node would now have its own filecoingw_{node_id} keyspace for groups, deals, and blockstore data.

But this raised a critical question: what about the S3 metadata? The S3 frontend proxy needs to know which Kuri node holds which object. This routing information must be shared across all nodes and proxies. If each node has its own keyspace, where does the shared S3 metadata live?

Message 516 is the answer. The assistant recognized that the S3 proxy—the stateless frontend that accepts S3 API requests on port 8078—needs to connect to a shared keyspace, not a per-node one. The edit updates the docker-compose.yml to configure the S3 proxy with filecoingw_s3 as its CQL keyspace. This is the shared namespace where object-to-node mappings are stored, accessible to all proxies and all Kuri nodes alike.

The Reasoning Process: What the Agent Was Thinking

The agent's reasoning header in message 516 is minimal—just "I need to update the S3 proxy to use the shared S3 keyspace filecoingw_s3." But this terseness belies the depth of the decision. The agent had just read the docker-compose.yml (message 515) to understand the current state of the S3 proxy configuration. The file showed the proxy's command, volumes, and dependencies, but not its keyspace configuration. The agent needed to add the environment variable or configuration parameter that tells the S3 proxy which keyspace to use for object routing queries.

The choice of filecoingw_s3 as the shared keyspace name is itself a design decision. It signals that this keyspace is distinct from the per-node keyspaces (filecoingw_kuri-1, filecoingw_kuri-2) and serves a different purpose. The naming convention establishes a clean separation of concerns: per-node keyspaces for node-local data (groups, deals, blockstore), shared keyspace for cross-node coordination (object routing).

Assumptions Made

The message rests on several assumptions:

  1. The S3 proxy connects to YugabyteDB directly. The architecture assumes the stateless frontend queries the shared keyspace to determine which Kuri node holds a given S3 object, rather than relying on a separate routing service or gossip protocol.
  2. Keyspace names are deterministic. The convention filecoingw_{node_id} for per-node keyspaces and filecoingw_s3 for the shared keyspace must be consistent across all configuration files and database initialization scripts.
  3. The S3 proxy does not need per-node keyspace access. Unlike Kuri nodes, which need both their own RIBS keyspace and the shared S3 keyspace, the proxy only needs the shared keyspace. This assumption holds because the proxy is stateless—it routes requests but never stores node-local data.
  4. The database initialization script creates both types of keyspaces. The db-init service in docker-compose must be updated to create filecoingw_s3 in addition to the per-node keyspaces.

Mistakes and Incorrect Assumptions

The most significant mistake was the assistant's initial approach of adding node_id to individual database queries. This was a classic "local fix" error: addressing a systemic architectural problem (shared keyspace pollution) with a tactical code change (filtering queries) rather than a structural one (keyspace segregation). The user correctly identified that this approach was unsustainable—it would require modifying every group, deal, and blockstore query in the codebase, creating a maintenance burden and a constant source of bugs.

The assistant also initially assumed that the S3 proxy and Kuri nodes could share the same keyspace configuration. It took the user's explicit reminder in message 520 ("kuri nodes connect to both the ribs db/keyspaces AND s3 keyspaces") to clarify that Kuri nodes need dual database connections—one for their own RIBS keyspace and one for the shared S3 keyspace. This led to further configuration work in subsequent messages.

Input Knowledge Required

To understand message 516, one needs:

Output Knowledge Created

Message 516 produces a concrete artifact: an updated docker-compose.yml that configures the S3 frontend proxy with the shared filecoingw_s3 keyspace. This edit is part of a larger transformation that includes:

The Broader Significance

Message 516 is a case study in architectural decision-making under pressure. The assistant was deep in debugging mode, trying to get two Kuri nodes running simultaneously. The natural instinct was to make the smallest possible fix—add a node_id filter to database queries. But the user recognized that this tactical fix would create long-term complexity and pushed for a cleaner structural solution.

The keyspace segregation approach that message 516 implements has profound implications for the system's scalability. New Kuri nodes can be added by simply creating a new keyspace and configuring the node to use it. There is no risk of group ID collisions or deal ownership conflicts because each node's data is in its own namespace. The shared S3 keyspace remains the only coordination point, and it is accessed only for object routing—a simple key-value lookup that can be heavily optimized and cached.

This is the kind of architectural insight that separates a working system from a maintainable one. The edit in message 516 is small, but the thinking behind it is large. It represents the moment when the assistant stopped fighting the architecture and started building the architecture that the roadmap actually specified.