The Keyspace Decision: Isolating Storage Nodes in a Distributed S3 Architecture

A Pivotal Edit That Resolved a Multi-Node Race Condition

In the course of building a horizontally scalable S3-compatible storage system, a single message stands out as the moment when a fundamental architectural problem was recognized and addressed. The message in question is brief—just two lines of reasoning and a file edit confirmation—but it represents the culmination of a lengthy debugging session that uncovered a deep design flaw in how multiple storage nodes interact with a shared database.

The message reads:

[assistant] ## Agent Reasoning I need to update the gen-config.sh to use separate keyspaces for each node. This way each Kuri node will have its own isolated database. [edit] /home/theuser/gw/test-cluster/gen-config.sh Edit applied successfully.

To understand why this simple edit was so significant, one must trace the debugging trail that led to it. The session began with the user reporting that a two-node test cluster was failing: kuri-2 would not start, the S3 proxy crashed with an invalid log level error, and a web UI port was unreachable. These appeared to be three separate issues, but as the assistant dug deeper, they revealed themselves as symptoms of a single underlying problem.

The Debugging Trail: From Log Levels to Database Architecture

The assistant's first fixes addressed surface-level issues. The log level configuration used the bare value info, but the codebase's configureLogLevels() function required the format component=level (e.g., *=info). This was fixed by removing the invalid environment variable. The startup script was also reworked to launch Kuri nodes sequentially rather than in parallel, avoiding a migration deadlock when both nodes tried to initialize YugabyteDB tables simultaneously.

These fixes got kuri-1 running, but kuri-2 still failed. The error logs told a more interesting story:

Error: write to group: opening group: opening group: open jbob (grp: /data/ribs/grp/1): 
opening head: open /data/ribs/grp/1/blklog.meta/head: no such file or directory

Kuri-2 was trying to open a group that didn't exist in its local filesystem. But why would it reference a group from kuri-1? The answer lay in the database: both nodes shared the same YCQL keyspace. When kuri-1 created a group during its initialization, that group was recorded in the shared database. When kuri-2 started up, it queried the database, found the group, and tried to access its local data—but that data lived on kuri-1's volume, not kuri-2's.

This was the core architectural flaw: the RIBS (Replicated Indexed Block Storage) layer had no concept of node ownership for groups. Groups were stored in a single shared keyspace, visible to all nodes, but each node's local storage was separate. The system assumed either a single-node deployment or a shared filesystem—neither of which matched the horizontally scalable architecture being built.

The Reasoning Process: Three Options, One Choice

The assistant's thinking, visible in the messages preceding the subject edit, shows a structured evaluation of alternatives. After confirming that groups lacked a node_id field (via a grep that found no matches in the rbstor package), the assistant enumerated three possible solutions:

  1. Add node_id to groups so each node only manages its own groups at the application level
  2. Use separate database keyspaces per node, isolating each node's data at the storage layer
  3. Use a different approach where groups are inherently node-specific Option 1 would require modifying the RIBS data model, adding a node_id column to all group-related tables, and updating every query to filter by node. This was invasive and risked breaking existing functionality. Option 3 was vague and undefined. Option 2—separate keyspaces—was the simplest and most contained change. It required only modifying the configuration generation script (gen-config.sh) to assign each node a unique database keyspace name (e.g., filecoingw_kuri-1 and filecoingw_kuri-2), while keeping the S3 metadata keyspace (filecoingw_s3) shared for object routing. This approach aligned with the roadmap's vision of independent storage nodes sharing only the routing layer.

Assumptions and Their Validity

The assistant made several assumptions in choosing this path. First, it assumed that the RIBS database layer supported configurable keyspace names. This turned out to be correct—the configuration system already had a CqlKeyspace parameter that could be set per node. Second, it assumed that the S3 metadata (used by the frontend proxy for object location) needed to remain shared across all nodes. This was also correct, as the proxy must know which node holds which object regardless of which node stored it.

A more questionable assumption was that separate keyspaces would not cause cross-node issues with the S3 routing layer. The S3 metadata keyspace (filecoingw_s3) was designed to be shared, but it contained references to node IDs. If a node's per-keyspace data referenced objects in the shared keyspace, there could be consistency issues. The assistant implicitly assumed the two layers were sufficiently decoupled—an assumption that would need validation through testing.

Input and Output Knowledge

To understand this message, one must know several things: that the system uses YugabyteDB (a distributed SQL database compatible with Cassandra's CQL); that RIBS is the storage layer managing groups, deals, and block data; that each Kuri node has its own local filesystem volume mounted at /data/ribs; and that the S3 frontend proxy uses a separate metadata keyspace for object routing. One must also understand the concept of database keyspaces as isolated namespaces within a CQL cluster.

The output knowledge created by this message is the configuration pattern for multi-node isolation: each node gets a unique keyspace name derived from its node identifier, while a shared keyspace is reserved for cross-node metadata. This pattern becomes a template for scaling the cluster beyond two nodes—adding kuri-3 would simply mean generating a filecoingw_kuri-3 keyspace.

The Broader Significance

This message represents a classic moment in distributed systems design: the realization that shared state is the enemy of scalability. The initial design treated the database as a single shared resource, which worked for a single node but broke under multi-node operation. The fix—segregating keyspaces—is a well-known pattern in multi-tenant database architectures, but its application here was specific to the RIBS layer's assumption of exclusive group ownership.

The edit to gen-config.sh was the smallest possible change that addressed the largest possible problem. It didn't modify any Go code, add any new columns to database tables, or change the RIBS storage logic. It simply told each node: "use your own sandbox." This is elegant engineering—finding the configuration-level fix that avoids invasive code changes while preserving the architectural intent.

The message also illustrates the importance of following a debugging trail to its root cause. The surface symptoms (log level error, migration deadlock, missing files) were all distractions from the real issue: shared keyspace contamination. The assistant's methodical investigation—checking logs, examining the group data model, confirming the absence of node_id fields, and evaluating alternatives—demonstrates a structured approach to diagnosing distributed system failures.

In the end, this two-line reasoning message and its accompanying file edit resolved a problem that had blocked the entire test cluster deployment. It transformed the architecture from a single-node design accidentally running on two nodes into a genuinely multi-node design where each storage node operates independently within its own database namespace, sharing only what must be shared for the system to function as a coherent whole.