The Architecture Reversal: Reverting to a Single Keyspace in a Distributed S3 System

Introduction

In the complex landscape of distributed systems architecture, few decisions carry as much weight as the choice between isolation and sharing at the database level. Message 456 of this coding session captures the quiet but significant moment when an assistant—acting as a technical architect and developer—reverts a critical infrastructure decision, undoing a separate-keyspace approach in favor of a single shared keyspace for a horizontally scalable S3 storage cluster. The message itself is deceptively brief:

Now let me also revert the docker-compose.yml db-init to use the original single keyspace. [edit] /home/theuser/gw/test-cluster/docker-compose.yml Edit applied successfully.

This single action, a file edit to a Docker Compose configuration, represents the culmination of a much larger architectural debate that unfolded across dozens of preceding messages. To understand why this seemingly trivial revert was necessary, one must trace the reasoning, assumptions, and corrections that led to this point.

The Context: A Distributed S3 Architecture Takes Shape

The project under development is a horizontally scalable S3-compatible storage system built on the Filecoin Gateway platform. The architecture follows a three-layer hierarchy: stateless S3 frontend proxies (port 8078) that route requests to independent Kuri storage nodes, which in turn store data in a shared YugabyteDB cluster. This design, outlined in the project roadmap, promises horizontal scalability by allowing the addition of more proxy nodes and storage nodes independently.

The test cluster, defined in Docker Compose, originally specified two Kuri nodes (kuri-1 and kuri-2) sharing a single YugabyteDB database with a unified keyspace. However, as the assistant worked through the implementation, a critical problem emerged: when both nodes started simultaneously, they raced to create the same database resources—groups, deals, and other shared state—causing conflicts, deadlocks, and startup failures.

The Initial Assumption: Isolation Through Separate Keyspaces

Faced with these failures, the assistant made an understandable but ultimately incorrect assumption. The reasoning, visible in the preceding messages, went something like this: "If both nodes are fighting over the same database resources, the solution is to give each node its own isolated keyspace." This is a common pattern in multi-tenant systems—database-level isolation prevents one tenant from interfering with another.

In messages 428 through 432, the assistant implemented this approach. The gen-config.sh script was updated to generate per-node settings files with distinct keyspace names (filecoingw_kuri1 and filecoingw_kuri2), and the docker-compose.yml was modified to initialize both keyspaces during the database initialization step. A third shared keyspace (filecoingw_s3) was reserved for S3 object metadata that needed to be visible across all nodes.

This approach had a certain logic to it. Separate keyspaces would provide clean isolation, prevent resource conflicts, and allow each node to operate independently. The assistant even updated the README documentation to reflect this architecture, noting that "multi-node support (kuri-2) requires implementing node_id in RIBS groups" as a separate concern.

The Correction: A Fundamental Architectural Insight

The user's response in message 436 was the pivotal moment that triggered the chain of reverts culminating in message 456. The user stated unequivocally: "No all nodes definitely should use a single shared keyspace - they must know about their keys. Tables which are e.g. kuri-specific should have a node id entry in tables."

This correction reveals a deeper understanding of the architecture. The separate-keyspace approach, while superficially clean, would have created a fundamental problem: if each node has its own keyspace, how does the S3 proxy know which node stores a given object? The shared keyspace for S3 metadata (filecoingw_s3) would partially address this, but the real issue is more subtle. The Kuri nodes need to know about each other's keys—they need to see the same group identifiers, deal records, and storage metadata to coordinate properly. Isolating them at the keyspace level would prevent this coordination, making cross-node operations impossible.

The correct approach, as the user articulated, is to keep a single shared keyspace but add a node_id column to node-specific tables. This way, all nodes share the same database, can see each other's data when necessary, but filter by node_id when operating on their own resources. This is a classic multi-tenancy pattern implemented at the row level rather than the database level.

The Revert Chain: Undoing Separate Keyspaces

Message 456 is the final step in a multi-stage revert process. Earlier messages (437 through 455) show the assistant systematically undoing the separate-keyspace changes:

  1. Message 437: The assistant acknowledges the correction and begins reverting, checking git diff to see what was changed.
  2. Messages 438-445: The assistant explores implementing node_id support in the RIBS group system, reading migration files, database schemas, and configuration structures. The scope of changes proves extensive—modifying the database schema, updating all queries, adding configuration fields, and wiring through dependency injection.
  3. Messages 446-455: Rather than implementing the full node_id solution immediately, the assistant pivots to a pragmatic intermediate step: reverting to single-node operation, updating the README, and documenting the limitation. The gen-config.sh is reverted to use a single shared keyspace. Message 456 completes this revert by updating the docker-compose.yml database initialization section. The db-init service, which creates the database schema on startup, had been modified to initialize three separate keyspaces. Now it must be reverted to initialize just the single shared keyspace.

Why This Message Matters

On its surface, message 456 is trivial—a single file edit, confirmed successful. But in the context of the broader session, it represents several important things:

The Cost of Architectural Missteps

The separate-keyspace approach, while well-intentioned, required changes to multiple files: gen-config.sh, docker-compose.yml, start.sh, and README.md. Each change had to be reasoned about, implemented, and then reverted. The time spent on this approach—spanning roughly 25 messages—could have been avoided with a clearer initial understanding of the architecture.

The Importance of Domain Knowledge

The user's correction reveals domain knowledge that the assistant lacked: in a distributed storage system where nodes must coordinate, database-level isolation defeats the purpose of sharing. The assistant's assumption that "separate keyspaces = clean isolation" was reasonable in a general sense but wrong for this specific architecture. The correct pattern—row-level tenant isolation via a node_id column—is a more nuanced solution that preserves shared visibility while enabling per-node operations.

The Pragmatic Pivot

After recognizing the scope of changes needed for full node_id support (schema migration, query updates, configuration changes, dependency injection), the assistant made a pragmatic decision: revert to single-node operation and document the limitation rather than attempting a half-implemented solution. This is a mature engineering judgment—better to have a working single-node system than a broken multi-node one.

Assumptions and Mistakes

Several assumptions and mistakes are visible in the reasoning leading to message 456:

  1. The isolation assumption: The assistant assumed that database conflicts between nodes should be resolved by separating their data at the keyspace level. This ignored the requirement for nodes to share knowledge of keys and objects.
  2. The complexity underestimation: When the assistant first proposed separate keyspaces, the complexity seemed manageable. Only after implementing the revert and exploring the node_id approach did the full scope become clear.
  3. The premature documentation: The README was updated to reflect the separate-keyspace architecture before the approach was validated with the user. This documentation then had to be reverted.
  4. The workaround preference: Faced with the complexity of node_id implementation, the assistant chose to reduce the cluster to a single node rather than implement the correct solution. This is a pragmatic tradeoff but delays the multi-node capability.

Input Knowledge Required

To understand message 456 fully, one needs knowledge of:

Output Knowledge Created

Message 456 produces:

The Thinking Process

The assistant's reasoning in message 456 is minimal—just "Now let me also revert the docker-compose.yml db-init to use the original single keyspace." But this simplicity masks the extensive deliberation that preceded it. The thinking process visible across the broader session shows:

  1. Problem identification: Recognizing that kuri-2 fails due to shared database resources
  2. Hypothesis formation: Proposing separate keyspaces as a solution
  3. Implementation: Modifying configuration files and Docker Compose
  4. Validation failure: The user rejects the approach
  5. Re-analysis: Exploring the node_id alternative
  6. Scope assessment: Recognizing the extensive changes required
  7. Pragmatic decision: Reverting to single-node operation
  8. Systematic cleanup: Reverting each file that was changed Message 456 is the last step in this cleanup—the final file to be reverted before the configuration is consistent again.

Conclusion

Message 456, for all its brevity, encapsulates a fundamental lesson in distributed systems architecture: the choice between isolation and sharing must align with the system's coordination requirements. The separate-keyspace approach would have created isolated silos that couldn't coordinate, undermining the very purpose of a shared storage cluster. By reverting to a single keyspace, the assistant aligned the implementation with the architecture's true requirements, paving the way for the correct node_id-based solution in future work. The message stands as a testament to the iterative nature of engineering—where even a "revert" is progress when it replaces an incorrect solution with a correct foundation.