The Architecture That Almost Wasn't: How One Message Rescued a Distributed S3 Design

Introduction

In the course of building a horizontally scalable S3-compatible storage system, a single message from an AI assistant marked the turning point between a fundamentally broken architecture and a correct, roadmap-aligned design. The message, message 549 in a long coding session, reads as follows:

Done. The architecture is now:

>

- 1 shared S3 keyspace (filecoingw_s3) - for S3 object routing, used by all Kuri nodes + S3 proxies - N per-node RIBS keyspaces (filecoingw_kuri1, filecoingw_kuri2) - for groups, deals, blockstore index

>

Each Kuri node connects to both: - Its own RIBS keyspace via RIBS_YUGABYTE_CQL_KEYSPACE - The shared S3 keyspace via RIBS_S3_CQL_KEYSPACE

>

You can now run ./start.sh /data/fgw2 - it will have the interactive step on first startup.

At first glance, this appears to be a simple status update—a few bullet points summarizing a configuration decision. But in context, this message represents the resolution of a multi-turn struggle to understand a core architectural principle, the correction of several incorrect assumptions, and the implementation of a design that had been specified in the roadmap but repeatedly misunderstood. This article examines that message in depth: why it was written, the reasoning that led to it, the mistakes it corrected, and the knowledge it crystallized.

The Context: A Session of Architectural Confusion

To understand why this message exists, one must understand the conversation that preceded it. The session was building a test cluster for a three-layer distributed S3 architecture: stateless S3 frontend proxies at the top, Kuri storage nodes in the middle, and a shared YugabyteDB database at the bottom. The roadmap had always specified this separation, but the assistant had been implementing it incorrectly.

The critical error was architectural: the assistant had been running Kuri nodes as direct S3 endpoints, with all nodes sharing a single database keyspace. This meant that when two Kuri nodes started simultaneously, they raced to create the same database resources—groups, deals, and blockstore indices—causing deadlocks, configuration validation failures, and runtime panics. The user had identified this flaw in message 512, saying bluntly: "This makes no sense to me at all whatsoever. Groups are entirely separate to nodes, owned by nodes."

The user proposed two solutions: either add node_id to all database queries, or segregate keyspaces per node at the RIBS layer while sharing only the S3 metadata keyspace. The assistant initially chose the keyspace segregation approach, but then made a critical mistake: it reverted to a shared keyspace for everything, assuming this would be simpler for testing. The user's response in message 532 was sharp: "No the shared keyspace makes no sense ffs, you want ONE S3 KEYSPACE and N RIBS/Blockstore keyspaces, it really is not hard at all."

This is the moment message 549 was born from—a direct response to that correction.

Why This Message Was Written: The Culmination of a Correction

Message 549 was written to declare that the architecture had finally been implemented correctly. It serves several purposes simultaneously:

First, it is a confirmation. The assistant is acknowledging that the user's specification has been understood and implemented. The bullet points mirror the user's own language from earlier messages: "1 shared S3 keyspace" and "N per-node RIBS keyspaces." This is the assistant saying, "I hear you, and here is the proof."

Second, it is a summary. The message condenses a complex set of code changes—adding a new S3CqlConfig to the configuration system, creating a S3CqlDB wrapper type, wiring dual CQL connections through the Kuri plugin's dependency injection, updating the Docker Compose file, and modifying the test cluster configuration generator—into three readable bullet points. This summary allows the user to verify the architecture at a glance without reading through dozens of edits.

Third, it is a transition. The final sentence—"You can now run ./start.sh /data/fgw2 - it will have the interactive step on first startup"—signals that the implementation phase is complete and testing can begin. The message closes the chapter on architecture design and opens the chapter on operational validation.

The Decision-Making Process: A Case Study in Iterative Correction

The reasoning visible in the assistant's thought process leading up to message 549 reveals a fascinating pattern of assumption, error, correction, and refinement.

The initial assumption was that a shared keyspace was acceptable for a test cluster. In message 525, the assistant reasoned: "For the test cluster to work simply, let me use the shared keyspace for now and document that production should use separate keyspaces." This was a pragmatic shortcut—get something running, then fix it later. But this violated the fundamental design principle that the user was trying to enforce: the architecture must be correct from the ground up, even in test.

The user's rejection in message 532 was forceful, and it triggered a rapid re-evaluation. The assistant's reasoning in message 533 shows the pivot: "You're right, sorry. Let me implement it properly." This is followed by a clear restatement of the correct architecture: "1 shared S3 keyspace - filecoingw_s3 - for S3 object routing (shared by all nodes + proxies). N RIBS keyspaces - filecoingw_kuri1, filecoingw_kuri2 - for groups/deals/blockstore (one per Kuri node)."

What follows is a rapid implementation sequence. The assistant adds a S3CqlConfig to the configuration system (messages 536-539), updates the Kuri plugin to use separate CQL connections (messages 540-541), and modifies the test cluster configuration (messages 542-544). The build succeeds (message 546), and the configuration generator runs successfully (message 548). Then comes message 549.

Assumptions Made and Mistakes Corrected

Several incorrect assumptions were made during this session, and message 549 represents their final resolution:

Assumption 1: A shared keyspace is acceptable for testing. The assistant assumed that operational simplicity in a test environment could justify architectural shortcuts. The user rejected this, insisting that the test cluster must accurately reflect the production architecture. This is a valuable lesson: test environments that diverge from production can mask critical bugs and create false confidence.

Assumption 2: The S3 layer can use the same database connection as RIBS. The assistant initially assumed that a single CQL database connection per node was sufficient. The user corrected this in message 520: "Of note kuri nodes connect to both the ribs db/keyspaces AND s3 keyspaces." This forced the assistant to recognize that each Kuri node requires two independent database connections—one for its own per-node keyspace and one for the shared S3 keyspace.

Assumption 3: The S3 frontend proxy is a single instance. The assistant had designed the Docker Compose file with a single s3-proxy service. The user corrected this in message 526: "There can be multiple stateless s3 frontend processes." This reminded the assistant that the proxy layer must be horizontally scalable, which influenced how the shared S3 keyspace was designed.

Mistake: Reverting to shared keyspace after starting keyspace segregation. Perhaps the most significant error was the assistant's decision in message 525 to abandon the keyspace segregation approach and revert to a shared keyspace. This was a step backward that wasted time and frustrated the user. The root cause was a misunderstanding of the codebase—the assistant thought implementing separate keyspaces would require extensive changes to the S3 index code, when in fact the solution was to add a second CQL configuration and wire it through the plugin system.

Input Knowledge Required

To understand message 549 fully, one needs knowledge of several concepts:

Keyspace segregation is the practice of using separate database namespaces (keyspaces in Cassandra/YugabyteDB terminology) to isolate data belonging to different logical domains. In this architecture, each Kuri node's internal state (groups, deals, blockstore indices) lives in its own keyspace, preventing nodes from interfering with each other's data.

CQL (Cassandra Query Language) is the query interface for YugabyteDB's YCQL API. Each database connection is configured with a specific keyspace, meaning that separate keyspaces require separate connection objects. This is why the assistant had to add a S3CqlConfig to the configuration system—to allow a second connection with a different keyspace.

The three-layer architecture consists of stateless S3 frontend proxies (layer 1), Kuri storage nodes (layer 2), and YugabyteDB (layer 3). The S3 proxies handle client requests and route them to the appropriate Kuri node based on object metadata stored in the shared S3 keyspace. Each Kuri node manages its own deals and blockstore data in its private RIBS keyspace.

The RIBS layer is the storage abstraction that handles replication, deals, and blockstore management. It was designed with the assumption of per-node isolation, which is why sharing a keyspace between nodes caused race conditions and deadlocks.

Output Knowledge Created

Message 549 crystallizes several important pieces of knowledge:

The final architecture blueprint. The three bullet points define the data isolation boundaries for the entire system. This becomes the reference model for all future configuration and deployment decisions.

The configuration interface. The message names the two environment variables that control keyspace assignment: RIBS_YUGABYTE_CQL_KEYSPACE for the per-node RIBS keyspace and RIBS_S3_CQL_KEYSPACE for the shared S3 keyspace. These become the contract between the configuration system and the application code.

The operational readiness signal. The final sentence confirms that the test cluster is ready for startup, establishing a clear transition point from implementation to testing.

The Thinking Process: A Window into Iterative Design

The reasoning traces visible in the assistant's messages reveal a genuine struggle to understand the architecture. The assistant repeatedly tried to simplify, to take shortcuts, to defer complexity. Each time, the user pushed back with increasing frustration. But the final message shows that the lesson was learned.

The key insight that unlocked the correct implementation was recognizing that the S3 layer and the RIBS layer have fundamentally different sharing requirements. The S3 object routing table must be shared because any proxy needs to know which node holds any object. But the RIBS data—groups, deals, blockstore indices—is internal to each node and must be isolated to prevent conflicts.

This distinction is subtle but critical. A shared S3 keyspace enables the stateless proxies to route requests correctly. Per-node RIBS keyspaces prevent nodes from interfering with each other's internal operations. The two requirements are not contradictory—they are complementary aspects of a well-designed distributed system.

Conclusion

Message 549 is deceptively simple. Three bullet points and a run command—that is all it contains on the surface. But beneath that simplicity lies the resolution of a multi-turn debugging session, the correction of fundamental architectural misunderstandings, and the implementation of a design that had been specified but not yet realized. It is a message that could only have been written after the assistant had tried the wrong approach, been corrected, re-evaluated its assumptions, and implemented the right solution.

In the end, the architecture that message 549 describes is exactly what the roadmap had always called for: stateless proxies that can scale horizontally, storage nodes with isolated data domains, and a shared routing layer that ties them together. The message does not just describe this architecture—it represents the journey of understanding that made it possible.