The Documentation That Tells the Truth: Scaling Back a Distributed S3 Cluster

Introduction

In the midst of building a horizontally scalable S3 architecture with independent Kuri storage nodes, a single message stands out as a quiet but significant turning point. Message 451 in the conversation is deceptively simple: the assistant reads the test cluster's README file and announces, "Now let me also update the components section." On its surface, this is a routine documentation update—a developer tidying up markdown. But beneath this mundane action lies a cascade of architectural discovery, pragmatic decision-making, and the difficult admission that a design assumption was wrong. This article examines that message in detail, tracing the reasoning that led to it, the trade-offs that were weighed, and the knowledge it both consumed and produced.

The Message Itself

The subject message reads in full:

[assistant] ## Agent Reasoning Now let me also update the components section. [read] /home/theuser/gw/test-cluster/README.md <file> 00039| Note: Multi-node support (kuri-2) requires implementing node_id in RIBS groups. 00040| 00041| Components: 00042| 00043| 1. S3 Frontend Proxy (:8078) 00044| - Stateless load balancer 00045| - Round-robin distribution for PUT requests 00046| - YCQL lookup to route GET requests to correct Kuri node 00047| - No local storage 00048| 00049| 2. Kuri Storage Nodes (internal ports only) 00050| - Each node stores independent subset of objects 00051| - E...

The message is one in a sequence of README edits (messages 449–453) that collectively rewrite the test cluster documentation to reflect a fundamental architectural retreat: the cluster would run only one Kuri node instead of two. The assistant reads the file, sees the note it has already placed about the node_id limitation, and prepares to update the components list to match the new single-node reality.

The Context: Why This Message Was Written

To understand why this message exists, we must step back to the events that precipitated it. The assistant had been building a test cluster for a horizontally scalable S3 architecture, following a roadmap that specified three layers: stateless S3 frontend proxies on port 8078, independent Kuri storage nodes, and a shared YugabyteDB database. The initial design called for two Kuri nodes (kuri-1 and kuri-2), each with its own local data directory but sharing a single YCQL keyspace in YugabyteDB.

When the assistant started the cluster, kuri-2 failed with a revealing error:

Error: write to group: opening group: opening group: open jbob (grp: /data/ribs/grp/1): 
opening head: open /data/ribs/grp/1/blklog.meta/head: no such file or directory

The root cause was architectural: both nodes shared the same database keyspace, and when kuri-1 created a group (group 1), it was recorded in the shared database. When kuri-2 started, it read that group from the database and tried to open its local files—but those files only existed on kuri-1's data directory. Each node had its own isolated local storage, but the database treated groups as a global resource. This was a fundamental mismatch between the storage model and the database model.

The user clarified the correct design: all nodes should share a single keyspace, but node-specific tables should include a node_id column so each node operates only on its own data. The assistant investigated implementing this change and discovered it would require modifying the database schema (adding node_id to the groups table), updating every database query in the RIBS layer to filter by node ID, adding a NodeID field to the configuration system, and wiring that configuration through the entire dependency chain. This was not a small change—it touched the database migration system, the rbstor/db.go query layer, the RibsConfig struct, and every caller that created groups or looked them up.## The Decision: When Pragmatism Trumps Perfection

The assistant's reasoning in the preceding messages (particularly message 446) reveals a clear decision-making process. Faced with the complexity of implementing node_id across the entire RIBS group system, the assistant weighed three options:

Implement full node_id support immediately — modify the database schema, update all queries, add configuration fields, and wire everything together. This was the architecturally correct solution but required extensive changes across multiple layers of the codebase.
Use separate keyspaces per node — give each Kuri node its own YCQL keyspace so groups would never collide. The assistant initially implemented this approach but the user rejected it, explaining that all nodes must share a keyspace to know about each other's keys.
Run only one Kuri node — document that multi-node support requires the node_id implementation and focus on getting a single-node cluster working correctly. The assistant chose option 3. This was a pragmatic retreat, not a permanent one. The reasoning was explicit: "Given the complexity of the changes needed (modifying the database schema, updating all queries, adding node_id to the config, etc.), I think we should: 1. Document that the test cluster currently only supports 1 Kuri node, 2. Create a todo list for implementing full node_id support, 3. Focus on making single-node work properly first." This decision reflects a mature engineering judgment. The assistant recognized that implementing node_id across the entire RIBS group system was a significant architectural change that deserved careful, deliberate implementation—not a rushed fix in the middle of test cluster debugging. By deferring the work to a todo item and documenting the limitation, the assistant preserved the ability to test the rest of the architecture (S3 proxy, object routing, health checks) while acknowledging the gap.

Assumptions Made and Mistakes Corrected

Several assumptions underpinned the original design, and the debugging process revealed which ones were incorrect.

Assumption 1: Groups are global resources. The RIBS group system had no concept of node ownership. Groups were created and tracked by a sequential integer ID, and any node could open any group. This assumption worked in a single-node deployment but broke immediately in a multi-node setup.

Assumption 2: Separate local storage is sufficient isolation. The assistant had carefully configured each Kuri node with its own Docker volume (${FGW_DATA_DIR}/kuri-1:/data/ribs and ${FGW_DATA_DIR}/kuri-2:/data/ribs), assuming that separate file systems would prevent conflicts. But the database layer didn't respect this isolation—it stored group metadata in a shared keyspace, and each node tried to access groups created by the other.

Assumption 3: The database schema already supports multi-node. The assistant had not verified that the groups table included a node_id column or that the RIBS queries filtered by node. The schema migration (1750766515_init_rbstor.up.sql) created the groups table without any node identifier, and the rbstor/db.go queries had no node_id parameter.

Mistake: Implementing separate keyspaces before consulting the user. The assistant initially changed the configuration to use separate keyspaces (filecoingw_kuri1, filecoingw_kuri2, filecoingw_s3) without first confirming this approach with the user. The user's response was clear: "No, all nodes definitely should use a single shared keyspace—they must know about their keys." This forced a revert and a rethinking of the approach. The lesson is that architectural decisions about data sharing boundaries should be validated early, especially when they affect the fundamental isolation model of the system.

Input Knowledge Required

To understand this message and the surrounding sequence, a reader needs knowledge of several domains:

Distributed storage architecture: The concept of stateless proxies vs. stateful storage nodes, and the trade-offs of shared vs. per-node databases.
The RIBS storage system: The codebase's internal storage layer that manages groups, block logs, and deal tracking. Understanding that groups are sequential integer IDs stored in a SQL database, with associated local file data.
YugabyteDB and YCQL: The distributed SQL database used for coordination, and the concept of keyspaces as logical database partitions.
Docker Compose orchestration: How the test cluster defines services, volumes, and dependencies for each component.
The S3 proxy architecture: The roadmap's three-layer design where frontend proxies route requests to Kuri nodes based on object metadata stored in the shared database. Without this context, the message reads as a trivial documentation fix. With it, the message becomes the visible trace of a significant architectural pivot.## Output Knowledge Created This message and the surrounding sequence produced several concrete artifacts and insights: 1. A corrected README that honestly documents the single-node limitation. The note "Multi-node support (kuri-2) requires implementing node_id in RIBS groups" is a clear signal to future developers about what needs to be done. 2. A migration file (1750766516_add_node_id_to_groups.up.sql) that was created but not yet applied. This file represents the first step toward the full node_id implementation—it adds the node_id column to the groups table. Even though the migration isn't executed in the test cluster, its existence documents the intended schema change. 3. A todo item for implementing node_id throughout the RIBS group system. The assistant used the todowrite tool to create a structured task list, ensuring the work wouldn't be forgotten. 4. An architectural lesson: The boundary between shared and per-node data must be explicitly designed, not assumed. The original design assumed that separate local file systems would provide isolation, but the database layer undermined that isolation. The corrected understanding is that groups are per-node resources that must include a node_id in the database, while S3 object metadata is a shared resource that all nodes and proxies can read. 5. A validated test cluster that works correctly with one Kuri node. By removing kuri-2 from the active configuration, the assistant ensured that the remaining architecture (S3 proxy → kuri-1 → YugabyteDB) could be tested end-to-end without the group collision error.

The Thinking Process Visible in the Reasoning

The assistant's reasoning traces a clear arc from confusion to clarity. In message 426, the assistant sees the error and initially suspects a path or permission issue. By message 427, the assistant has identified the real problem: "The issue is that both Kuri nodes share the same YCQL database but have separate local storage. When kuri-1 creates a group, it's recorded in the shared database, then kuri-2 tries to access it but doesn't have the local files."

In message 428, the assistant searches for a node_id concept in the codebase and finds that S3 objects have node_id but groups don't. This is the moment of discovery—the assistant realizes that the group system was never designed for multi-node operation.

In message 437, after the user rejects the separate-keyspaces approach, the assistant pivots: "I understand—all nodes share one keyspace, but node-specific tables need a node_id column. Let me revert and implement this properly."

But then, in message 446, the assistant confronts the full scope of the change. Reading through the configuration system, the database schema, and the RIBS query layer, the assistant realizes this is not a quick fix. The reasoning shows the assistant thinking through the options: "The simplest fix for the test cluster is to just not run kuri-2 for now, or to accept that this is a known limitation until we implement proper node_id support throughout the codebase."

This is the critical decision point. The assistant could have pressed forward with the node_id implementation, spending hours modifying schema, queries, configuration, and dependency injection. Instead, it chose to document the limitation, create a todo item, and focus on getting a working single-node cluster. This is a judgment call that prioritizes momentum over completeness—a common and often wise choice in complex system development.

Conclusion

Message 451 is, on its surface, a routine documentation update. But when read in context, it is the capstone of a significant architectural discovery and a pragmatic retreat. The assistant identified a fundamental flaw in the multi-node design—the lack of node-level isolation in the group system—and made the difficult decision to defer the fix rather than rush it. The README note that the assistant reads in this message is a promise: the architecture knows its own limitations, and the path to fixing them is documented. In a codebase as complex as a distributed S3 storage system, that kind of honest documentation is not a sign of failure—it is a sign of engineering maturity.