The Architecture That Couldn't Scale: A Strategic Retreat in Distributed Systems Design

The Message

[assistant] ## Agent Reasoning
Let me update the components section to reflect single node.
[edit] /home/theuser/gw/test-cluster/README.md
Edit applied successfully.

At first glance, this message from an AI-assisted coding session appears trivial—a simple documentation update, a README edit that changes a few words. But this brief exchange is the quiet culmination of a much larger struggle. It represents the moment when an ambitious distributed systems architecture met the hard reality of code complexity, and a deliberate decision was made to retreat, document the limitation, and move forward with a working subset rather than chase an elusive ideal.

The Context: Building a Horizontally Scalable S3

The assistant had been constructing a test cluster for a horizontally scalable S3 storage system built on the Filecoin Gateway platform. The architecture, as specified in the project's roadmap, called for a three-layer hierarchy: stateless S3 frontend proxies on port 8078 that could scale horizontally, independent Kuri storage nodes that held the actual data, and a shared YugabyteDB cluster for coordination and metadata. The vision was elegant: any number of proxy nodes could route requests to any number of Kuri storage nodes, all coordinated through a single database.

The test cluster initially specified two Kuri nodes (kuri-1 and kuri-2), each with its own local data directory but sharing a YugabyteDB instance. This seemed straightforward—until the second node refused to start.

The Error That Exposed the Flaw

When kuri-2 attempted to initialize, it failed with a cryptic error:

Error: write to group: opening group: opening group: open jbob (grp: /data/ribs/grp/1): opening head: open /data/ribs/grp/1/blklog.meta/head: no such file or directory

The problem was architectural. Both Kuri nodes shared the same YugabyteDB keyspace. When kuri-1 created a group (group 1), it recorded that group in the shared database. When kuri-2 started up, it read the group listing from the database, found group 1, and tried to open its local files—but those files only existed on kuri-1's local storage. Each node had its own /data/ribs directory mounted from the host, so kuri-2 was trying to access files that simply didn't exist on its filesystem.

This was not a configuration bug or a startup race condition. It was a fundamental design problem in how the RIBS (Reliable Immutable Block Store) layer managed groups across multiple nodes.

The Failed Attempt: Separate Keyspaces

The assistant's first instinct was to solve the problem by giving each Kuri node its own database keyspace. The reasoning was straightforward: if each node had isolated storage for groups and deals, they couldn't interfere with each other. The gen-config.sh script was updated to generate per-node keyspace configurations, and the docker-compose.yml was modified to initialize separate keyspaces (filecoingw_kuri1 and filecoingw_kuri2) alongside a shared filecoingw_s3 keyspace for object routing metadata.

This approach had surface-level appeal. It preserved the shared S3 metadata that the proxy needed for object routing while isolating each node's internal state. But it violated a deeper architectural principle: in a properly designed distributed system, nodes should be able to discover each other's state through a shared coordination layer. Separate keyspaces meant nodes were blind to each other's existence.

The User's Correction: One Keyspace, Node IDs

The user rejected the separate keyspaces approach with a clear directive: "No, all nodes definitely should use a single shared keyspace—they must know about their keys. Tables which are e.g. kuri-specific should have a node id entry in tables."

This was the correct architectural insight. In a horizontally scalable system, the database is the source of truth. All nodes read from and write to the same database, but they filter by their own identity. Tables that hold node-specific data (groups, deals, blockstore metadata) need a node_id column so that each node only operates on its own records. The shared keyspace allows the S3 proxy to query object placement across all nodes, while the node_id column ensures each node only touches its own data.

The assistant immediately understood the correction and began implementing it. A new database migration was created (1750766516_add_node_id_to_groups.up.sql) to add a node_id column to the groups table. The assistant began reading the rbstor/db.go file to understand how to add node_id filtering to all database queries.

The Realization: Scope Too Large

Then came the critical moment of recognition. As the assistant traced through the codebase, the full scope of the change became apparent. Adding node_id to groups required:

1. A database schema migration (already created)
2. Adding a NodeID field to the RibsConfig configuration structure
3. Modifying the RbsDB struct to carry a node identifier
4. Updating every database query in rbstor/db.go to filter by node_id
5. Updating the rbstor/rbs.go initialization to pass the node ID
6. Updating the configuration generation scripts to include node IDs
7. Ensuring the S3 proxy could still query across all nodes for object routing This was not a quick fix. It was a cross-cutting change that touched configuration, database access, business logic, and deployment scripts. The assistant's reasoning shows the moment of reckoning: "Given the complexity of the changes needed... I think we should: 1. Document that the test cluster currently only supports 1 Kuri node, 2. Create a todo list for implementing full node_id support, 3. Focus on making single-node work properly first."

The Decision to Document and Proceed

The subject message—"Let me update the components section to reflect single node"—is the final step in this strategic retreat. The assistant had already:

• Removed kuri-2 from docker-compose.yml
• Updated the start.sh script for single-node operation
• Updated the architecture diagram in the README
• Added a note about the multi-node limitation Now it was updating the components section to consistently describe a single-node architecture. The README would honestly state: "Note: Multi-node support (kuri-2) requires implementing node_id in RIBS groups." This documentation update was not an admission of failure. It was a responsible engineering decision. Rather than ship a broken two-node cluster or delay the entire project while implementing a complex cross-cutting change, the assistant chose to deliver a working single-node system with clearly documented limitations. The node_id work was tracked as a future task, the migration file was already created, and the path forward was understood.

The Deeper Lesson: Distributed Systems Are Hard

This message illuminates a fundamental truth about distributed systems engineering: the gap between architectural vision and working implementation is often vast. The three-layer hierarchy with horizontal scaling at every level is a beautiful design on paper. But the devil is in the details—specifically, in how shared state is partitioned across nodes.

The RIBS group system had been designed for a single-node deployment. Groups were identified by a sequential ID (serial primary key), with no concept of ownership. When the system was extended to multiple nodes, this assumption broke. The groups table needed a node_id column, but that column didn't exist, and adding it required changes throughout the stack.

The assistant's journey through this problem demonstrates a mature engineering approach: understand the error, propose a fix, receive architectural guidance, attempt the proper solution, recognize the scope, and make a pragmatic decision to deliver value now while documenting the path forward. The README update in the subject message is the artifact of that decision—a small change that represents hours of debugging, analysis, and architectural deliberation.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

• The RIBS storage layer: How groups are created, stored, and accessed in the Reliable Immutable Block Store
• The YugabyteDB schema: How the groups table is defined without a node_id column, and how migrations work
• The configuration system: How RibsConfig is structured and how node-specific settings are passed
• The Docker Compose test cluster: How volumes, environment variables, and service dependencies are orchestrated
• The S3 routing architecture: How the stateless proxy uses database queries to route GET requests to the correct storage node

Output Knowledge Created

This message produces a clear documentation artifact: the test cluster README now accurately reflects that multi-node operation requires node_id support in the RIBS group system. This documentation serves multiple purposes:

1. Transparency: Anyone reading the README understands the current limitation
2. Roadmap clarity: The next development step is clearly identified
3. Knowledge preservation: The architectural decision and its rationale are captured
4. Scope management: The team knows what's working now and what needs future work

Conclusion

The message "Let me update the components section to reflect single node" is deceptively simple. It is the visible output of a complex reasoning chain that touched database schema design, distributed systems architecture, configuration management, and pragmatic engineering judgment. It represents the moment when an AI assistant and a human user collaboratively recognized that the proper solution to a distributed systems problem—adding node_id to shared tables—was too extensive for the current session, and chose to deliver a working system with honest documentation rather than a broken system with ambitious scope. In the world of distributed systems, knowing when to document a limitation and move forward is as valuable as knowing how to fix it.