The Moment the Architecture Clicked: Wiring Dual Database Connections in a Horizontally Scalable S3 System

Subject Message: [assistant] Now update the Kuri plugin to use separate CQL connections: [edit] /home/theuser/gw/integrations/kuri/ribsplugin/kuboribs.go Edit applied successfully.

Introduction

In a coding session that spanned debugging test clusters, reverting misguided changes, and absorbing pointed user corrections, one message stands out as the inflection point where a distributed storage architecture finally snapped into focus. The message is deceptively short—barely a sentence followed by an edit command. But to understand its weight, one must trace the thread of reasoning that led to it: a journey through incorrect assumptions, frustrated corrections, and the hard-won clarity of a proper three-layer system design.

The subject of this article is message index 540 in the conversation, where the assistant writes: "Now update the Kuri plugin to use separate CQL connections:" and applies an edit to kuboribs.go. On its surface, this is a routine implementation step. In context, it represents the culmination of a fundamental architectural correction—the moment when the codebase finally aligned with the roadmap's vision of one shared S3 keyspace and N per-node RIBS keyspaces.

The Road to This Message

To understand why this message was written, we must first understand what went wrong before it. The assistant had been building a test cluster for a horizontally scalable S3 architecture, following a roadmap that specified stateless frontend proxies routing to independent Kuri storage nodes. The architecture was supposed to be clean: S3 frontend proxies on port 8078 → Kuri storage nodes → shared YugabyteDB.

But the assistant had been running Kuri nodes as direct S3 endpoints, violating the roadmap's requirement for separate stateless frontend proxy nodes. When the user pointed this out, the assistant restructured the docker-compose into a proper three-layer hierarchy. However, a deeper problem remained: all Kuri nodes were sharing the same database keyspace for their RIBS (Remote Indexed Block Storage) data—groups, deals, and blockstore metadata. This caused race conditions, deadlocks, and configuration validation errors.

The user's message at index 512 was blunt: "This makes no sense to me at all whatsoever. Groups are entirely separate to nodes, owned by nodes. There is one global table tracking them that is supposed to have a nodeid. All SQL/CQL APIs / calls related to groups/deals/etc MUST contain node ID. Or maybe easier we should segregate db/keyspace for kuri nodes at the RIBS layer and only share at S3 layer."

The assistant initially attempted to add node_id filtering to all RIBS queries—a reasonable but invasive approach that would require modifying every database call in the RIBS layer. Then the user's alternative suggestion of keyspace segregation took hold. But the assistant then made another wrong turn: it tried to use a single shared keyspace for everything, reasoning that it would be "simpler" for the test cluster.

The user's response at index 532 was even more pointed: "No the shared keyspace makes no sense ffs, you want ONE S3 KEYSPACE and N RIBS/Blockstore keyspaces, it really is not hard at all."

This was the critical correction. The assistant finally understood: the architecture required two separate database connections per Kuri node—one for the node's private RIBS keyspace (groups, deals, blockstore) and one for the shared S3 metadata keyspace used by all nodes and proxies.

What the Message Actually Does

Message 540 is the implementation of that understanding. The assistant had already added a S3CqlConfig to the configuration system in messages 536–539, creating a new configuration block with its own environment variable prefix (RIBS_S3_CQL_*) and a helper method GetS3CqlConfig() that falls back to the main YugabyteCql config if the S3-specific config isn't set. This was the plumbing.

Now, in message 540, the assistant wires that plumbing into the Kuri plugin's dependency injection. The file kuboribs.go is the Fx (Uber's dependency injection framework) wiring point for the Kuri storage node. It's where database connections are created and provided to the application's components. The edit applies the dual-connection pattern: instead of providing a single CQL database connection that everything shares, the plugin now provides two connections—one for RIBS operations (groups, deals, blockstore) using the per-node keyspace, and one for S3 object metadata using the shared keyspace.

The exact edit content isn't visible in the message text, but from the surrounding context we can infer what changed. Previously, makeCqlDb() created a single cqldb2.Database from configuration.GetConfig().YugabyteCql. After the edit, there would be a makeS3CqlDb() as well, creating a second connection from configuration.GetConfig().GetS3CqlConfig(). The makeS3ObjectIndex function would then receive the S3-specific connection instead of the RIBS connection.

Input Knowledge Required

To understand this message, one needs knowledge of several interconnected systems:

The Fx Dependency Injection Framework: The Kuri plugin uses Uber's Fx to wire together its components. Functions like makeCqlDb are registered as providers (fx.Provide(makeCqlDb)), and their return types are injected into components that request them. Understanding that adding a second provider function creates a second database connection available for injection is essential.

The Configuration System: The codebase uses envconfig tags to map environment variables to Go struct fields. The YugabyteCqlConfig struct has fields for hosts, port, keyspace, user, and password, all configurable via RIBS_YUGABYTE_CQL_* environment variables. The newly added S3CqlConfig duplicates this structure with RIBS_S3_CQL_* variables, and GetS3CqlConfig() provides fallback logic.

The CQL Database Layer: The cqldb2.Database interface abstracts YugabyteDB's CQL (Cassandra-compatible) connections. Each connection is bound to a specific keyspace at the driver level, meaning separate connections are required to access different keyspaces.

The S3 Object Index: The S3ObjectIndex interface is how Kuri nodes record and query object placement metadata. It needs to write to the shared S3 keyspace so that frontend proxies can query it for GET routing.

The Roadmap Document: The scalable-roadmap.md specifies that "Each RIBS node has its own keyspace for Blockstore level metadata, S3 metadata moves to a new s3 keyspace accessed by ALL Kuri AND S3 Frontend nodes." This message directly implements that specification.

Output Knowledge Created

This message creates several important pieces of knowledge:

The Dual-Connection Pattern: The codebase now has a documented pattern for how Kuri nodes connect to two separate database keyspaces simultaneously. This pattern becomes the template for all future nodes in the cluster.

The S3CqlConfig Wiring: The configuration system now supports an S3-specific CQL configuration that can either be set independently or fall back to the main CQL config. This provides flexibility for deployments where S3 metadata lives on the same cluster but in a different keyspace.

The Dependency Injection Structure: The Fx wiring in kuboribs.go now explicitly separates RIBS concerns from S3 concerns at the database connection level, making the architecture's intent clear in the code structure.

A Working Test Cluster Configuration: After this edit, the test cluster's gen-config.sh can generate per-node settings files where each Kuri node has its own RIBS_YUGABYTE_CQL_KEYSPACE (e.g., filecoingw_kuri1) while all share the same RIBS_S3_CQL_KEYSPACE (e.g., filecoingw_s3).

Assumptions and Mistakes

The most significant mistake in the lead-up to this message was the assistant's repeated attempt to simplify the architecture by using a single shared keyspace. This stemmed from an assumption that "simpler is better for testing"—that the test cluster could get away with a shared keyspace and the keyspace segregation could be added later. But the user correctly identified that this would defeat the entire purpose of the architecture: if all nodes share the same RIBS keyspace, they're not truly independent, and the system doesn't demonstrate the horizontally scalable pattern.

Another assumption was that the RIBS layer's internal queries could be retrofitted with node_id filtering without breaking existing functionality. The assistant had started down this path, modifying rbstor/db.go to add a nodeID field and filter queries. But this approach was invasive, error-prone, and would have required changes to every database call in the RIBS layer. The keyspace segregation approach is cleaner because it provides isolation at the database connection level rather than at the query level.

The assistant also initially misunderstood the user's statement that "Kuri nodes connect to both the ribs db/keyspaces AND s3 keyspaces." The assistant interpreted this as a future requirement rather than an immediate architectural necessity, leading to the abortive attempt to use a shared keyspace "for now."

The Thinking Process

The assistant's reasoning in the messages leading up to 540 shows a gradual dawning of understanding. In message 521, after the user notes that Kuri nodes connect to both keyspaces, the assistant writes: "So each Kuri node needs TWO database connections." But then it immediately backtracks: "For the test cluster to work simply, let me use the shared keyspace for now."

This is the critical error in reasoning—the assumption that architectural correctness can be deferred. The user's frustration in message 532 ("it really is not hard at all") suggests that the assistant was overcomplicating what should have been a straightforward implementation.

After that correction, message 533 shows the assistant finally committing to the correct architecture: "You're right, sorry. Let me implement it properly: 1 shared S3 keyspace - filecoingw_s3 - for S3 object routing... N RIBS keyspaces - filecoingw_kuri1, filecoingw_kuri2 - for groups/deals/blockstore."

Then the assistant methodically works through the implementation: adding the S3 CQL config to configuration/config.go (messages 536–539), then updating the Kuri plugin's Fx wiring (message 540), and finally updating the test cluster configuration (messages 542–544). The thinking is systematic: configuration first, then dependency injection, then deployment configuration.

Why This Message Matters

Message 540 is the keystone of the architectural correction. Without it, the configuration changes in messages 536–539 would be unused code—a second CQL config defined but never instantiated. With it, the entire system snaps into alignment: each Kuri node starts with two database connections, one for its private RIBS data and one for the shared S3 metadata. The frontend proxies can query the shared S3 keyspace to route GET requests to the correct node. The per-node keyspaces ensure that group and deal operations don't conflict across nodes.

The message also demonstrates an important principle in collaborative development: sometimes the most valuable contribution is not the code itself but the architectural clarity it enforces. The edit to kuboribs.go is small, but it encodes a design decision that affects every component in the system. The dual-connection pattern becomes part of the codebase's DNA, shaping how future developers think about database access in the distributed S3 architecture.

Conclusion

The subject message—"Now update the Kuri plugin to use separate CQL connections"—is a masterclass in the power of small, focused commits that encode significant architectural decisions. It came after a series of incorrect assumptions, user corrections, and conceptual breakthroughs. It represents the moment when the codebase stopped fighting the architecture and started embodying it. The dual-database-connection pattern it establishes is the foundation upon which the entire horizontally scalable S3 system is built: one shared keyspace for object routing, N independent keyspaces for storage node data, and a clean separation of concerns that allows both layers to scale independently.