Keyspace Segregation and Dual Connections: How 14 Commits Rescued a Horizontally Scalable S3 Architecture

Introduction

In the previous segment of this coding session, the assistant and user built and iteratively debugged a test cluster for a horizontally scalable S3 architecture, correcting a major architectural error that had embedded the S3 API directly inside Kuri storage nodes. The stateless S3 frontend proxies were properly separated from the Kuri storage nodes, each with independent configurations. But as this segment opens, the system is still not functional. The test cluster fails to start both Kuri nodes simultaneously. Port 9010 (the Kuri LocalWeb monitoring UI) returns "connection refused," and port 8078 (the S3 frontend proxy) returns "internal server error." The three-layer architecture—stateless proxies, independent storage nodes, and a shared YugabyteDB—exists in code but does not work in practice.

This segment of the conversation captures the transition from "the code compiles" to "the system is ready to start." The work spans several interlocking threads: building the S3 frontend proxy binary and integrating it into the test cluster infrastructure, discovering that all Kuri nodes were sharing the same database keyspace (causing deadlocks and race conditions), implementing a dual CQL connection architecture to segregate per-node RIBS keyspaces from the shared S3 metadata keyspace, staging all changes into 14 logical git commits, and fixing critical bugs in the CQL schema, multipart upload coordination, health checking, and response headers. Each thread represents a distinct engineering challenge, and together they tell a story about what it really means to build a distributed system—not just designing the architecture on paper, but making it actually work through iterative debugging, architectural correction, and disciplined version control.

The Missing Entry Point: From Library Code to Runnable Binary

One of the most striking patterns in this segment is the repeated discovery that components which were "implemented" in source code were not actually deployable. The S3 frontend proxy package (server/s3frontend/) contained seven well-structured Go files—a server with S3 request handlers, a router with YCQL integration for object lookup, a backend pool for Kuri node connection management with health checks, multipart upload coordination, and dependency injection wiring. But as the assistant discovered through a series of file reads and glob searches, there was no main.go entry point to build it as a standalone binary.

This gap between "code exists" and "system works" is a recurring theme throughout the segment. The docker-compose.yml for the test cluster contained a placeholder command that simply echoed a message and slept forever: command: ["sh", "-c", "echo 'S3 Frontend Proxy - requires s3frontend binary' && sleep infinity"]. The entire three-layer architecture—stateless S3 frontend proxies routing requests to independent Kuri storage nodes backed by a shared YugabyteDB—was blocked by the absence of a single file.

The assistant's response to this gap reveals a methodical, pattern-driven approach to software development. Rather than guessing at the structure of the entry point, the assistant studied the existing project conventions. It read the Makefile to understand how binaries were built. It examined the Kuri node's main.go at integrations/kuri/cmd/kuri/main.go to understand the project's entry point conventions. It searched for all existing main.go files in the project to confirm the pattern. It read the configuration package to understand the S3APIConfig struct and its environment variable bindings. Each read operation answered a specific question needed to make the upcoming implementation decision.

When the assistant finally wrote the main.go file, the Language Server Protocol immediately flagged multiple compilation errors: unused imports, a mismatched function call signature for s3.NewAuthenticator (which returns two values, not one), a pointer type mismatch, and a reference to a non-existent GetNodeIDs() method. These errors are not failures—they are the natural friction of integrating separate components. The assistant's debugging cycle—read the actual API signatures, fix the code, rebuild—is the engine that drives the entire session. The build eventually succeeds, and the assistant immediately pivots to integrate the binary into the Docker Compose infrastructure.

The Deployment Pipeline: Thinking in Systems

What distinguishes this segment from a simple "write code, compile, done" narrative is the assistant's systematic attention to the deployment pipeline. After the build succeeds, the assistant does not declare victory. Instead, it traces the chain from source code to running container: source → build (Makefile) → binary → Docker image (Dockerfile) → container (docker-compose.yml).

The Dockerfile check is a revealing moment. The assistant has just updated docker-compose.yml to run s3-proxy as the command for the proxy service. But the binary was built on the host machine, and the Docker Compose test cluster runs inside containers. If the Docker image does not contain the s3-proxy binary, the container will fail to start with a "command not found" error. The assistant's decision to check the Dockerfile before restarting the cluster is a classic example of proactive debugging: finding and fixing problems before they manifest as runtime errors.

This systematic thinking extends to the todo list management that runs throughout the segment. The assistant maintains a persistent todo list with four items: create the main.go entry point, update the Makefile, update docker-compose.yml, and test the full cluster flow. Each status transition—from "pending" to "in_progress" to "completed"—provides a clear, visible chain of dependencies. The todo list is not just a log; it is a cognitive artifact that helps the assistant manage its own working memory across a session spanning hundreds of messages.

The assistant also updates the gen-config.sh script to generate proper per-node configurations, maps Kuri LocalWeb ports from 8443/8444 to 7001/7002 per the user's request, and updates the chain API endpoint from the defunct api.chain.love to pac-l-gw.devtty.eu. Each of these changes is small in isolation, but together they represent the difference between a theoretical architecture and a running system.

The Keyspace Revelation: A Fundamental Architectural Correction

Just as the assistant was completing the proxy binary integration, a deeper architectural issue emerged. When the test cluster was started with both Kuri nodes, the second node failed to start due to YugabyteDB migration deadlocks and a configuration validation error around RetrievableRepairThreshold. The assistant initially attributed these failures to startup ordering and configuration issues, implementing fixes like sequential startup and log level corrections.

But the root cause was more fundamental. Through a series of diagnostic investigations, the assistant discovered that all Kuri nodes were sharing the same database keyspace. This meant that both nodes were writing to the same groups table, the same deals table, and the same blockstore—causing deadlocks, race conditions, and data corruption. The architecture had been designed with the assumption that nodes would be isolated, but the implementation had not enforced that isolation.

The user's intervention was critical at this point. The user clarified that groups are per-node resources—each Kuri node must have its own database keyspace for RIBS data (deals, groups, blockstore), while only the S3 metadata keyspace should be shared between nodes for object routing. This distinction between per-node keyspaces and a shared keyspace is the architectural key that unlocks horizontal scalability.

The conversation that follows this revelation is a masterclass in architectural decision-making under pressure. The assistant initially proposes adding a node_id column to every RIBS database table and filtering all queries by node ID. This approach would keep a single keyspace but partition data at the row level. The user pushes back, clarifying that the correct approach is keyspace segregation—each node gets its own filecoingw_{node_id} keyspace for RIBS data, while the filecoingw_s3 keyspace is shared.

The assistant implements this approach, but then the user corrects again: the RIBS layer already has a node_id column in the groups table, and the correct approach is to use node-ID filtering within a single keyspace, not separate keyspaces per node. The assistant reverts the keyspace segregation and implements node-ID filtering instead.

Then the user clarifies again: the architecture actually requires both—separate keyspaces per node for RIBS data AND a shared keyspace for S3 metadata, with dual CQL connections to support both. This is the final, correct design: each Kuri node connects to its own filecoingw_{node_id} keyspace for deals, groups, and blockstore data, while all nodes share the filecoingw_s3 keyspace for S3 object routing metadata. The assistant implements this by adding a second CQL connection configuration, creating an S3CqlDB wrapper type, and wiring dual CQL connections through the dependency injection system.

This back-and-forth is not a failure of communication. It is the natural process of refining an architectural design through concrete implementation. The user has a clear mental model of the system, but that model only becomes fully specified when it meets the resistance of actual code. Each correction from the user reveals a subtlety that was not captured in the earlier discussions: the RIBS layer already has node_id support, the keyspace segregation must happen at the CQL connection level, and the S3 index needs its own database connection to the shared keyspace. The assistant's willingness to implement, revert, and re-implement is essential to this process.## Dual CQL Connections: The Implementation That Made It Work

The implementation of dual CQL connections is the most architecturally significant code change in this segment. The assistant needed to modify the database layer to support two separate connections: one for the per-node RIBS keyspace and one for the shared S3 metadata keyspace. This required changes to the configuration system, the database abstraction layer, and the dependency injection wiring.

The pivotal realization came when the assistant inspected the code to understand how the S3 object index was created. The function makeS3ObjectIndex in kuboribs.go took a single cqldb2.Database parameter and passed it directly to the S3 index constructor. There was no mechanism for the S3 index to use a different database connection than the RIBS layer. The assistant's reasoning was explicit: "The S3 index uses the same CQL database as RIBS currently. I need to check how the S3 index is created and if it can use a different keyspace."

This discovery transformed the problem from a configuration issue to a code architecture issue. The keyspace segregation that had been configured in gen-config.sh—where each node gets its own filecoingw_{node_id} keyspace plus a shared filecoingw_s3 keyspace—could not work without changing how database connections were created and injected into the Kuri plugin.

The solution required several coordinated changes across the codebase:

Configuration layer: A new S3CqlConfig struct was added to configuration/config.go, with its own set of environment variable bindings (FGW_S3_YCQL_HOSTS, FGW_S3_YCQL_PORT, FGW_S3_YCQL_KEYSPACE, etc.) and a fallback to the existing YugabyteCql settings if not explicitly configured. This allowed the S3 metadata keyspace to be configured independently from the RIBS keyspace.

Database abstraction: A new S3CqlDB wrapper type was created to represent the separate S3 CQL connection. This type wraps the same underlying cqldb2.Database interface but is semantically distinct—it is used exclusively for S3 object metadata queries against the shared keyspace.

Dependency injection: The Kuri plugin's fx module was updated to provide both connections. A new makeS3CqlDb() function was added alongside the existing makeCqlDb() function, and both were wired through fx.Provide(). The makeS3ObjectIndex() function was updated to receive the S3 CQL connection instead of the RIBS connection.

The verification of this change shows the assistant tracing through the codebase with a grep to confirm that every database call goes to the correct keyspace. This kind of systematic verification—checking not just that the code compiles, but that the architectural intent is correctly implemented across all layers—is the hallmark of careful engineering.

Each Kuri node now maintains two database connections: one for its own per-node RIBS keyspace (containing groups, deals, and blockstore data) and one for the shared S3 metadata keyspace used by all nodes and proxies. This architectural correction was the foundation upon which all subsequent work was built. Without it, the system could not scale beyond a single node without data corruption.

The 14 Commits: Decomposing Architecture into Atomic Changes

With the architectural design clarified and the dual CQL connections implemented, the assistant faced the task of capturing all changes in a clean, logical git history. The user's instruction was explicit: "make commits for all changes so far." This was not just about version control hygiene—it was about creating a recoverable checkpoint before diving into bug fixes.

The assistant decomposed the work into 14 logical commits, each representing a distinct architectural layer:

  1. Configuration (bc51033): Added S3CqlConfig for a separate S3 metadata keyspace, with 47 insertions and 2 deletions across 1 file.
  2. Interface (b440a27): Added NodeID and ExpiresAt to the S3Object interface in iface/s3.go, defining the contract for multi-node routing.
  3. Kuri S3 plugin (ca5098e): Wired node_id through the S3 object index, bucket handler, and fx module across 3 files with 29 insertions.
  4. Dual CQL connections (13d020a): Added separate S3 CQL connection in kuboribs, including the S3CqlDB wrapper type and makeS3CqlDb() function.
  5. S3 frontend proxy (6beaefc): The entire stateless proxy package—1,150 lines across 8 files covering the server, router, backend pool, multipart tracker, and dependency injection.
  6. Build system (9f7e888): Added s3-proxy binary to Makefile and Dockerfile, completing the deployment pipeline.
  7. Test cluster (2083415): Docker Compose setup for multi-node testing with scripts, configurations, and per-node isolation.
  8. Documentation (3b2705d): Updated README with scalable roadmap and architecture documentation.
  9. Environment support (302ceee): Added DATA_DIR variable for flexible deployment.
  10. Database migration (a12507c): Added node_id column to groups table in the SQL migration.
  11. Monitoring API (f53562a): Cluster monitoring RPC methods for observability.
  12. Monitoring dashboard (59d35c9): Web-based cluster monitoring UI with React components.
  13. CQL schema fixes (c3fff2a): Added node_id and expires_at to S3Objects table, created MultipartUploads table.
  14. Endpoint fixes (c75739f): Added /healthz endpoint and X-Node-ID response header. This decomposition was itself an architectural statement. Each commit corresponded to a distinct concern: configuration defines the parameters, interfaces define the contracts, plugins implement the contracts, wiring connects the layers, the proxy provides the stateless routing tier, the build system makes it deployable, and the test infrastructure makes it testable. The assistant attempted to parallelize some of these commits using subagents, but the user immediately corrected this: "do not do commits in parallel." The user understood that a clean, linear git history is a form of communication with future developers, and that parallel commits introduce ordering anomalies that obscure the narrative of development. The assistant respected this constraint and produced a linear sequence of commits that tells the story of the system's construction in a coherent, reviewable form.

The Bug Fixes: Schema, Endpoints, and Observability

With all prior work committed, the assistant turned to the critical bugs that were preventing the cluster from starting. These were the issues identified in the alignment assessment and confirmed by the user's report of UI failures.

The S3Objects Schema Fix

The most critical bug was the missing node_id and expires_at columns in the S3Objects CQL migration. Without node_id, the frontend proxy's router had no way to determine which Kuri node stored a given object. Every GET request would fail because the proxy could not find the target backend. Every PUT request would silently discard the routing information.

The fix was straightforward in code terms—adding two columns to a CREATE TABLE statement—but architecturally profound. The node_id column is the routing key that connects the stateless proxy layer to the storage node layer. The expires_at column enables time-based garbage collection of temporary multipart parts. The updated migration now matched the S3Object struct in iface/s3.go, ensuring schema-code alignment.

The assistant found the migration file at database/cqldb/migrations/1754293669_init_s3_object_index.up.cql and updated it to include both columns. The corrected schema now reads:

create table S3Objects
(
    bucket     text,
    key        text,
    cid        text,
    size       bigint,
    updated    timestamp,
    node_id    text,
    expires_at timestamp,
    primary key (bucket, key)
);

This migration is applied when the database is initialized for the first time, which is why the roadmap explicitly states "no migration files - new deployments only." Existing deployments would need a separate ALTER TABLE migration, but for the test cluster and new deployments, the CREATE TABLE statement is sufficient.

The MultipartUploads Table

The second critical bug was the complete absence of a MultipartUploads table. The user explained why this table was never needed before: "it wasn't needed before because we used unixfs dags which had all related blocks in one blockstore but now it's no longer the case / we have multiple blockstores."

This explanation reveals the architectural shift that made the new table essential. In the previous design, all blocks of a multipart upload lived in a single blockstore. With multiple independent Kuri storage nodes, each having its own isolated blockstore, parts could be distributed across nodes. The MultipartUploads table provides a globally visible coordination primitive that any S3 proxy or Kuri node can query, regardless of which node originally initiated the upload.

The new migration creates a table that tracks upload state across nodes:

create table MultipartUploads
(
    upload_id   text,
    bucket      text,
    key         text,
    initiated   timestamp,
    node_id     text,
    primary key (upload_id)
);

This table is the coordination backbone for cross-node multipart uploads, enabling any proxy to discover the status of an in-progress upload and any node to complete an assembly that was initiated on a different node.

The /healthz Endpoint

The third fix addressed a missing operational endpoint. The S3 frontend proxy's backend pool maintained a list of healthy Kuri nodes by polling their /healthz endpoints. But no such endpoint existed on the Kuri S3 server. The proxy was trying to check node health by hitting a URL that returned 404 or connection refused, and the error propagated upward as an internal server error on port 8078.

The fix was minimal—a single handler returning HTTP 200 with "OK"—but it was the missing link in the proxy's health-checking mechanism. Without it, the proxy could not distinguish a healthy node from a crashed one, and all requests would fail.

The implementation followed the existing handler pattern in the Kuri S3 server, adding a new route to the router and a simple handler function. The health check endpoint is intentionally minimal: it does not check database connectivity, disk space, or any other internal state. It simply confirms that the HTTP server is running and accepting connections. This is sufficient for the proxy's backend pool, which performs its own health tracking based on response success rates.

The X-Node-ID Header

The fourth fix added an X-Node-ID HTTP response header to all S3 operations. This header provides observability into routing decisions: clients can see which Kuri node handled their PUT, GET, or DELETE request. In a multi-node cluster, this information is crucial for debugging, monitoring, and verifying that the routing layer is correctly directing requests.

The implementation followed a clean three-layer pattern: adding NodeID() string to the Region interface, implementing it on the concrete Region struct (returning the nodeID field that had been wired through the configuration chain), and adding the header to five HTTP handlers in server/s3/handlers.go.

The header is set on every S3 response, regardless of whether the request succeeded or failed. This ensures that even error responses carry the node identification, which is essential for debugging in a multi-node environment.

The Verification: Subagent Investigations and Docker Builds

After the bug fixes were committed, the assistant engaged in a series of verification steps that reveal the discipline of the development process.

First, a subagent was dispatched to investigate why integrations/kuri/ribsplugin/s3/region.go had not been included in the endpoint fixes commit. The assistant had assumed the file was in a gitignored directory. The subagent's investigation revealed the opposite: the file was already committed, the working tree was clean, and there were no uncommitted changes. The assumption was wrong, and the subagent provided the correction. This is a powerful example of how subagents can be used to verify assumptions and prevent incorrect conclusions from propagating.

Second, the Docker image was rebuilt to verify that all changes compiled and that the s3-proxy binary was correctly included in the final image. The build output showed three COPY steps—kuri, gwcfg, and s3-proxy—confirming that the stateless frontend proxy was now part of the deployable artifact. The image SHA b2a663c27adb... became a historical marker: the first build of the corrected three-layer architecture.

The assistant then issued a git log --oneline -20 command to display the commit history. The seven visible commits told the story of a system coming together: health checks for operational reliability, schema migrations for data integrity, monitoring for observability, configuration for deployability, and documentation for maintainability.

The Final Handoff: Ready for Testing

The segment concludes with the assistant's summary of all 14 commits and the announcement that the Docker image was rebuilt and ready for testing. The startup command was provided: cd test-cluster && ./start.sh /data/fgw2. The user noted that the public address is filecoingateway.devtty.eu and that the start script will have an interactive step after first startup, which is fine but should not be run automatically.

The test cluster is now ready for its first real test. The architecture correctly implements the roadmap's three-layer hierarchy:

Conclusion: The Architecture That Wouldn't Stay Dead

The work in this segment represents a complete arc of distributed systems development: from a broken architecture with shared keyspaces and missing connections, through a fundamental redesign requiring dual database connections, through systematic decomposition into 14 logical commits, through targeted bug fixes addressing schema gaps and missing endpoints, to a verified Docker build ready for testing.

The key themes that emerge are:

Database keyspace segregation was the architectural insight that saved the system. Each Kuri node now has its own RIBS keyspace for private data while sharing an S3 metadata keyspace for object routing. This prevents race conditions and enables true horizontal scaling.

Dual CQL connections were the implementation mechanism that made keyspace segregation possible. The codebase was restructured to support two independent database connections per node, each targeting a different keyspace. This required changes to the configuration system, the database abstraction layer, and the dependency injection wiring.

Git discipline transformed chaotic debugging into a coherent architectural narrative. The 14 commits, each representing a logical unit of change, serve as both a checkpoint and a documentation artifact. The commit structure itself becomes a form of documentation—a map of how the system is organized and what each component depends on.

Schema and endpoint fixes closed the gap between the architectural design and the running system. The S3Objects schema, MultipartUploads table, /healthz endpoint, and X-Node-ID header were the missing pieces that made the cluster functional. Each fix was small in isolation, but together they transformed a non-functional cluster into a system ready for testing.

The human-AI collaboration was essential to the success of this segment. The user provided the architectural vision and corrected the assistant's assumptions at critical junctures. The assistant provided the implementation discipline, systematic debugging, and attention to detail. The back-and-forth between user corrections and assistant re-implementations produced a design that was more refined than either could have achieved alone.

The architecture that wouldn't stay dead—that kept demanding corrections, new connections, and schema fixes—finally stabilized into a coherent, committed, and built system ready for testing. The 14 commits in the git log are not just a record of changes; they are a map of the architectural decisions, corrections, and implementations that define the horizontally scalable S3 system. The test cluster awaits its first startup, and with it, the first real validation of whether the three-layer architecture can deliver on its promise of horizontal scalability.