The Architecture That Wouldn't Stay Dead: How 14 Commits, Two Database Connections, and a Handful of Schema Fixes Rescued a Distributed S3 System

Introduction

In the span of a few intense hours, a distributed S3 architecture was built, broken, corrected, rebuilt, and finally committed to version control in a cascade of 14 logical git commits. This chunk of the coding session captures the moment when a horizontally scalable storage system transitioned from a fundamentally flawed design—where all Kuri nodes shared a single database keyspace and the S3 frontend was embedded in storage nodes—to a clean three-layer hierarchy of stateless proxies, independent storage nodes, and a shared metadata database. The journey was not linear. It involved a user forcefully correcting the assistant's architectural assumptions, a deep investigation into how database connections were wired through the codebase, a systematic decomposition of changes into atomic commits, and a series of targeted bug fixes that addressed the critical failures preventing the test cluster from starting.

This article synthesizes the work across this chunk, tracing the arc from architectural crisis to coherent implementation, and examining the decisions, assumptions, and corrections that shaped the final result.

The Architectural Crisis: Shared Keyspaces and Missing Connections

The chunk opens with a system in crisis. The test cluster was failing: port 9010 (the Kuri LocalWeb monitoring UI) returned "connection refused," and port 8078 (the S3 frontend proxy) returned "internal server error" [37]. These were not cosmetic issues. They indicated that the entire three-layer architecture was non-functional.

The root cause traced back to a fundamental architectural error. Earlier in the session, the assistant had configured Kuri nodes to act as direct S3 endpoints, violating the roadmap's requirement for separate stateless frontend proxy nodes. The user caught this and insisted on a complete redesign. But even after the proxy layer was separated, deeper problems remained.

The most critical issue was database keyspace isolation. All Kuri nodes were sharing the same database keyspace, causing race conditions on shared group resources. The user clarified the correct design in stark terms: groups are per-node resources, and the database must be segregated so that each node gets its own keyspace for deals, groups, and blockstore data, while sharing only the S3 metadata keyspace for object routing [1]. This was not a minor configuration tweak—it required fundamental changes to how database connections were created and wired through the system.

The Moment of Discovery: Two Database Connections

The pivotal realization came when the assistant inspected the code to understand how the S3 object index was created. The function makeS3ObjectIndex in kuboribs.go took a single cqldb2.Database parameter and passed it directly to the S3 index constructor [1]. There was no mechanism for the S3 index to use a different database connection than the RIBS layer. The assistant's reasoning was explicit: "The S3 index uses the same CQL database as RIBS currently. I need to check how the S3 index is created and if it can use a different keyspace" [1].

This discovery transformed the problem from a configuration issue to a code architecture issue. The keyspace segregation that had been configured in gen-config.sh—where each node gets its own filecoingw_{node_id} keyspace plus a shared filecoingw_s3 keyspace—could not work without changing how database connections were created and injected into the Kuri plugin [2].

The solution required adding a second CQL connection configuration, creating an S3CqlDB wrapper type, and wiring dual CQL connections through the Kuri plugin's dependency injection framework [3][4]. Each Kuri node would now maintain two database connections: one for its own per-node RIBS keyspace (containing groups, deals, and blockstore data) and one for the shared S3 metadata keyspace used by all nodes and proxies [5].

This architectural correction was the foundation upon which all subsequent work was built. Without it, the system could not scale beyond a single node without data corruption.

The 14 Commits: Decomposing Architecture into Atomic Changes

With the architectural design clarified, the assistant faced the task of capturing all changes in a clean, logical git history. The user's instruction was explicit: "make commits for all changes so far" [38]. This was not just about version control hygiene—it was about creating a recoverable checkpoint before diving into bug fixes.

The assistant decomposed the work into 14 logical commits, each representing a distinct architectural layer [51]:

1. Configuration (bc51033): Added S3CqlConfig for a separate S3 metadata keyspace
2. Interface (b440a27): Added NodeID and ExpiresAt to the S3Object interface
3. Kuri S3 plugin (ca5098e): Wired node_id through the S3 object index
4. Dual CQL connections (13d020a): Added separate S3 CQL connection in kuboribs
5. S3 frontend proxy (6beaefc): The entire stateless proxy package (1,150 lines across 8 files)
6. Build system (9f7e888): Added s3-proxy binary to Makefile and Dockerfile
7. Test cluster (2083415): Docker Compose setup for multi-node testing
8. Documentation (3b2705d): Updated README with scalable roadmap
9. Environment support (302ceee): Added DATA_DIR variable
10. Database migration (a12507c): Added node_id column to groups table
11. Monitoring API (f53562a): Cluster monitoring interfaces
12. Monitoring dashboard (59d35c9): Web-based cluster monitoring UI
13. CQL schema fixes (c3fff2a): Added node_id/expires_at to S3Objects, created MultipartUploads table
14. Endpoint fixes (c75739f): Added /healthz endpoint and X-Node-ID header This decomposition was itself an architectural statement. Each commit corresponded to a distinct concern: configuration defines the parameters, interfaces define the contracts, plugins implement the contracts, wiring connects the layers, the proxy provides the stateless routing tier, the build system makes it deployable, and the test infrastructure makes it testable [38]. The assistant attempted to parallelize some of these commits using subagents, but the user immediately corrected this: "do not do commits in parallel" [39]. The user understood that a clean, linear git history is a form of communication with future developers, and that parallel commits introduce ordering anomalies that obscure the narrative of development.

The Bug Fixes: Schema, Endpoints, and Observability

With all prior work committed, the assistant turned to the critical bugs that were preventing the cluster from starting. These were the issues identified in the alignment assessment and confirmed by the user's report of UI failures [37].

The S3Objects Schema Fix

The most critical bug was the missing node_id and expires_at columns in the S3Objects CQL migration [43]. Without node_id, the frontend proxy's router had no way to determine which Kuri node stored a given object. Every GET request would fail because the proxy could not find the target backend. Every PUT request would silently discard the routing information.

The fix was straightforward in code terms—adding two columns to a CREATE TABLE statement—but architecturally profound. The node_id column is the routing key that connects the stateless proxy layer to the storage node layer. The expires_at column enables time-based garbage collection of temporary multipart parts. The updated migration now matched the S3Object struct in iface/s3.go, ensuring schema-code alignment [43].

The MultipartUploads Table

The second critical bug was the complete absence of a MultipartUploads table [44]. The user explained why this table was never needed before: "it wasn't needed before because we used unixfs dags which had all related blocks in one blockstore but now it's no longer the case / we have multiple blockstores" [44].

This explanation reveals the architectural shift that made the new table essential. In the previous design, all blocks of a multipart upload lived in a single blockstore. With multiple independent Kuri storage nodes, each having its own isolated blockstore, parts could be distributed across nodes. The MultipartUploads table provides a globally visible coordination primitive that any S3 proxy or Kuri node can query, regardless of which node originally initiated the upload [44].

The /healthz Endpoint

The third fix addressed a missing operational endpoint [45]. The S3 frontend proxy's backend pool maintained a list of healthy Kuri nodes by polling their /healthz endpoints. But no such endpoint existed on the Kuri S3 server. The proxy was trying to check node health by hitting a URL that returned 404 or connection refused, and the error propagated upward as an internal server error on port 8078.

The fix was minimal—a single handler returning HTTP 200 with "OK"—but it was the missing link in the proxy's health-checking mechanism. Without it, the proxy could not distinguish a healthy node from a crashed one, and all requests would fail [45].

The X-Node-ID Header

The fourth fix added an X-Node-ID HTTP response header to all S3 operations [46]. This header provides observability into routing decisions: clients can see which Kuri node handled their PUT, GET, or DELETE request. In a multi-node cluster, this information is crucial for debugging, monitoring, and verifying that the routing layer is correctly directing requests.

The implementation followed a clean three-layer pattern: adding NodeID() string to the Region interface, implementing it on the concrete Region struct (returning the nodeID field that had been wired through the configuration chain), and adding the header to five HTTP handlers in server/s3/handlers.go [46].

The Verification: Subagent Investigations and Docker Builds

After the bug fixes were committed, the assistant engaged in a series of verification steps that reveal the discipline of the development process.

First, a subagent was dispatched to investigate why integrations/kuri/ribsplugin/s3/region.go had not been included in the endpoint fixes commit [48]. The assistant had assumed the file was in a gitignored directory. The subagent's investigation revealed the opposite: the file was already committed, the working tree was clean, and there were no uncommitted changes. The assumption was wrong, and the subagent provided the correction [48].

Second, the Docker image was rebuilt to verify that all changes compiled and that the s3-proxy binary was correctly included in the final image [49]. The build output showed three COPY steps—kuri, gwcfg, and s3-proxy—confirming that the stateless frontend proxy was now part of the deployable artifact. The image SHA de30cb3a... became a historical marker: the first build of the corrected three-layer architecture [49].

The assistant then issued a git log --oneline -20 command to display the commit history [50]. The seven visible commits told the story of a system coming together: health checks for operational reliability, schema migrations for data integrity, monitoring for observability, configuration for deployability, and documentation for maintainability [50].

The Final Handoff: Ready for Testing

The chunk concludes with the assistant's summary of all 14 commits and the announcement that the Docker image was rebuilt and ready for testing [51]. The startup command was provided: cd test-cluster && ./start.sh /data/fgw2. The user responded with an empty message, which the assistant interpreted as a request for a comprehensive project summary [52].

This empty message is a fascinating artifact of human-AI collaboration. The user sent nothing—just empty XML tags—and the assistant produced a meticulously structured summary of the entire session's work. The silence was a signal: "I see your work, I accept it, and I trust you to know what comes next" [52].

Conclusion: The Architecture That Wouldn't Stay Dead

The work in this chunk represents a complete arc of distributed systems development: from a broken architecture with shared keyspaces and missing connections, through a fundamental redesign requiring dual database connections, through systematic decomposition into 14 logical commits, through targeted bug fixes addressing schema gaps and missing endpoints, to a verified Docker build ready for testing.

The key themes that emerge are:

Database keyspace segregation was the architectural insight that saved the system. Each Kuri node now has its own RIBS keyspace for private data while sharing an S3 metadata keyspace for object routing. This prevents race conditions and enables true horizontal scaling.

Dual CQL connections were the implementation mechanism that made keyspace segregation possible. The codebase was restructured to support two independent database connections per node, each targeting a different keyspace.

Git discipline transformed chaotic debugging into a coherent architectural narrative. The 14 commits, each representing a logical unit of change, serve as both a checkpoint and a documentation artifact.

Schema and endpoint fixes closed the gap between the architectural design and the running system. The S3Objects schema, MultipartUploads table, /healthz endpoint, and X-Node-ID header were the missing pieces that made the cluster functional.

The architecture that wouldn't stay dead—that kept demanding corrections, new connections, and schema fixes—finally stabilized into a coherent, committed, and built system ready for testing. The 14 commits in the git log are not just a record of changes; they are a map of the architectural decisions, corrections, and implementations that define the horizontally scalable S3 system.