From Library Code to Running Cluster: The Architecture of Incremental Integration in a Distributed S3 System

Introduction

In the span of a few dozen messages in a sprawling coding session, an AI assistant and its human collaborator transformed a collection of disconnected Go packages into a running two-node distributed S3 cluster. The work described in this chunk of the conversation represents the critical transition from "the code compiles" to "the system works"—a transition that is often the most challenging phase in any software engineering effort. This article synthesizes the key themes, decisions, and debugging moments that defined this phase of the work, tracing the arc from building a binary to discovering a fundamental architectural flaw to staging 14 logical commits that completed the three-layer horizontally scalable S3 architecture.

The chunk covers several interlocking threads: the creation and debugging of the S3 frontend proxy binary, the integration of that binary into the test cluster's Docker Compose infrastructure, the discovery that all Kuri nodes were sharing the same database keyspace (causing deadlocks and race conditions), the decision to segregate keyspaces per node while sharing only the S3 metadata keyspace, the implementation of dual CQL database connections, and the final staging of all changes into a coherent set of git commits. Each of these threads represents a distinct engineering challenge, and together they tell a story about what it really means to build a distributed system—not just designing the architecture, but making it actually work.

The Missing Entry Point: From Library Code to Runnable Binary

One of the most striking patterns in this chunk is the repeated discovery that components which were "implemented" in source code were not actually deployable. The S3 frontend proxy package (server/s3frontend/) contained seven well-structured Go files—a server, a router with YCQL integration, a backend pool for connection management, multipart upload coordination, and dependency injection wiring. But as the assistant discovered through a series of file reads and glob searches, there was no main.go entry point to build it as a standalone binary [5][6][7].

This gap between "code exists" and "system works" is a recurring theme in the chunk. The docker-compose.yml for the test cluster contained a placeholder command that simply echoed a message and slept forever: command: ["sh", "-c", "echo 'S3 Frontend Proxy - requires s3frontend binary' && sleep infinity"] [21]. The entire three-layer architecture—stateless S3 frontend proxies routing requests to independent Kuri storage nodes backed by a shared YugabyteDB—was blocked by the absence of a single file.

The assistant's response to this gap reveals a methodical, pattern-driven approach to software development. Rather than guessing at the structure of the entry point, the assistant studied the existing project conventions. It read the Makefile to understand how binaries were built [9][10]. It examined the Kuri node's main.go at integrations/kuri/cmd/kuri/main.go to understand the project's entry point conventions [10]. It searched for all existing main.go files in the project to confirm the pattern [9]. It read the configuration package to understand the S3APIConfig struct and its environment variable bindings [13]. Each read operation answered a specific question needed to make the upcoming implementation decision.

When the assistant finally wrote the main.go file (message 364), the Language Server Protocol immediately flagged multiple compilation errors: unused imports, a mismatched function call signature for s3.NewAuthenticator (which returns two values, not one), a pointer type mismatch, and a reference to a non-existent GetNodeIDs() method [15]. These errors are not failures—they are the natural friction of integrating separate components. The assistant's debugging cycle—read the actual API signatures, fix the code, rebuild—is the engine that drives the entire session [16][17][18]. The build eventually succeeds (message 369), and the assistant immediately pivots to integrate the binary into the Docker Compose infrastructure [20][21].

The Deployment Pipeline: Thinking in Systems

What distinguishes this chunk from a simple "write code, compile, done" narrative is the assistant's systematic attention to the deployment pipeline. After the build succeeds, the assistant does not declare victory. Instead, it traces the chain from source code to running container: source → build (Makefile) → binary → Docker image (Dockerfile) → container (docker-compose.yml) [22][23][24].

The Dockerfile check (message 371) is a revealing moment. The assistant has just updated docker-compose.yml to run s3-proxy as the command for the proxy service. But the binary was built on the host machine, and the Docker Compose test cluster runs inside containers. If the Docker image does not contain the s3-proxy binary, the container will fail to start with a "command not found" error. The assistant's decision to check the Dockerfile before restarting the cluster is a classic example of proactive debugging: finding and fixing problems before they manifest as runtime errors [22].

This systematic thinking extends to the todo list management that runs throughout the chunk. The assistant maintains a persistent todo list with four items: create the main.go entry point, update the Makefile, update docker-compose.yml, and test the full cluster flow [19][21]. Each status transition—from "pending" to "in_progress" to "completed"—provides a clear, visible chain of dependencies. The todo list is not just a log; it is a cognitive artifact that helps the assistant manage its own working memory across a session spanning hundreds of messages [19].

The Keyspace Revelation: A Fundamental Architectural Correction

Just as the assistant was completing the proxy binary integration, a deeper architectural issue emerged. When the test cluster was started with both Kuri nodes, the second node failed to start due to YugabyteDB migration deadlocks and a configuration validation error around RetrievableRepairThreshold [60][61][62]. The assistant initially attributed these failures to startup ordering and configuration issues, implementing fixes like sequential startup and log level corrections.

But the root cause was more fundamental. Through a series of diagnostic investigations, the assistant discovered that all Kuri nodes were sharing the same database keyspace [76][77][78]. This meant that both nodes were writing to the same groups table, the same deals table, and the same blockstore—causing deadlocks, race conditions, and data corruption. The architecture had been designed with the assumption that nodes would be isolated, but the implementation had not enforced that isolation.

The user's intervention was critical at this point. The user clarified that groups are per-node resources—each Kuri node must have its own database keyspace for RIBS data (deals, groups, blockstore), while only the S3 metadata keyspace should be shared between nodes for object routing [79][80][81]. This distinction between per-node keyspaces and a shared keyspace is the architectural key that unlocks horizontal scalability.

The conversation that follows this revelation is a masterclass in architectural decision-making under pressure. The assistant initially proposes adding a node_id column to every RIBS database table and filtering all queries by node ID [88][89][90][91]. This approach would keep a single keyspace but partition data at the row level. The user pushes back, clarifying that the correct approach is keyspace segregation—each node gets its own filecoingw_{node_id} keyspace for RIBS data, while the filecoingw_s3 keyspace is shared [92][93][94].

The assistant implements this approach, but then the user corrects again: the RIBS layer already has a node_id column in the groups table, and the correct approach is to use node-ID filtering within a single keyspace, not separate keyspaces per node [95][96][97]. The assistant reverts the keyspace segregation and implements node-ID filtering instead [104][105][106][107].

Then the user clarifies again: the architecture actually requires both—separate keyspaces per node for RIBS data AND a shared keyspace for S3 metadata, with dual CQL connections to support both [170][171][172]. This is the final, correct design: each Kuri node connects to its own filecoingw_{node_id} keyspace for deals, groups, and blockstore data, while all nodes share the filecoingw_s3 keyspace for S3 object routing metadata. The assistant implements this by adding a second CQL connection configuration, creating an S3CqlDB wrapper type, and wiring dual CQL connections through the dependency injection system.

Dual CQL Connections: The Implementation That Made It Work

The implementation of dual CQL connections is the most architecturally significant code change in this chunk. The assistant needed to modify the database layer to support two separate connections: one for the per-node RIBS keyspace and one for the shared S3 metadata keyspace. This required changes to the configuration system (adding a second set of YCQL connection parameters), the database abstraction layer (creating the S3CqlDB wrapper type), and the dependency injection wiring (passing both connections to the components that need them) [170][171][172].

The verification of this change (message 522) shows the assistant tracing through the codebase with a grep to confirm that every database call goes to the correct keyspace. This kind of systematic verification—checking not just that the code compiles, but that the architectural intent is correctly implemented across all layers—is the hallmark of careful engineering [172].

The 14 Logical Commits: Staging for Coherence

One of the most impressive aspects of this chunk is the assistant's discipline in staging all changes into 14 logical git commits. The commit list covers: configuration changes, interface definitions, the Kuri S3 plugin, dual CQL connections, the S3 frontend proxy package, build system updates, test cluster infrastructure, documentation, CQL schema migrations, and endpoint fixes [analyzer_summary].

This is not just housekeeping. The act of organizing changes into logical commits forces the assistant to think about the system's architecture at a higher level. Each commit represents a coherent unit of work that can be reviewed, tested, and potentially reverted independently. The commit structure itself becomes a form of documentation—a map of how the system is organized and what each component depends on.

The Fixes That Made It Work

Beyond the architectural corrections, the chunk includes several critical bug fixes that were necessary to make the system actually work:

S3Objects table schema update: The table was updated to include node_id and expires_at columns, enabling the proxy to route GET requests to the correct node and enforce object expiration [analyzer_summary].
MultipartUploads table: A new table was created for cross-node coordination of multipart uploads, which is essential for the S3 API's multipart upload feature [analyzer_summary].
/healthz endpoint: A health check endpoint was added to the S3 proxy, allowing the test cluster to verify that the proxy is running and ready to accept requests [analyzer_summary].
X-Node-ID response headers: The proxy was modified to include the node ID in response headers, providing visibility into which backend node handled each request [analyzer_summary].
Sequential startup: The test cluster startup script was modified to start Kuri nodes sequentially rather than simultaneously, avoiding migration deadlocks in YugabyteDB [63][64].

Conclusion

This chunk of the coding session is a microcosm of what it means to build a distributed system. It is not just about writing code that compiles—it is about tracing the entire pipeline from source to running service, discovering architectural assumptions that turn out to be wrong, correcting those assumptions through iterative clarification with a domain expert, and staging the final result into a coherent set of changes that can be reviewed and deployed.

The arc of the chunk—from building a binary, to discovering a keyspace crisis, to implementing dual database connections, to staging 14 logical commits—reveals the true nature of systems engineering. It is iterative, it is collaborative, and it requires constant attention to the gap between what the architecture says and what the code actually does. The assistant's methodical approach—read before writing, verify before deploying, organize before committing—is a model for how complex software should be built, whether by human or machine.