Blueprint for Distributed S3: Deconstructing a Horizontal Scaling Architecture Plan

Introduction: A Pivot Point in Distributed Systems Design

In any complex software engineering project, there comes a moment when abstract requirements crystallize into a concrete architectural vision. Message 38 of this OpenCode coding session represents precisely such a moment: it is the delivery of a comprehensive, multi-layered architecture plan for transforming a single-node S3-compatible storage system into a horizontally scalable distributed storage cluster. This message, written by an AI coding assistant in response to a user's detailed request, is not merely a document—it is the architectural blueprint that will guide weeks of implementation, the conceptual foundation upon which dozens of files, hundreds of tests, and a complete infrastructure overhaul will be built.

The message arrives at a critical juncture. The conversation preceding it reveals an intensive exploration phase: the assistant has dispatched multiple exploration agents to dissect the existing codebase, examining the S3 implementation, the CQL/YCQL schema, the multipart upload handling, and the blockstore interface. It has read interface definitions, traced data flows through the storage stack, and analyzed how the current system handles authentication, routing, and object indexing. Message 38 is the synthesis of all that research—the moment when understanding transforms into design.

What makes this message particularly fascinating is what it reveals about the architectural thinking process itself. The plan is ambitious, well-structured, and technically sophisticated. Yet, as later segments of the conversation will reveal, it contains a fundamental architectural misconception that will require significant correction. This tension between the plan's apparent completeness and its hidden flaw makes the message a rich subject for analysis—a case study in how even thorough architectural planning can harbor assumptions that only become visible when tested against reality.

The Genesis: Why This Message Was Written

To understand why message 38 exists, we must trace back to the user's original requirements. The user wanted a "simple horizontal scaling" architecture for their Filecoin Gateway's S3 storage system, with a specific set of constraints that shaped every subsequent decision. The requirements were explicit and demanding:

No replication for performance: The user explicitly stated that data should not be replicated across nodes. Each node stores different data. This is a performance-over-redundancy design philosophy, justified by the fact that Filecoin itself provides the redundancy layer. The storage system's job is throughput, not durability.

All nodes expose S3 endpoints with independent blockstores: Every node in the cluster must be a fully functional S3 endpoint backed by its own RIBS blockstore. The data at the blockstore level is completely independent—no two nodes store the same objects.

A shared CQL database for object placement tracking: While the blockstores are independent, a shared YCQL (Yugabyte CQL) database must track which object resides on which node. This is the coordination layer that enables the system to function as a coherent cluster rather than a collection of isolated servers.

Stateless S3 frontend proxy nodes: The user envisioned a new tier of nodes—stateless load balancers that handle S3 protocol, authenticate requests, and route them to the appropriate backend storage node. These frontend nodes are the entry point for all client traffic.

Round-robin writes, directed reads: PUT requests should be distributed across storage nodes using simple round-robin load balancing. GET requests, however, require a lookup: the frontend must query the shared database to determine which specific node holds the requested object, then proxy the request to that node.

Multipart upload assembly across nodes: This is perhaps the most architecturally challenging requirement. Multipart uploads—where a large file is split into parts that can be uploaded independently and then assembled—must work across multiple storage nodes. Parts of a single upload could reside on different nodes, and the final assembly must fetch and combine them.

Minimal changes to existing Kuri nodes: The Kuri storage nodes, which form the backbone of the system, should require minimal modification. The user wanted the complexity concentrated in the new frontend layer, not spread across the existing infrastructure.

Read-after-write consistency: A GET immediately following a PUT must return the newly written data. This is a non-trivial requirement in a distributed system where writes and reads may traverse different paths.

These requirements collectively describe a system that is architecturally elegant but operationally complex. The assistant's task was to translate these high-level constraints into a concrete, implementable design. Message 38 is that translation.## The Context: What Came Before

Message 38 did not emerge from a vacuum. The conversation leading up to it reveals a methodical, multi-phase investigation that mirrors how a senior engineer would approach an unfamiliar codebase. The assistant began by dispatching exploration agents—automated sub-tasks designed to read and summarize specific areas of the code. Four exploration tasks were executed in parallel:

The first agent explored the S3 implementation, tracing how requests flow from the HTTP API layer through authentication, routing, and into the storage backend. It identified the key interfaces (Region, Bucket, S3Object), the authentication flow (AWS SigV4), and the existing multipart upload handling.

The second agent examined the CQL/YCQL schema, discovering the existing MultihashToGroup table that maps content multihashes to storage group IDs. This table is the backbone of the current indexing system, but it lacks any notion of node-level placement—it maps content to logical groups, not to physical storage nodes.

The third agent analyzed the multipart upload implementation in detail, understanding how parts are stored, tracked, and assembled into final UnixFS DAGs. This was critical because the new architecture would need to extend this mechanism to work across multiple nodes.

The fourth agent explored the blockstore interface and the RIBS blockstore wrapper, understanding how data is written to groups and how the blockstore interacts with the S3 layer.

The result of this exploration was a deep understanding of the existing architecture. The assistant now knew that the current system was monolithic: a single Kuri node handled all S3 requests, stored all data in its local RIBS blockstore, and used YCQL only for the multihash-to-group mapping. There was no concept of node identity, no object placement tracking, and no load balancing layer. Every requirement the user had specified would need to be built from scratch.

The assistant then attempted to write the architecture plan to a file (/home/theuser/gw/doc/architecture-horizontal-scale.md), but was blocked by a tool restriction—the OpenCode environment prevented writing new files outside specific directories. This forced the assistant to ask the user how they wanted the plan delivered. The user chose "Show in chat," which is why message 38 appears as a long-form chat message rather than a file creation.

This delivery constraint shaped the message's format. Rather than a formal document with sections, appendices, and cross-references, the plan is a chat-optimized summary—comprehensive but condensed, structured with headings and code blocks but designed for immediate reading rather than offline reference. The assistant explicitly notes at the end that a "Full detailed document available upon request," acknowledging that this chat message is a summary of a more detailed plan that could be expanded.

The Plan Itself: A Detailed Walkthrough

The architecture plan presented in message 38 is organized into eight sections, each addressing a critical aspect of the distributed system. Let us walk through each section to understand the design decisions and their implications.

Section 1: Architecture Overview

The plan opens with a visual ASCII diagram showing three layers stacked vertically. At the top are the S3 Frontend Proxy nodes—stateless, horizontally scalable, handling authentication and routing. In the middle are the Kuri Storage Nodes—each with independent RIBS blockstore data, no replication, each node storing different objects. At the bottom is the Shared YCQL Database (Yugabyte), containing five tables: S3ObjectsV2, MultipartUploads, MultipartParts, MultihashToGroup, and CidGroups.

This three-layer architecture is the core design pattern. It separates concerns cleanly: the frontend layer handles protocol and routing, the storage layer handles data persistence, and the database layer handles coordination. Each layer can be scaled independently—add more frontend proxies to handle more concurrent requests, add more Kuri nodes to increase storage capacity and throughput, scale the YCQL database vertically or horizontally as the index grows.

The diagram also reveals a key design choice: the frontend proxies communicate with Kuri nodes, and Kuri nodes communicate with YCQL, but frontend proxies also communicate directly with YCQL for read routing. This creates a hybrid topology where the database serves both as a coordination layer for writes (updated by Kuri nodes) and a lookup layer for reads (queried by frontend proxies).

Section 2: Component Details

This section provides concrete implementation guidance. For the S3 Frontend Proxy, the plan specifies a S3FrontendServer struct with four fields: an Authenticator for SigV4 validation, a BackendPool managing connections to Kuri nodes, a gocql.Session for YCQL queries, and an atomic counter for round-robin write distribution. The plan also lists four new files that will need to be created: server.go, router.go, backend_pool.go, and multipart_coord.go.

For the Kuri Storage Nodes, the plan emphasizes minimal changes: add a NODE_ID for identification, extend the S3ObjectV2 index to include node_id, and add a lightweight internal API for fetching parts from other nodes. The modified index struct shows a new NodeID field alongside the existing Bucket, Key, Cid, Size, and Updated fields.

The YCQL Schema Extensions section defines three new tables in SQL. The S3ObjectsV2 table is the core placement index, mapping (bucket, key) to node_id—this is what enables the frontend to locate objects. The MultipartUploads table tracks ongoing multipart uploads, recording which node is the coordinator. The MultipartParts table records which node stores each part of a multipart upload, enabling cross-node assembly.

Section 3: Data Flows

This section is arguably the most important, as it translates the static architecture into dynamic behavior. Three data flows are described:

Single PUT (Round-Robin Write): A client sends a PUT request to the frontend proxy, which selects a Kuri node using round-robin and forwards the request. The Kuri node stores the object locally in its RIBS blockstore, then updates the S3ObjectsV2 table in YCQL with the object's bucket, key, and its own node ID. The plan asserts that "YCQL QUORUM consistency ensures immediate visibility" for read-after-write.

GET (Directed Read): A client sends a GET request to the frontend proxy, which first validates authentication, then queries YCQL to find which node holds the object (SELECT node_id FROM S3ObjectsV2), and finally proxies the request to that specific node. This is the "directed read" pattern—the frontend doesn't guess or broadcast; it consults the authoritative index.

Multipart Upload (Cross-Node Assembly): This is the most complex flow, described in three phases. In Phase 1 (Initiate), the frontend selects a coordinator node (Kuri #1) and records the upload in the MultipartUploads table. In Phase 2 (Upload Parts), each part is distributed to potentially different nodes via round-robin, and each node records its part in the MultipartParts table. In Phase 3 (Complete), the coordinator queries MultipartParts to find all part locations, fetches parts from local storage (for parts it holds) or from other nodes via an internal API, assembles the final UnixFS DAG, stores it locally, and updates S3ObjectsV2.

The inter-node API is defined as a simple Go interface: FetchPart(ctx context.Context, cid string) (io.ReadCloser, error). This is a minimal, focused contract—just enough to enable cross-node part retrieval without exposing the full storage internals.

Section 4: Read-After-Write Consistency

The plan acknowledges the challenge of ensuring that a GET immediately after a PUT returns the new data. The solution combines three mechanisms: YCQL QUORUM consistency for all writes and reads, no frontend caching for new objects during the first 30 seconds, and blockstore flushing before returning PUT success. An optional verification function is sketched that polls YCQL with QUORUM until the object appears.

Sections 5-8: Testing, Implementation Phases, Design Decisions, and Success Metrics

The remaining sections provide practical guidance for execution. The test suite plan covers unit tests (router, multipart), integration tests (read-after-write, multipart with node failures), load tests (horizontal scaling, concurrent reads), and failure scenario tests (node failure, YCQL unavailability). The implementation phases span 10 weeks across five phases. The design decisions table captures four key choices with rationales. The success metrics define quantitative targets for throughput, latency, assembly time, and scalability.## The Hidden Assumptions: What the Plan Takes for Granted

Every architecture plan rests on assumptions, and message 38 is no exception. Some of these assumptions are explicitly stated; others are embedded in the design choices. Examining them reveals both the plan's strengths and its potential blind spots.

Assumption 1: YCQL QUORUM consistency is sufficient for read-after-write. The plan states this as a given, but in practice, distributed consistency is more nuanced. QUORUM reads and writes ensure that a majority of replicas agree, but they don't guarantee that a read following a write will see the write if the read reaches a different set of replicas than the write. YugabyteDB's implementation of QUORUM may handle this correctly, but the assumption deserves testing—which is why the plan includes a waitForIndexVisible polling function as a safety net.

Assumption 2: Round-robin distribution provides even load. Round-robin is simple and stateless, but it assumes that all nodes have equal capacity and that all requests have equal cost. In practice, objects vary in size, and nodes may have different resource utilization. A round-robin scheme can lead to hotspots if some objects are much larger than others or if some nodes are slower due to background tasks.

Assumption 3: The coordinator node can fetch parts from other nodes without significant overhead. The multipart assembly design relies on the coordinator fetching parts from remote nodes over the network. If parts are large or the network is slow, this could become a bottleneck. The plan doesn't address whether parts are streamed or buffered, whether there are timeouts, or how the coordinator handles a node that becomes unavailable during assembly.

Assumption 4: The existing Kuri node codebase requires only minimal changes. The plan asserts that Kuri nodes need only a NODE_ID, an extended index, and an internal API. But modifying the index to include node_id touches the CQL schema, the Go struct definitions, the write path (to populate the new field), and the read path (to use it for lookups). The internal API requires adding an HTTP endpoint, authentication for inter-node requests, and error handling. These changes, while conceptually minimal, ripple through multiple layers of the codebase.

Assumption 5: The frontend proxy can be implemented as a new standalone package. The plan proposes creating server/s3frontend/ as a new package with four files. But the frontend proxy must replicate significant portions of the existing S3 server—authentication, request parsing, response formatting, error handling. The plan doesn't address how much code will be shared versus duplicated between the frontend proxy and the existing Kuri S3 server.

Assumption 6: The system can scale to 16 nodes and 1 billion objects. The success metrics include ambitious targets, but the plan doesn't address how the YCQL database will perform under that scale, how the frontend proxies will discover new nodes, how node failures will be detected and handled, or how the system will recover from a total YCQL outage.

These assumptions are not necessarily wrong—many are reasonable starting points for a first implementation. But they represent risks that would need to be validated through testing, which is why the test suite plan includes failure scenarios and load tests.

The Mistake That Wasn't Visible Yet

One of the most instructive aspects of message 38 is what it gets wrong—not in its technical details, but in its architectural conception. The plan describes a system where Kuri storage nodes expose their own S3 APIs. The ASCII diagram shows "S3 API" listed under each Kuri node. The data flow diagrams show the frontend proxy forwarding requests to Kuri nodes' S3 interfaces. The entire design assumes that Kuri nodes are S3 endpoints that the frontend proxies talk to.

This is incorrect. As the user will later point out (in subsequent segments of the conversation), the architecture roadmap explicitly specifies that S3 frontend proxies are a separate stateless node type that routes requests to Kuri storage nodes. The Kuri nodes are not supposed to expose S3 APIs directly to the frontend proxies—they are storage backends that communicate through a different mechanism. The frontend proxies are the only S3-facing layer.

The plan's mistake is subtle but significant. By treating Kuri nodes as S3 endpoints, the design creates a redundant S3 layer—requests would go through S3 frontend proxy → Kuri S3 API → Kuri storage, adding unnecessary overhead and complexity. The correct architecture, as the roadmap specifies, has the frontend proxy as the sole S3 entry point, with Kuri nodes operating as pure storage backends behind an internal protocol.

This error is particularly interesting because it stems from a reasonable interpretation of the user's requirements. The user said "All nodes expose S3 endpoint and have ribsbs," which could be read as "each Kuri node should have its own S3 endpoint." But the user's intent was that the system as a whole exposes an S3 endpoint, with the frontend proxies handling the protocol and the Kuri nodes handling the storage. The assistant's interpretation added an unnecessary layer.

The correction of this error becomes a major theme in later segments, where the assistant redesigns the test cluster to use a proper three-layer hierarchy (S3 proxy on port 8078 → Kuri nodes internally → YugabyteDB) and generates per-node independent configuration files. The fact that the plan in message 38 contains this error, despite being thorough and well-researched, is a powerful reminder that even the most careful architectural analysis can miss the forest for the trees.

Input Knowledge: What You Need to Understand This Message

To fully grasp message 38, a reader needs familiarity with several domains:

S3 API semantics: Understanding PUT, GET, multipart upload initiation, part upload, and completion is essential. The plan assumes the reader knows how S3 multipart uploads work—the UploadId, part numbers, ETags, and the assembly process.

YCQL/CQL concepts: The plan references QUORUM consistency, primary keys, indexes, and CQL data types. Knowledge of Cassandra/YugabyteDB's distributed database model is helpful for understanding why QUORUM is chosen and what it guarantees.

Go programming: The code snippets use Go structs, interfaces, and standard library types. The plan assumes the reader can interpret atomic.Uint64, gocql.Session, and io.ReadCloser.

UnixFS and IPFS concepts: The multipart assembly section mentions "UnixFS DAG" and "links to all parts." Understanding that IPFS represents files as directed acyclic graphs of content-addressed blocks is necessary to grasp why assembly involves building a DAG node rather than concatenating byte streams.

Docker and container orchestration: While not explicit in this message, the plan's references to "Docker Compose setup" and "deployment" assume familiarity with containerized infrastructure.

Distributed systems concepts: The plan touches on load balancing, consistency models, failure handling, and horizontal scaling. A reader unfamiliar with these concepts might miss the trade-offs embedded in the design decisions.

Output Knowledge: What This Message Creates

Message 38 creates several forms of knowledge that propagate through the rest of the project:

A shared mental model: The architecture diagram and data flow descriptions give everyone involved in the project a common understanding of how the system should work. This is perhaps the most important output—alignment on architecture before implementation begins.

Implementation guidance: The component details, file lists, and code snippets provide concrete starting points for implementation. A developer reading this message knows exactly which files to create, what structs to define, and what interfaces to implement.

Schema definitions: The CQL table definitions are directly translatable to migration files. The S3ObjectsV2, MultipartUploads, and MultipartParts tables will be created from these specifications.

Test scenarios: The test suite plan identifies specific scenarios that must be validated, from basic round-robin routing to complex failure cases like node unavailability during multipart assembly.

Success criteria: The metrics define what "done" means—linear throughput scaling, sub-100ms P99 latency, 30-second multipart assembly, 100% read-after-write success. These become the acceptance criteria for the implementation.

A baseline for iteration: Perhaps most importantly, the plan creates a baseline that can be critiqued, refined, and corrected. The architectural error discovered later would not have been found without this concrete plan to evaluate. The plan's very existence enables its own improvement.

The Thinking Process: Reasoning in the Agent's Mind

The agent reasoning section at the top of message 38 provides a window into the assistant's cognitive process. The assistant lists all 15 user requirements, then states its intention to "provide a comprehensive summary in the chat." This reveals a methodical, checklist-driven approach to architecture design.

The reasoning shows the assistant synthesizing the exploration results into a coherent design. It has internalized the constraints: no replication, stateless frontends, independent blockstores, minimal Kuri changes. It has identified the key architectural innovations needed: a new frontend proxy layer, YCQL schema extensions for object placement, a round-robin write distribution mechanism, a directed read lookup path, and a cross-node multipart assembly protocol.

The assistant's thinking is fundamentally about translation—converting user requirements into technical specifications. Each requirement maps to one or more design elements:

Conclusion: The Plan as Artifact and Process

Message 38 is more than a document—it is an artifact of a particular moment in the software development lifecycle. It represents the transition from exploration to design, from understanding to planning. It captures the assistant's best understanding of the system at a point in time, complete with the assumptions, decisions, and yes, the mistakes that characterize any first draft of a complex architecture.

The message's greatest strength is its comprehensiveness. In a single chat message, it covers architecture, component design, data flows, consistency, testing, implementation planning, and success metrics. It provides enough detail for a developer to begin implementation while leaving room for refinement. The code snippets, while brief, are concrete enough to guide file creation.

The message's greatest weakness is the architectural error that will later require correction. But this weakness is also a strength—it demonstrates that architecture is an iterative process, that plans must be validated against reality, and that the best response to discovering an error is to redesign and improve.

For the reader studying this conversation, message 38 offers a rare window into the architectural thinking of an AI system. It shows how research, synthesis, and design combine to produce a coherent plan. It reveals the assumptions that underpin architectural decisions. And it demonstrates that even the most thorough planning can harbor misconceptions that only become visible when the plan meets the test of implementation.

The plan in message 38 would go on to guide the implementation of a distributed S3 storage system. It would be tested, debugged, and corrected. The architectural error would be fixed, the test cluster would be rebuilt, and the system would eventually work as intended. But the plan itself—in its original, imperfect form—remains a valuable artifact, a snapshot of the design process at the moment of creation.