Architecture by Delegation: How an AI Assistant Planned a Horizontally Scalable S3 Storage System

Introduction

In the development of complex distributed systems, the planning phase is often where the most critical architectural decisions are made—and where the most costly mistakes can be avoided. This article examines a single message from an AI-assisted coding session that represents a pivotal moment of architectural planning. In this message, an AI assistant (acting as a technical architect and developer) receives a set of ambitious requirements for horizontally scaling an S3-compatible storage system, and responds by systematically breaking down the problem, delegating codebase exploration tasks to specialized sub-agents, and synthesizing the findings into a comprehensive understanding of the existing system.

The message in question is a planning and exploration message. It does not yet implement any code. Instead, it represents the critical "think before you code" phase—the moment where the assistant internalizes the user's requirements, maps them against the existing codebase, and prepares the groundwork for a multi-phase implementation that will ultimately span dozens of files, hundreds of lines of code, and multiple architectural corrections. Understanding this message is essential to understanding how the entire subsequent implementation unfolded, including the architectural missteps that would later require correction.

The Subject Message: A Planning Manifesto

Let us examine the message in its entirety. The assistant begins with a structured reasoning section that breaks down the user's requirements into fourteen discrete points:

## Agent Reasoning The user wants me to plan a horizontally scalable architecture for their Filecoin gateway system. Let me break down the requirements:

1. Simple horizontal scaling - no replication, purely for performance 2. All nodes expose S3 endpoint and have ribsbs (RIBS blockstore) 3. At ribsbs level data is independent - each node stores different data 4. S3 integration uses common CQL database - tables track which object is on which node 5. S3 request LB nodes - stateless, just talking to CQL 6. New "S3 front nodes" - round-robin puts 7. Multiparts can be assembled from parts across multiple nodes - need detailed plan 8. Reads - S3 front proxy looks up in CQL the object info then makes directed request at the correct backing kuri S3 interface 9. Clear plan for multipart reads 10. Test suite plan 11. Kuri nodes should see minimal changes - mostly just YCQL schema that is object-placement aware 12. On PUT, the Kuri node should update the index (validate this assumption) 13. Read-after-write guarantee 14. Test plan for read-after-write

This breakdown is remarkable for its clarity and structure. The assistant has taken what was presumably a conversational request from the user and transformed it into a numbered list of engineering requirements. Each point captures a distinct architectural concern: scaling strategy (point 1), node responsibilities (points 2-3), data coordination (point 4), load balancing (points 5-6), multipart handling (points 7, 9), read routing (point 8), minimal invasiveness (point 11), index consistency (point 12), and testing strategy (points 10, 13-14).

The assistant then articulates its plan of action:

This is a complex architecture planning task. I should: 1. Create a comprehensive architecture document 2. Delegate to multiple agents to explore different aspects: - Current S3 implementation in Kuri - Current CQL/YCQL schema - How multipart uploads work currently - How the blockstore and indexing work

Let me start by delegating exploration tasks to understand the current codebase better.

This is the core of the message's strategy. Rather than diving directly into implementation—a common pitfall in software development—the assistant recognizes that it needs to understand the existing system before it can extend it. It delegates four parallel exploration tasks, each targeting a different subsystem of the codebase.

The Delegation Strategy: Four Exploration Tasks

The assistant proceeds to dispatch four exploration tasks, each with a specific focus and a detailed prompt. Let us examine each one.

Task 1: Exploring the S3 Implementation

The first task targets the existing S3 implementation within the Kuri integration:

[task] {"description":"Explore S3 implementation","prompt":"Explore the codebase to understand the current S3 implementation in the Kuri integration. Look for:\n1. How S3 requests are currently handled (PUT, GET, multipart uploads)\n2. The S3 API endpoint implementation\n3. How objects are currently stored..."

The sub-agent returns a comprehensive summary of the S3 implementation, describing a multi-layered architecture with clear separation between the HTTP API layer, business logic layer, and storage layer. It identifies key files and interfaces, including the core interface definitions in /home/theuser/gw/iface/s3.go, the Region and Bucket interfaces, and how requests flow through the system.

Task 2: Exploring the CQL/YCQL Schema

The second task investigates the database schema:

[task] {"description":"Explore CQL/YCQL schema","prompt":"Explore the codebase to understand the current CQL (Cassandra/YCQL) schema and usage. Look for:\n1. Current CQL table schemas in rbstor/index_cql.go or similar files\n2. How the CQL index tracks multihash to group mappings\n3. Any existing object metadata..."

The sub-agent discovers the existing MultihashToGroup table, which maps multihashes to storage groups, and identifies that there is no existing table for tracking which node holds which S3 object. This gap is critical—it means the scalable architecture will need to introduce a new S3Object table (or similar) to support the read-routing requirement.

Task 3: Exploring Multipart Upload Handling

The third task examines how multipart uploads currently work:

[task] {"description":"Explore multipart handling","prompt":"Explore the codebase to understand how multipart uploads currently work. Look for:\n1. Multipart upload initiation, parts upload, and completion\n2. How multipart parts are stored before assembly\n3. How the final object is assembled from parts..."

The sub-agent returns a detailed analysis of the multipart upload flow, from initiation through part upload to final assembly. This understanding is crucial because the user's requirements specify that multipart uploads should be assembled from parts distributed across multiple nodes—a significant extension of the current single-node multipart implementation.

Task 4: Exploring the Blockstore Interface

The fourth task examines the blockstore layer:

[task] {"description":"Explore blockstore interface","prompt":"Explore the blockstore interface and implementation to understand:\n1. The iface.RBS and iface.RIBS interfaces\n2. How the blockstore wrapper works (integrations/blockstore/ribsbs.go)\n3. How data is written to groups\n4. The relationship between..."

The sub-agent discovers the core RBS (Replicated Blockstore) and RIBS interfaces, the blockstore wrapper in ribsbs.go, and how data flows from S3 objects down to the blockstore storage layer. This understanding is essential for ensuring that the scalable architecture can work with the existing storage primitives without requiring a complete rewrite.

Why This Message Matters: The Architecture of Understanding

The subject message is not about writing code. It is about building understanding. The assistant's approach reveals several important assumptions and design decisions that will shape the entire implementation.

Assumption 1: The Existing System is a Foundation, Not an Obstacle

The assistant assumes that the existing Kuri codebase provides a solid foundation that can be extended rather than replaced. This is evident in the delegation strategy: each exploration task is framed as "understand the current implementation so we can extend it." The assistant is looking for extension points, not replacement points.

Assumption 2: Minimal Changes to Kuri Nodes

The user explicitly specified that "Kuri nodes should see minimal changes." The assistant internalizes this constraint and makes it a guiding principle. The exploration tasks are designed to understand what minimal changes would be required—primarily adding a NodeID field to the S3 object metadata and ensuring that PUT operations update a shared index.

Assumption 3: The CQL Database is the Coordination Point

The assistant assumes that the existing CQL (YCQL) database, which is already used for multihash-to-group mappings, can serve as the coordination point for the distributed S3 architecture. This is a reasonable assumption given that YCQL is a distributed database (YugabyteDB's Cassandra-compatible API) and can be shared across nodes.

Assumption 4: Round-Robin PUT Distribution is Sufficient

For write operations, the assistant plans to use round-robin distribution across backend nodes. This assumes that the workload is relatively uniform and that a simple round-robin strategy will provide adequate load distribution without requiring more sophisticated algorithms like consistent hashing or load-aware routing.

The Thinking Process: A Window into Architectural Reasoning

The agent reasoning section at the beginning of the message provides a rare window into the assistant's thinking process. We can see the assistant:

Parsing the user's requirements into discrete, actionable points
Categorizing the requirements by concern (scaling, data independence, coordination, routing, multipart handling, testing)
Identifying knowledge gaps (the assistant doesn't yet know how the S3 implementation works, what the CQL schema looks like, how multipart uploads are handled, or how the blockstore interface works)
Formulating a strategy to fill those knowledge gaps through parallel exploration
Prioritizing understanding over action (the assistant explicitly decides to explore before implementing) This thinking process is notable for its metacognitive awareness. The assistant recognizes its own lack of knowledge about the codebase and takes deliberate steps to remedy it. This is a sophisticated approach that mirrors how experienced human developers approach unfamiliar codebases.

Input Knowledge Required to Understand This Message

To fully understand this message, a reader would need:

Domain knowledge of S3-compatible storage systems: Understanding what S3 PUT, GET, and multipart upload operations do, and how object storage systems work at a high level.
Familiarity with distributed systems concepts: Horizontal scaling, stateless vs. stateful nodes, load balancing, read-after-write consistency, and the trade-offs between replication and parallelism.
Understanding of the Filecoin/Gateway context: The message references "Kuri" nodes, "RIBS blockstore," "CAR file staging," and "YCQL" without explanation. These are domain-specific terms from the Filecoin ecosystem.
Knowledge of the CQL/YCQL data model: The message assumes familiarity with Cassandra-compatible databases, table schemas, and how they handle distributed coordination.
Awareness of the existing codebase structure: The message references file paths like /home/theuser/gw/iface/s3.go and /home/theuser/gw/database/cqldb/migrations/ that would only be meaningful to someone familiar with this specific project.

Output Knowledge Created by This Message

The message creates several forms of output knowledge:

A structured requirements document: The fourteen-point breakdown transforms the user's conversational request into an engineering specification.
A codebase map: The exploration task summaries create a map of the existing codebase, identifying key files, interfaces, and data flows.
A gap analysis: By comparing the existing system against the requirements, the message implicitly identifies what needs to be built: a new S3 object tracking table, a stateless frontend proxy, node identification, and multipart coordination logic.
An implementation strategy: The decision to delegate exploration tasks and then create a comprehensive architecture document establishes a workflow for the implementation phase.
Risk identification: The message identifies potential risks, such as the need for read-after-write guarantees and the complexity of cross-node multipart assembly.

Mistakes and Incorrect Assumptions

While the message is primarily a planning artifact, several assumptions embedded within it would later prove to be incorrect or incomplete:

Mistake 1: Underestimating the Architectural Separation

The assistant's initial planning assumes that the S3 frontend proxy and Kuri storage nodes can be configured with shared settings. This assumption would later be corrected by the user, who pointed out that the roadmap clearly shows S3 frontend proxies are a separate stateless node type with their own configuration, and that each Kuri node needs its own independent external HTTP endpoint for CAR file staging. This correction would lead to a complete redesign of the test cluster infrastructure in a later chunk.

Mistake 2: Assuming Direct S3 Access to Kuri Nodes

The assistant initially plans to have Kuri nodes expose S3 APIs directly, with the frontend proxy simply routing requests. The user would later correct this, explaining that the architecture requires Kuri nodes to be internal storage nodes that do not expose S3 endpoints directly—all S3 traffic must go through the stateless frontend proxies.

Mistake 3: Overlooking CAR Staging Configuration

The exploration tasks do not specifically investigate CAR file staging configuration, which would later cause a startup failure when Kuri nodes fail with "no external module configured." This oversight highlights the challenge of comprehensive codebase exploration—it is difficult to anticipate every dependency.

The Deeper Significance: Planning as a Collaborative Act

This message is significant not just for what it contains, but for what it represents: a collaborative planning process between a human user and an AI assistant. The assistant does not simply implement the user's requirements verbatim. Instead, it:

Interprets and structures the requirements
Identifies knowledge gaps and fills them through exploration
Formulates a strategy that balances multiple concerns
Creates artifacts (the architecture document, the exploration summaries) that serve as shared understanding This is a fundamentally different model from the traditional "user specifies, AI implements" paradigm. Here, the AI acts as an architectural partner, engaging in the messy, iterative process of understanding a complex system before modifying it.

Conclusion

The subject message is a planning message in the truest sense. It does not implement a single line of production code, yet it shapes everything that follows. The fourteen-point requirements breakdown, the four parallel exploration tasks, and the synthesized understanding of the existing codebase provide the foundation for a multi-phase implementation that will ultimately transform a single-node S3 storage system into a horizontally scalable distributed architecture.

The message also reveals the assistant's architectural philosophy: understand before you build, delegate exploration to fill knowledge gaps, and create shared understanding through structured artifacts. These principles, while seemingly obvious, are often violated in practice—both by human developers and by AI systems. The assistant's disciplined approach to planning is a model worth studying.

In the subsequent chunks of this session, we will see how this planning translates into implementation, how some assumptions prove incorrect and require correction, and how the architecture evolves through iterative refinement. But it all starts here, with a message that chooses understanding over action—and in doing so, sets the stage for a successful implementation.