The Architecture Epiphany: How a Single Correction Reshaped a Distributed S3 Test Cluster

In the course of building complex distributed systems, few moments are as valuable as the one where a fundamental architectural misunderstanding is laid bare. This article examines precisely such a moment—a single message from an AI assistant during a coding session for a horizontally scalable S3-compatible storage system, where the assistant recognized and began correcting a deep architectural error that had permeated its test cluster setup.

The Subject Message

The message that serves as our subject is concise but carries enormous weight:

You're absolutely right! I completely misunderstood the architecture. Looking back at the roadmap:

1. S3 Frontend Proxies are a SEPARATE node type (stateless, no storage) 2. Kuri nodes are storage nodes that expose S3 API internally 3. The architecture is: Client → Frontend Proxy → Kuri Node (with S3 API)

The current test cluster setup has: - Kuri nodes with S3 API exposed directly (wrong) - No frontend proxy layer (missing)

I need to fix the test cluster to match the actual architecture from the roadmap. The frontend proxy should be a separate service that routes to Kuri nodes.

Let me update the docker-compose.yml and scripts to match the correct architecture. [write] /home/theuser/gw/test-cluster/docker-compose.yml Wrote file successfully.

This message, though brief, represents a critical turning point. It is the moment of recognition, the pivot from a flawed implementation toward the correct one. To understand its significance, we must examine the context that led to it, the assumptions that were broken, and the cascade of changes it triggered.

The Context: Building a Test Cluster

Prior to this message, the assistant had been working diligently on a test cluster infrastructure for a distributed S3 storage system called the Filecoin Gateway. The system's architecture, as defined in a comprehensive roadmap document (scalable-roadmap.md), called for a clean three-layer design: stateless S3 frontend proxies at the top, independent Kuri storage nodes in the middle, and a shared YugabyteDB database at the bottom providing YCQL-based coordination.

The assistant had already implemented significant portions of this architecture in code. It had created the server/s3frontend/ package with round-robin request distribution, backend health checking, and YCQL-based read routing. It had modified the Kuri node codebase to support node identification via the FGW_NODE_ID environment variable and updated the S3Object interface and CQL object index to track which node stores each object. The roadmap was clear: frontend proxies are stateless and handle routing; Kuri nodes are storage backends that expose an S3 API internally for the proxies to consume.

Yet when it came time to build the test cluster—the Docker Compose environment meant to validate this architecture—the assistant fell into a subtle but critical trap. It configured the Kuri nodes to expose their S3 APIs directly to the outside world, bypassing the frontend proxy layer entirely. The test cluster had Kuri nodes on public ports, sharing a single configuration, with no frontend proxy in sight.## The User's Intervention

The correction came from the user, who had been testing the cluster and noticed the discrepancy. In the message immediately preceding our subject (index 345), the user called out: "Wait the S3 was supposed to be a separate node type and also all kuri nodes were meant to have S3 with subsets of objects managed by them," and then explicitly referenced the roadmap document by calling @scalable-roadmap.md. This was not a vague complaint—it was a precise, document-backed challenge to the assistant's implementation.

The user then read the full roadmap file, which laid out in ASCII art the exact architecture:

┌─────────────────────────────────────────────────────────────────────────────┐
│                      S3 FRONTEND PROXY LAYER (Stateless)                     │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                          │
│  │  Proxy #1   │  │  Proxy #2   │  │  Proxy #N   │        Round-robin writes │
│  │  • Auth     │  │  • Auth     │  │  • Auth     │        YCQL lookup reads  │
│  │  • Routing  │  │  • Routing  │  │  • Routing  │                          │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘                          │
└─────────┼────────────────┼────────────────┼──────────────────────────────────┘
          │                │                │
          └────────────────┴────────────────┘
                              │
┌─────────────────────────────┼─────────────────────────────────────────────────┐
│                      KURI STORAGE NODE LAYER                                   │
│  ┌─────────────┐  ┌────────┴────┐  ┌─────────────┐  Independent data         │
│  │  Kuri #1    │  │  Kuri #2    │  │  Kuri #N    │  No replication           │
│  │  • RIBS BS  │  │  • RIBS BS  │  │  • RIBS BS  │  Each node different objs │
│  │  • S3 API   │  │  • S3 API   │  │  • S3 API   │                          │
│  └─────────────┘  └─────────────┘  └─────────────┘                          │
└─────────────────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────┼─────────────────────────────────────────────────┐
│                      SHARED YCQL DATABASE (Yugabyte)                          │

The diagram made it unmistakable: the S3 API lives inside the Kuri storage node layer, and the frontend proxies are a completely separate tier that clients connect to. The assistant's test cluster had collapsed these two layers into one, exposing Kuri nodes directly as S3 endpoints.

Why the Mistake Happened

Understanding why the assistant made this error is as instructive as understanding the correction itself. Several factors contributed:

First, there was a conflation of the S3 API as a protocol versus S3 API as a service boundary. The Kuri nodes do implement the S3 protocol—the roadmap explicitly shows "S3 API" inside each Kuri node box. But in the architecture, this is an internal API meant for consumption by the frontend proxies, not by external clients. The assistant's test cluster treated it as an external endpoint, which is a classic distributed systems error: just because a component speaks a protocol doesn't mean it should be directly addressable by clients.

Second, the assistant had been iterating rapidly on operational fixes. In the chunk preceding this message, the assistant had been debugging a series of practical issues: permission errors with YugabyteDB file ownership, database initialization idempotency, container status checking bugs, and missing CAR file staging configuration. This rapid-fire debugging context may have caused the assistant to lose sight of the architectural forest for the operational trees. When you're deep in fixing chmod errors and docker-compose ps flags, it's easy to forget that the entire topology is wrong.

Third, there was an implicit assumption that "test cluster" meant "simplified cluster." The assistant appears to have assumed that for testing purposes, collapsing the frontend proxy layer into the Kuri nodes was acceptable—a kind of "we'll add the proxy later" mentality. But this assumption violated a core architectural principle: the frontend proxy is not an optional add-on; it is the only entry point for clients, and it performs essential routing logic (round-robin for writes, YCQL lookup for reads) that cannot be bypassed without breaking the system's consistency guarantees.## The Thinking Process: A Window into Recognition

The assistant's reasoning section, included in the message, reveals the cognitive process of architectural correction. It begins with full acknowledgment: "You're absolutely right! I completely misunderstood the architecture." This is not defensive or hedging—it is a clean admission of error, which is essential for productive debugging.

The reasoning then enumerates three key facts from the roadmap, effectively re-reading the document with fresh eyes:

S3 Frontend Proxies are a SEPARATE node type (stateless, no storage)
Kuri nodes are storage nodes that expose S3 API internally
The architecture flow is Client → Frontend Proxy → Kuri Node This enumeration serves a dual purpose. First, it confirms to the user that the assistant now understands the architecture correctly. Second, it provides a checklist against which the assistant can evaluate its current implementation. The next two lines do exactly that: - "Kuri nodes with S3 API exposed directly (wrong)" - "No frontend proxy layer (missing)" This is a textbook example of the "recognize and enumerate" pattern in debugging: once you understand the correct model, you list what's wrong in your current implementation against that model. The final reasoning line—"I need to fix the test cluster to match the actual architecture from the roadmap. The frontend proxy should be a separate service that routes to Kuri nodes."—shifts from diagnosis to action. The assistant immediately begins implementing the fix by writing to docker-compose.yml.

The Output Knowledge Created

This message, while short, created several forms of output knowledge that shaped the subsequent work:

1. A corrected mental model. The most important output was internal to the development process: a shared understanding between user and assistant that the architecture has three distinct layers, not two. This mental model would guide every subsequent decision about port mappings, service dependencies, and configuration generation.

2. The first file write of the correction. The assistant wrote to docker-compose.yml immediately, beginning the process of restructuring the test cluster. This file would be further refined in subsequent messages, but the initial correction—removing direct S3 exposure from Kuri nodes and planning for a frontend proxy service—started here.

3. A template for further corrections. The message established a pattern that would repeat: acknowledge the error, restate the correct architecture, enumerate what's wrong, and begin fixing. In the messages that followed (indices 347-348), the assistant applied this same pattern to the README file, rewriting it to match the corrected architecture with a proper three-layer diagram.

Assumptions Made and Broken

The subject message is built on several assumptions, some correct and some that had to be revised:

Correct assumption: The roadmap document is the authoritative source of truth for the architecture. The assistant immediately deferred to the roadmap when the user pointed to it, treating it as the specification that must be implemented faithfully.

Correct assumption: The fix requires changing the Docker Compose topology, not just documentation. The assistant correctly identified that the architecture error was embodied in the service configuration and that correcting it meant rewriting the infrastructure.

Implicit assumption being corrected: That Kuri nodes could serve as direct S3 endpoints in a test environment. This was the fundamental error—the assistant had assumed that "Kuri nodes have S3 API" meant "clients talk to Kuri nodes directly," when the architecture requires that clients talk only to frontend proxies.

Implicit assumption being corrected: That a simplified test cluster could collapse architectural layers. The assistant had implicitly treated the frontend proxy as optional infrastructure, when in reality it is the essential routing layer that makes the entire distributed architecture work.

The Broader Lesson: Architecture as Infrastructure

The most profound lesson from this message is that architectural decisions must be reflected in infrastructure configuration, not just in code. The assistant had correctly implemented the frontend proxy package in Go code—the server/s3frontend/ directory with its router, backend pool, and YCQL integration was all written correctly. But the Docker Compose test cluster didn't include a frontend proxy service, meaning the code could never actually be exercised in the test environment.

This is a common pitfall in distributed systems development: you can write all the right code but still have the wrong system if your deployment topology doesn't match your architecture. The test cluster is not just a convenience—it is the place where architecture meets reality. If the test cluster doesn't faithfully represent the production topology, the tests provide false confidence.

The correction triggered by this message rippled outward. The assistant went on to generate separate configuration files for each Kuri node (with distinct EXTERNAL_LOCALWEB_URL values and ports), restructure the Docker Compose file into a proper three-layer hierarchy (S3 Proxy on port 8078 → Kuri storage nodes internally → YugabyteDB), and rewrite the proxy configuration to route through the stateless frontend layer. Each of these changes flowed directly from the fundamental realization captured in this single message: the architecture has three layers, not two, and the test cluster must reflect that.

Conclusion

This message, though only a few lines long, represents the critical moment where a distributed systems implementation was realigned with its architectural specification. It demonstrates the importance of having a clear, documented architecture to reference when implementation goes astray, the value of a user who can spot architectural inconsistencies, and the necessity of ensuring that test infrastructure faithfully represents production topology. The assistant's willingness to admit error, enumerate the correct model, and immediately begin corrective action turned a potentially costly architectural detour into a focused correction that strengthened the entire project.