The Architecture Correction That Saved a Distributed System

A Single User Message That Unraveled a Fundamental Design Flaw

In the course of building complex distributed systems, there are moments when a single observation cuts through layers of accumulated assumptions and reveals a fundamental misalignment between implementation and intent. This article examines one such moment: a user message in an AI-assisted coding session that identified a critical architectural error in a horizontally scalable S3-compatible storage system. The message, delivered with the simple preface "Wait," triggered a complete redesign of a test cluster infrastructure, realigning it with the documented architecture roadmap and preventing what would have been a deeply flawed deployment.

The subject of this analysis is message 345 in a longer conversation about building the Filecoin Gateway's distributed S3 storage system. In this message, the user calls attention to a discrepancy between what the assistant had implemented and what the project's architecture roadmap actually specified. The consequences of this correction rippled through the entire test cluster setup, changing the fundamental topology from a flat collection of Kuri storage nodes into a properly layered architecture with stateless S3 frontend proxies, independent storage backends, and per-node configuration management.

The Message in Full

The user's message, reproduced exactly as it appeared in the conversation, was:

Wait the S3 was supposed to be a separate node type and also all kuri nodes were meant to have S3 with subsets of objects managed by them, @scalable-roadmap.md

This was followed by a tool call that read the file /home/theuser/gw/scalable-roadmap.md, a 496-line document that laid out the complete architecture for the scalable S3 system. The roadmap document was displayed in full, providing both the user and the assistant with a definitive reference point for what the system was supposed to look like.

The brevity of the user's message belies its significance. The word "Wait" signals a moment of realization, a pause to reconsider. The user is not issuing a command or making a request; they are pointing out an inconsistency. The reference to @scalable-roadmap.md is both a citation and an appeal to authority—the roadmap document is the agreed-upon design specification, and the user is using it as evidence that the current implementation has gone off course.

The Context: What Led to This Moment

To understand why this message was necessary, we must examine the trajectory of the conversation that preceded it. The assistant had been working on building a test cluster for the horizontally scalable S3 architecture—a system designed to provide S3-compatible object storage using a distributed architecture with stateless frontend proxies and independent backend storage nodes.

The work had progressed through several phases. First, the assistant had created a comprehensive architecture roadmap document (scalable-roadmap.md) that specified a three-layer architecture:

S3 Frontend Proxy Layer (stateless) - Handles authentication, request routing, and load balancing
Kuri Storage Node Layer - Independent nodes storing object data with no replication between them
Shared YCQL Database Layer (YugabyteDB) - Tracks object placement metadata The roadmap was clear: frontend proxies were a separate node type that routed requests to Kuri storage nodes. Kuri nodes themselves had S3 APIs, but these were internal endpoints meant to be accessed by the frontend proxies, not directly by clients. Despite having created this roadmap, the assistant's subsequent implementation work had drifted from the specification. When building the test cluster infrastructure, the assistant had configured Kuri nodes to expose S3 APIs directly to clients, essentially treating them as the public-facing S3 endpoints. The frontend proxy layer—the entire point of the architecture—was missing from the test cluster setup. The conversation leading up to message 345 shows the assistant working on the test cluster with a series of fixes and adjustments. The assistant had created a gen-config.sh script to generate configuration files, updated docker-compose.yml to map ports, and modified start.sh to auto-generate configurations. The assistant's mental model at this point was that the test cluster consisted of: - Two Kuri nodes, each with their own S3 API port and LocalWeb (CAR file staging) port - A shared YugabyteDB instance - A web UI on kuri-1 for monitoring The assistant had even drawn an architecture diagram showing clients connecting directly to kuri-1's S3 API on port 8078 and kuri-2's LocalWeb on port 8444. The S3 frontend proxy layer was entirely absent from the diagram. Just before the critical message, the user had asked a more limited question: "Is there just one config? there needs to be one http endpoint per kuri node no?" This question was about the CAR file staging HTTP endpoints (LocalWeb), not about the S3 API architecture. The assistant had responded by creating separate configuration files for each Kuri node, each with its own LocalWeb URL and port. This was a step in the right direction—it recognized that each node needed its own identity—but it still operated within the flawed assumption that Kuri nodes were the public-facing S3 endpoints.

What the Roadmap Actually Specifies

The scalable-roadmap.md document that the user referenced is a detailed architectural specification. Its opening paragraph sets the tone:

Horizontally scalable S3-compatible storage with performance over replication. Stateless frontend proxies route to independent Kuri storage nodes. Shared YCQL tracks object placement.

The key phrase is "Stateless frontend proxies route to independent Kuri storage nodes." This establishes a clear separation of concerns: frontend proxies handle routing, and Kuri nodes handle storage. The roadmap's architecture diagram reinforces this with three distinct layers:

┌─────────────────────────────────────────────────────────────────────────────┐
│                      S3 FRONTEND PROXY LAYER (Stateless)                     │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                          │
│  │  Proxy #1   │  │  Proxy #2   │  │  Proxy #N   │        Round-robin writes │
│  │  • Auth     │  │  • Auth     │  │  • Auth     │        YCQL lookup reads  │
│  │  • Routing  │  │  • Routing  │  │  • Routing  │                          │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘                          │
└─────────┼────────────────┼────────────────┼──────────────────────────────────┘
          │                │                │
          └────────────────┴────────────────┘
                              │
┌─────────────────────────────┼─────────────────────────────────────────────────┐
│                      KURI STORAGE NODE LAYER                                   │
│  ┌─────────────┐  ┌────────┴────┐  ┌─────────────┐  Independent data         │
│  │  Kuri #1    │  │  Kuri #2    │  │  Kuri #N    │  No replication           │
│  │  • RIBS BS  │  │  • RIBS BS  │  │  • RIBS BS  │  Each node different objs │
│  │  • S3 API   │  │  • S3 API   │  │  • S3 API   │                          │
│  └─────────────┘  └─────────────┘  └─────────────┘                          │
└─────────────────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────┼─────────────────────────────────────────────────┐
│                      SHARED YCQL DATABASE (Yugabyte)                          │
│                                                                               │
│  • S3Objects (bucket, key) → ADD: node_id, expires_at                        │
│  • MultihashToGroup (unchanged)                                               │
│  • CidGroups (unchanged)                                                      │
└─────────────────────────────────────────────────────────────────────────────┘

This diagram makes the architecture explicit. The S3 frontend proxies are a separate layer from the Kuri storage nodes. Clients connect to proxies, not to Kuri nodes directly. The Kuri nodes have S3 APIs, but these are internal endpoints used by the proxies. The roadmap's data flow section further clarifies:

PUT Object (Round-Robin): Client → Frontend Proxy → Kuri Node #2 → YCQL
GET Object (Directed): Client → Frontend Proxy → YCQL Lookup → Kuri Node #2 → Client In both flows, the client never touches a Kuri node directly. The frontend proxy is the sole entry point for S3 operations. The roadmap also specifies configuration variables that make the separation clear. Frontend proxies have FGW_NODE_TYPE=frontend and a list of backend nodes via FGW_BACKEND_NODES. Kuri storage nodes have FGW_NODE_TYPE=storage and their own FGW_NODE_ID. These are distinct configuration profiles for distinct node types.

The Assistant's Misunderstanding

How did the assistant end up with a test cluster that violated the architecture it had itself documented? This is a fascinating question about the dynamics of AI-assisted development and the gap between documentation and implementation.

The assistant's reasoning traces show a gradual drift from the architectural specification. When initially building the test cluster, the assistant focused on getting Kuri nodes running with S3 APIs exposed. The thinking was pragmatic: "Let me create a test cluster with 2 Kuri nodes and see if they can serve S3 requests." The frontend proxy layer was acknowledged in comments as a future addition ("Note: This is a simplified test setup. The frontend proxy layer (for round-robin load balancing)...") but was never actually implemented in the test cluster.

The assistant's reasoning in message 328 (responding to the user's question about per-node HTTP endpoints) reveals the limited scope of the assistant's understanding at that point:

Looking at the docker-compose.yml, I currently have: - kuri-1 with ports 8078, 8443, 9010 exposed - kuri-2 with no ports exposed (internal only) - Both nodes share the same settings.env file

The assistant was thinking about LocalWeb endpoints (for CAR file staging), not about the S3 API architecture. The question about "one http endpoint per kuri node" was interpreted through the lens of CAR file staging URLs, not through the lens of the overall S3 routing architecture. The assistant's response created separate configuration files with different LocalWeb URLs and ports, but the fundamental architecture remained unchanged: Kuri nodes were still the public-facing S3 endpoints.

The assistant's architecture diagram from message 344 shows this clearly:

                         Clients
                            │
          ┌─────────────────┼─────────────────┐
          │                 │                 │
     ┌────┴────┐      ┌────┴────┐      ┌────┴────┐
     │ kuri-1  │      │ kuri-2  │      │  WebUI  │
     │ :8078   │      │ :8444   │      │ :9010   │
     │ :8443   │      │ (LocalWeb)     │         │
     │ (S3+Web)│      │ (CAR files)    │         │
     └────┬────┘      └────┬────┘      └─────────┘

Clients connect directly to kuri-1's S3 API on port 8078. There is no frontend proxy layer. This is a two-layer architecture (Kuri nodes → YugabyteDB) when the roadmap specifies a three-layer architecture (Frontend proxies → Kuri nodes → YugabyteDB).

The User's Reasoning and Motivation

The user's message reveals a sharp architectural awareness. The user had been reviewing the assistant's work and noticed a discrepancy between the implementation and the documented design. The use of @scalable-roadmap.md is telling—the user is not just pointing out a bug but is citing the project's own specification document as evidence.

The user's motivation appears to be architectural integrity. The scalable S3 architecture was designed with specific principles in mind:

Stateless frontend proxies that can be horizontally scaled independently
Independent Kuri storage nodes with no data replication between them
Performance through parallelism rather than through redundancy
Read-after-write consistency guaranteed through YCQL lookups By exposing Kuri nodes directly as S3 endpoints, the assistant's implementation would have undermined several of these principles. Clients would need to know which Kuri node to connect to for which objects, defeating the purpose of a unified S3 API. Load balancing would be impossible without the frontend proxy layer. The system would not be horizontally scalable in the way the architecture intended. The user's message also identifies a second issue: "all kuri nodes were meant to have S3 with subsets of objects managed by them." This points to the data distribution model. In the correct architecture, each Kuri node manages a subset of objects, and the YCQL database tracks which node holds which object. The frontend proxy uses this information to route GET requests to the correct node. The assistant's implementation, where both Kuri nodes were essentially identical and both exposed S3 APIs, didn't properly implement this object-subset model.

Assumptions Embedded in the Assistant's Implementation

The assistant's work leading up to message 345 was built on several assumptions, many of which were incorrect:

Assumption 1: Kuri Nodes Are the Public S3 Endpoints

The most fundamental incorrect assumption was that Kuri nodes should expose S3 APIs directly to clients. This assumption is visible throughout the docker-compose.yml configuration, where port 8078 is mapped for kuri-1's S3 API and described as the primary client-facing endpoint.

Assumption 2: The Frontend Proxy Is Optional for Testing

The assistant treated the frontend proxy layer as an optional component that could be added later. Comments in the README described it as a "simplified test setup" with the proxy layer noted as a future addition. This assumption conflated "simplified" with "architecturally different"—a simplified test setup should still reflect the correct architecture, just with fewer nodes or simpler configuration.

Assumption 3: Shared Configuration Is Sufficient

The assistant initially created a single shared configuration file for both Kuri nodes. Even after the user pointed out the need for per-node HTTP endpoints, the assistant's fix created separate LocalWeb URLs but kept the overall architecture the same. The assumption was that the only per-node difference was the CAR file staging URL, not the entire role in the architecture.

Assumption 4: The S3 API Is a Property of Kuri Nodes

The assistant treated the S3 API as an integral part of each Kuri node, similar to how a web server has an HTTP port. In the correct architecture, the S3 API is a property of the frontend proxy layer, and Kuri nodes have internal S3 endpoints that are not directly exposed to clients.

Assumption 5: The Test Cluster Should Mirror a Traditional S3 Setup

The assistant's mental model seems to have been influenced by traditional S3-compatible storage systems where each storage node exposes an S3 API directly. The assistant was building what looked like a standard S3 cluster rather than the specific distributed architecture specified in the roadmap.

Input Knowledge Required to Understand This Message

To fully grasp the significance of the user's message, several pieces of context are necessary:

Knowledge of the Scalable S3 Architecture

The reader needs to understand the three-layer architecture: stateless frontend proxies, independent Kuri storage nodes, and a shared YCQL database. The architecture is designed for horizontal scalability where performance comes from parallelism rather than data replication.

Knowledge of the Roadmap Document

The scalable-roadmap.md file is the definitive specification for the system. It contains the architecture diagram, data flow descriptions, configuration details, and implementation phases. The user's reference to this document assumes that both the user and the assistant have a shared understanding of its contents.

Knowledge of the Test Cluster Infrastructure

The test cluster consists of Docker Compose services, shell scripts for configuration generation and startup, and environment variable-based configuration. Understanding the assistant's error requires understanding how the test cluster was structured and what ports and services were exposed.

Knowledge of Kuri Node Capabilities

Kuri nodes are storage backends that use RIBS (a block storage system) and have built-in S3 API capabilities. They also have a LocalWeb server for CAR file staging (used for Filecoin deal-making) and a web UI for monitoring. Understanding which of these capabilities should be exposed externally and which should be internal is crucial.

Knowledge of Filecoin and CAR Files

The system is built for the Filecoin network, where storage providers download CAR files (Content Addressable aRchives) containing deal data. The LocalWeb server on each Kuri node serves these CAR files to storage providers. This explains why each node needs its own HTTP endpoint for CAR file staging.

Output Knowledge Created by This Message

The user's message created several important outputs:

A Corrected Understanding of the Architecture

The most significant output was a corrected mental model of the system architecture. The assistant's reasoning after message 345 shows a complete re-evaluation:

You're absolutely right! I completely misunderstood the architecture. Looking back at the roadmap: 1. S3 Frontend Proxies are a SEPARATE node type (stateless, no storage) 2. Kuri nodes are storage nodes that expose S3 API internally 3. The architecture is: Client → Frontend Proxy → Kuri Node (with S3 API)

This corrected understanding led to a complete rewrite of the test cluster infrastructure.

A Redesigned Docker Compose Configuration

The assistant rewrote docker-compose.yml to implement the correct three-layer architecture. The new configuration included:

A frontend proxy service that clients connect to
Kuri storage nodes that are internal (no public S3 ports)
Each Kuri node with its own LocalWeb port for CAR file staging
Proper service dependencies and health checks

Per-Node Independent Configuration

The configuration generation was redesigned to produce separate settings files for each node type. Frontend proxies get one configuration profile, and Kuri storage nodes get another. Each Kuri node has its own FGW_NODE_ID and its own EXTERNAL_LOCALWEB_URL.

An Updated Architecture Diagram

The assistant's new architecture diagram reflected the correct three-layer structure:

                         Clients
                            │
                     ┌──────┴──────┐
                     │  S3 Proxy   │
                     │  (port 8078)│
                     └──────┬──────┘
                            │
          ┌─────────────────┼─────────────────┐
          │                 │                 │
     ┌────┴────┐      ┌────┴────┐      ┌────┴────┐
     │ kuri-1  │      │ kuri-2  │      │  WebUI  │
     │ :8443   │      │ :8444   │      │ :9010   │
     │ (LocalWeb)     │ (LocalWeb)     │         │
     └────┬────┘      └────┬────┘      └─────────┘
          │                │
          └────────────────┘
                   │
          ┌────────┴────────┐
          │   YCQL/YSQL     │
          │   YugabyteDB    │
          └─────────────────┘

A New Implementation Trajectory

The corrected understanding changed the assistant's implementation priorities. Instead of continuing to refine a flawed test cluster, the assistant now needed to:

Create the frontend proxy service (which didn't exist yet in the test cluster)
Reconfigure Kuri nodes as internal storage backends
Implement the routing logic that the frontend proxy would use
Update all scripts and documentation to reflect the correct architecture

The Thinking Process: Before and After

The assistant's reasoning traces provide a window into the cognitive shift triggered by the user's message.

Before the Correction

In message 328, the assistant's reasoning shows a focus on the LocalWeb endpoint issue:

The user is asking about the configuration - specifically whether there should be one config or separate configs per node, particularly regarding HTTP endpoints. ... If both nodes share the same EXTERNAL_LOCALWEB_URL, they'll both try to use the same URL for CAR file staging, which doesn't make sense. Each node should have its own LocalWeb endpoint for storage providers to fetch CAR files from.

The assistant is thinking about CAR file staging URLs, not about the overall S3 architecture. The reasoning is tactical, not architectural. The assistant is solving a specific configuration problem without questioning the fundamental topology.

After the Correction

In message 346, the assistant's reasoning shows a complete re-evaluation:

You're absolutely right! I completely misunderstood the architecture. Looking back at the roadmap: 1. S3 Frontend Proxies are a SEPARATE node type (stateless, no storage) 2. Kuri nodes are storage nodes that expose S3 API internally 3. The architecture is: Client → Frontend Proxy → Kuri Node (with S3 API)

The current test cluster setup has: - Kuri nodes with S3 API exposed directly (wrong) - No frontend proxy layer (missing)

The language is strikingly different. The assistant uses words like "completely misunderstood" and identifies specific errors ("Kuri nodes with S3 API exposed directly (wrong)"). The reasoning is now architectural, not tactical. The assistant is thinking about the system as a whole and how its components relate to each other.

This cognitive shift is the most valuable output of the user's message. The assistant had been operating with a flawed mental model, and the user's intervention forced a re-examination of that model against the documented specification.

The Broader Lessons

This episode offers several lessons about system design and AI-assisted development:

Documentation Must Be Referenced, Not Just Written

The assistant had written the roadmap document but then proceeded to implement something different. This is a common pitfall in software development: the act of writing documentation can create a false sense of having internalized it. The user's act of referencing the document—literally calling it up and reading it—forced a confrontation between the documented design and the actual implementation.

The Value of Architectural Review

The user's message is essentially an architectural review. The user didn't write code or fix bugs; they identified a misalignment between design and implementation. This is a high-value intervention that prevents the accumulation of architectural debt. In many projects, such misalignments go unnoticed until they cause significant problems during integration or deployment.

Small Questions Can Reveal Big Problems

The user's initial question ("Is there just one config? there needs to be one http endpoint per kuri node no?") was about configuration files, but it led to the discovery of a fundamental architectural error. This illustrates the value of following a thread of inquiry even when it seems like a minor detail. The assistant's response to the first question fixed the configuration issue but didn't address the underlying architectural problem. The user's follow-up ("Wait the S3 was supposed to be a separate node type") showed that the user was thinking at a higher level of abstraction.

The Danger of "Simplified" Test Setups

The assistant's comment that the test cluster was a "simplified test setup" with the frontend proxy layer noted as a future addition is a classic example of a dangerous assumption. Simplified test setups that deviate from the production architecture can mask fundamental design issues. If the test cluster doesn't match the production architecture, tests run on the cluster may not validate the correct behavior.

The Importance of Per-Node Identity

The user's insistence on per-node HTTP endpoints and the assistant's eventual implementation of per-node configuration files highlights the importance of node identity in distributed systems. Each node in a distributed system needs its own identity, configuration, and endpoints. Shared configuration that treats all nodes as identical can work for simple cases but breaks down when nodes need to be addressed individually.

The Aftermath: How the Architecture Was Corrected

Following the user's message, the assistant undertook a comprehensive redesign of the test cluster. The key changes included:

Complete Rewrite of docker-compose.yml

The assistant rewrote the Docker Compose configuration to implement the correct three-layer architecture. The new configuration included:

A frontend proxy service with FGW_NODE_TYPE=frontend
Kuri storage nodes with FGW_NODE_TYPE=storage and no public S3 ports
Internal networking between the proxy and Kuri nodes
Per-node LocalWeb ports for CAR file staging

Redesigned Configuration Generation

The gen-config.sh script was rewritten to generate separate configuration files for each node type. Frontend proxies get configuration with FGW_BACKEND_NODES listing the Kuri nodes. Kuri nodes get configuration with their own FGW_NODE_ID and EXTERNAL_LOCALWEB_URL.

Updated Documentation

The README was updated to reflect the correct architecture, including a new architecture diagram showing the three-layer structure. The documentation now clearly describes the role of each component and how they interact.

New Implementation Priorities

The corrected understanding shifted the assistant's implementation priorities. Instead of continuing to refine the flawed test cluster, the assistant now needed to:

Implement the frontend proxy service (which was previously deferred)
Configure Kuri nodes as internal storage backends
Implement YCQL-based routing in the frontend proxy
Test the complete PUT/GET round-trip through the proxy layer

Conclusion

The user's message at index 345 is a masterclass in architectural correction. In just a few words—"Wait the S3 was supposed to be a separate node type"—the user identified a fundamental misalignment between implementation and design that had been accumulating across multiple coding sessions. The message is remarkable not for its length or complexity but for its precision and timing.

The episode illustrates several truths about building distributed systems. First, architecture matters more than code—a perfectly implemented system with the wrong architecture is still wrong. Second, documentation is only useful if it's actively referenced against implementation. Third, the most valuable interventions in a development process are often not code changes but conceptual corrections that realign the implementer's mental model with the design specification.

For the AI-assisted development process, this message demonstrates both the strengths and limitations of current systems. The assistant was capable of writing sophisticated code, generating configuration files, and building complex infrastructure. But it was also capable of drifting from the architectural specification without realizing it, building an internally consistent but architecturally incorrect system. The human user's role was not to write code but to provide the architectural oversight that kept the system aligned with its design goals.

The corrected test cluster that emerged from this intervention is a better system: it properly separates concerns, allows independent scaling of frontend proxies and storage nodes, and provides a foundation for the remaining implementation phases. More importantly, the correction happened early, before the flawed architecture could propagate into production code or become entrenched through further development.

In the end, the most important line in the entire conversation might be the user's simple "Wait"—a pause, a reconsideration, and a reference back to the document that everyone had agreed would guide the implementation. It is a reminder that in complex system design, the most valuable tool is not a programming language or a framework but a clear architectural vision and the willingness to stop and check whether the implementation still matches it.