The Moment Architecture Met Reality: A Debugging Session That Reshaped a Distributed S3 Cluster

Introduction

In the life of any complex software project, there are moments when carefully laid plans collide with the unforgiving reality of running code. These moments often arrive not as dramatic revelations but as quiet failures—a container that won't start, a log message that doesn't make sense, a configuration error that points to something deeper. This article examines one such moment: a single user message (index 301) from a coding session building a horizontally scalable S3-compatible storage architecture for the Filecoin Gateway. What begins as a routine test cluster startup quickly becomes a diagnostic journey that uncovers a fundamental architectural misunderstanding, ultimately forcing a complete redesign of the system's core topology.

The message captures the user running the test cluster startup script, observing both storage nodes fail to launch, inspecting the logs of one failed node, and then delivering a piercing insight that reframes the entire approach. To understand the significance of this message, we must first understand the context in which it was written, the assumptions that preceded it, and the cascade of realizations it triggered.

The Message: What the User Actually Saw and Said

The subject message begins with the user executing the startup script for the test cluster:

theuser@biryani  ~/gw/test-cluster  pgf-port ● ? ↑2  ./start.sh /data/fgw2

The script runs through its initialization sequence—checking for the Docker image, creating data directories, handling permissions—and then launches the cluster with Docker Compose. All five containers are created: the YugabyteDB database, the database initialization container, and two Kuri storage nodes. The database initialization succeeds. But then the script reports:

❌ kuri-1 is not running. Check logs: docker-compose logs kuri-1
❌ kuri-2 is not running. Check logs: docker-compose logs kuri-2

Both storage nodes have failed to start. The user then runs docker-compose logs kuri-2 to investigate, and the logs reveal a cascade of initialization steps followed by a fatal error:

kuri-2-1  | 2026/01/30 21:18:33 [watchdog] initialized watermark watchdog policy...
kuri-2-1  | Configuration load failed: %w invalid log level: 
kuri-2-1  | generating ED25519 keypair...done
kuri-2-1  | peer identity: 12D3KooWH1vT6NsLsdhgz8AhSEGsyRU9csdctCEk9PXf3dgwnNEQ
kuri-2-1  | initializing IPFS node at /root/.ipfs
kuri-2-1  | CREATED NEW GATEWAY WALLET
kuri-2-1  | ADDRESS:  f1y656v3gjmaw7ynejq45fggi6yqtwfq2dspns5da
kuri-2-1  | RIBS Wallet:  f1y656v3gjmaw7ynejq45fggi6yqtwfq2dspns5da
kuri-2-1  | 2026-01-30T21:18:33.406Z  ERROR  core  core/builder.go:158
kuri-2-1  | constructing the node: could not build arguments...
kuri-2-1  | ...failed to build iface.RIBS: received non-nil error...
kuri-2-1  | ...trying to initialize external offload: no external module configured

The error chain traces through the dependency injection framework (using Uber's fx library) to reveal that the RIBS blockstore cannot be constructed because "no external module configured." This is the immediate technical failure.

But the true significance of this message lies in the user's final line—a single sentence that reframes everything:

start should do initial gwcfg, then transplant settings.env to the other node and only ask for second LocalWeb endpoint

This is not merely a debugging observation. It is a fundamental critique of the architecture as implemented, pointing toward a completely different deployment topology than what the assistant had built.

The Context: What Led to This Moment

To understand why this message matters, we need to understand what came before it. The assistant had been building a test cluster infrastructure for the horizontally scalable S3 architecture. The architecture, as documented in a scalable-roadmap.md file, called for a clean separation between stateless S3 frontend proxy nodes and backend Kuri storage nodes. The frontend proxies would handle request routing, load balancing, and S3 API compatibility, while the Kuri nodes would store data independently using the RIBS blockstore, coordinated through a shared YugabyteDB database tracking object placement.

However, in the process of implementing the test cluster, the assistant had made a critical architectural error. The Docker Compose configuration exposed Kuri nodes directly as S3 endpoints, with kuri-1 exposing its S3 API on port 8078 and its web UI on port 9010, while kuri-2 ran internally with no exposed ports. Both nodes shared a single configuration file (settings.env), and neither was configured with the CAR file staging storage that Kuri nodes require for external data offloading.

The preceding messages in the conversation show a series of smaller bugs being fixed: permission errors when trying to chmod files owned by root inside Docker containers, the database initialization failing because the database already existed from a previous run, and the startup script's wait logic failing because docker-compose ps doesn't show exited containers by default (requiring the -a flag). Each of these was a relatively straightforward operational fix.

But the failure of the Kuri nodes to start was different. It wasn't a simple scripting error or a missing flag. It was a symptom of a deeper architectural mismatch between what the roadmap specified and what had been implemented.

The Error: "No External Module Configured"

The error message itself deserves careful analysis. The Kuri node's initialization process involves several stages:

Watchdog initialization: The watermark watchdog policy is set up for monitoring disk usage.
Configuration loading: The node attempts to load its configuration but reports "invalid log level," suggesting a configuration parsing issue.
Key generation: An ED25519 keypair is generated for peer identity.
IPFS node initialization: The node initializes an IPFS node at the default data directory.
Wallet creation: A new Filecoin gateway wallet is created and backed up.
RIBS blockstore construction: This is where the fatal error occurs. The error propagates through the dependency injection graph: - StartS3Server depends on S3Server - S3Server depends on Blockstore - Blockstore depends on RIBS (the core storage interface) - RIBS construction fails because it tries to initialize "external offload" but finds "no external module configured" The "external module" in this context refers to the CAR file staging mechanism. Kuri nodes are designed to stage incoming data as CAR (Content Addressable aRchive) files through an external HTTP endpoint before committing them to the blockstore. This is a fundamental part of the Kuri architecture—without it, the node cannot accept data. The assistant had not configured this external endpoint, assuming the nodes could operate with just the shared database configuration. But the deeper issue, as the user recognized, was not merely a missing configuration value. It was that the entire deployment topology was wrong. The Kuri nodes should not be exposed as direct S3 endpoints at all. They should be internal storage nodes, each with their own independent external HTTP endpoint for CAR file staging, behind a stateless S3 frontend proxy layer that handles routing and load balancing.

The User's Insight: A Fundamental Architecture Correction

The user's final comment reveals a sophisticated understanding of the architecture that the assistant had not fully internalized. The phrase "start should do initial gwcfg" refers to the need for an initial configuration step that sets up the first Kuri node properly—creating its wallet, configuring its external endpoint, and establishing its identity. Then, crucially, "transplant settings.env to the other node"—each node needs its own configuration file, not a shared one, because each node has its own external HTTP endpoint for CAR staging. And finally, "only ask for second LocalWeb endpoint"—the user recognizes that the second node needs its own distinct EXTERNAL_LOCALWEB_URL configuration.

This insight reveals several incorrect assumptions that had been baked into the assistant's implementation:

Assumption 1: Kuri nodes can share a single configuration. The assistant had created a single settings.env file used by both nodes. In reality, each Kuri node needs its own configuration because each has a unique external endpoint for CAR file staging, a unique peer identity, and potentially different port allocations.

Assumption 2: Kuri nodes function as S3 endpoints. The assistant had configured kuri-1 to expose its S3 API directly on port 8078, treating it as the S3 service endpoint. The roadmap, however, specifies that S3 frontend proxies are a separate stateless node type that routes requests to Kuri storage nodes. The Kuri nodes should be internal, accessible only through the proxy layer.

Assumption 3: The external module is optional or can be configured later. The error message "no external module configured" is a fatal initialization error, not a warning. The external CAR staging module is a required component of the Kuri node, not an optional feature.

Assumption 4: The test cluster can use a simplified topology for testing. The assistant had created a two-node cluster where one node was "primary" (exposed ports) and the other was "internal only." This asymmetry was a shortcut that violated the architecture's principle of symmetric, independently-operating storage nodes.

The Thinking Process Revealed

The user's message reveals a diagnostic thinking process that moves from observation to inference to architectural insight. The sequence is worth examining:

Observation: The startup script runs but both Kuri nodes fail to start.
Data collection: The user runs docker-compose logs kuri-2 to gather diagnostic information.
Error analysis: The log shows a chain of dependency injection failures culminating in "no external module configured."
Pattern recognition: The user recognizes that this isn't just a missing config value but a symptom of a fundamentally incorrect deployment topology.
Synthesis: The user articulates the correct approach in a single, dense sentence that captures the entire workflow: configure the first node, generate its settings, transplant the configuration to the second node with modifications, and ensure each node has its own external endpoint. The final line is particularly remarkable for its density of meaning. In 18 words, the user communicates: - The need for an initial configuration step (start should do initial gwcfg) - The need for per-node configuration files (transplant settings.env to the other node) - The need for distinct external endpoints (only ask for second LocalWeb endpoint) - The sequencing of operations (then) This is the kind of insight that only comes from deep familiarity with the system architecture. The user isn't just debugging a configuration error—they are recognizing that the entire deployment model is wrong and articulating the correct model.

The Aftermath: A Complete Redesign

The consequences of this message were immediate and far-reaching. In the subsequent messages (not shown in our context but summarized in the chunk analysis), the assistant completely redesigned the test cluster:

A configuration generation script (gen-config.sh) was created to produce separate settings.env files for each node, with distinct EXTERNAL_LOCALWEB_URL values and port allocations.
The Docker Compose file was restructured into a proper three-layer architecture: an S3 proxy on port 8078 routing to Kuri storage nodes internally, which in turn connect to the shared YugabyteDB.
The proxy configuration was rewritten to route through the stateless frontend layer as specified in the roadmap, completing Phase 3 of the implementation plan. This redesign addressed all four incorrect assumptions identified above. The Kuri nodes became internal storage nodes with independent configurations, the S3 frontend proxy became a separate stateless service, and the external CAR staging module was properly configured for each node.

Lessons in Distributed Systems Architecture

This message and its aftermath offer several valuable lessons for distributed systems design:

The architecture document is the source of truth. The assistant had read the scalable-roadmap.md but had not fully internalized its implications. The user, by pointing directly to the document, reminded the assistant that the architecture had already been designed—the task was to implement it faithfully, not to reinterpret it.

Operational failures often reveal design failures. The "no external module configured" error could have been fixed by simply adding a configuration value. But the user recognized that the error was a symptom of a deeper problem: the nodes were deployed in the wrong topology. Fixing the surface error without addressing the underlying design would have led to more problems later.

Per-node configuration is essential in distributed systems. The assumption that two nodes could share a single configuration file is a common mistake in distributed systems design. In practice, each node in a distributed system needs its own identity, its own network endpoints, and often its own cryptographic keys. Configuration must be generated per-node, not shared.

Stateless proxies enable horizontal scalability. The core insight of the architecture is that S3 frontend proxies are stateless and can be scaled horizontally, while Kuri storage nodes are stateful and scale through parallelism rather than replication. The assistant's initial implementation blurred this distinction by making Kuri nodes act as both storage and S3 endpoints.

Conclusion

Message 301 of this coding session is a masterclass in architectural debugging. What begins as a routine test cluster startup—a script run, a failure observed, a log inspected—becomes a moment of fundamental insight that reshapes the entire project. The user's ability to look past the immediate error message ("no external module configured") and recognize the deeper architectural problem is the kind of skill that separates experienced systems architects from developers who merely fix bugs.

The message also illustrates a crucial dynamic in human-AI collaboration: the AI assistant can implement complex systems rapidly, but it can also make subtle but profound architectural errors. The human collaborator, with deeper domain knowledge and a clearer vision of the intended architecture, provides the essential corrective feedback that keeps the project on track. The final line of the message—"start should do initial gwcfg, then transplant settings.env to the other node and only ask for second LocalWeb endpoint"—is not just a debugging note. It is the articulation of a correct architectural vision that the assistant had missed, and it set the project back on the path toward a properly designed distributed S3 storage system.

In the end, the test cluster that emerged from this correction was architecturally sound: stateless frontend proxies routing to independently-configured Kuri storage nodes, all coordinated through a shared metadata database. The failure of the initial implementation was not a waste of effort but a necessary step in the learning process—for both the human and the AI—that ultimately produced a better design.