The Per-Node Configuration Epiphany: Correcting a Distributed Architecture's Identity Crisis

Introduction

In distributed systems, few things are more fundamental than identity. Each node in a cluster must know who it is, what resources it owns, and how other nodes and clients can reach it. When building a horizontally scalable storage system, conflating node identities can silently undermine the entire architecture, creating bottlenecks, routing failures, and operational confusion. Message 344 in this coding session captures the moment when a subtle but critical architectural flaw was identified and corrected—the realization that sharing a single configuration across multiple storage nodes was fundamentally wrong, and that each node required its own independent identity and network endpoint.

This message, written by the assistant in response to a sharp user observation, represents more than just a configuration fix. It embodies a deeper lesson about distributed system design: that statelessness and independence must be enforced at every layer, and that convenience-driven shortcuts in configuration management can violate core architectural principles. The message serves as the culmination of a rapid debugging and redesign cycle, transforming a shared-configuration test cluster into a properly isolated multi-node deployment where each Kuri storage node stands on its own.

The Message in Full

The assistant's response reads:

Fixed! You're absolutely right - each Kuri node needs its own HTTP endpoint for CAR file staging.

>

## Changes Made

>

### 1. Separate Configs Per Node - gen-config.sh now creates two separate configs: - config/kuri-1/settings.env → http://localhost:8443 - config/kuri-2/settings.env → http://localhost:8444

>

### 2. Updated Port Mapping - kuri-1: :8078 (S3), :8443 (LocalWeb), :9010 (Web UI) - kuri-2: :8444 (LocalWeb) - now exposed for SP downloads

>

### 3. Architecture `` Clients │ ┌─────────────────┼─────────────────┐ │ │ │ ┌────┴────┐ ┌────┴────┐ ┌────┴────┐ │ kuri-1 │ │ kuri-2 │ │ WebUI │ │ :8078 │ │ :8444 │ │ :9010 │ │ :8443 │ │ (LocalWeb) │ │ │ (S3+Web)│ │ (CAR files) │ │ └────┬────┘ └────┬────┘ └─────────┘ │ │ └────────────────┘ │ ┌────────┴────────┐ │ YCQL/YSQL │ │ YugabyteDB │ └─────────────────┘ ``

>

### 4. Usage ```bash # Generate configs and start ./start.sh /data/fgw-test

>

# Or with custom domain for NAT/reverse proxy ./gen-config.sh /data/fgw-test fgw.example.com ./start.sh /data/fgw-test ```

>

For production NAT setup: - fgw.example.com:8443 → routes to host:8443 (kuri-1 CAR files) - fgw.example.com:8444 → routes to host:8444 (kuri-2 CAR files)

The message is concise, confident, and corrective. It acknowledges the user's insight, enumerates the specific changes made, and provides a clear architectural diagram showing the corrected topology. But to understand why this message matters, we must examine the chain of events that led to it.

The Context: A Cluster Built on a Shared Illusion

The conversation leading up to message 344 reveals a test cluster infrastructure that had been rapidly assembled over several coding sessions. The assistant had built a Docker Compose-based test environment with two Kuri storage nodes, a shared YugabyteDB instance, and supporting scripts for initialization, startup, and monitoring. The architecture was intended to implement a horizontally scalable S3-compatible storage system for the Filecoin Gateway, where stateless frontend proxies route requests to independent backend storage nodes.

However, in the rush to get something working, a critical shortcut had been taken. Both Kuri nodes shared a single settings.env configuration file. The assistant had attempted to work around this by having kuri-2 override its EXTERNAL_LOCALWEB_URL to point to kuri-1's endpoint—a kludge that meant kuri-2's CAR files would be served from kuri-1's HTTP server. This approach violated the fundamental principle that each storage node should be independently operable and directly accessible.

The user's question in message 327—"Is there just one config? there needs to be one http endpoint per kuri node no?"—was deceptively simple. It exposed a flaw that, if left uncorrected, would have caused cascading problems: storage providers trying to download CAR files from kuri-2 would be redirected to kuri-1, creating routing confusion, potential data unavailability, and a complete breakdown of the horizontal scaling model. The user understood that in a distributed storage system, every node that can make Filecoin deals must have its own reachable endpoint for storage providers to fetch staged CAR files.

Why This Message Was Written: The Motivation and Reasoning

The assistant wrote message 344 to document and summarize a rapid correction cycle that spanned messages 327 through 343. The motivation was threefold.

First, there was an immediate operational need: the test cluster was non-functional for its intended purpose. If both nodes shared a single LocalWeb URL, storage providers could not independently download CAR files from kuri-2. The architecture as implemented would not survive even basic multi-node testing.

Second, there was an architectural principle at stake. The entire project was built around a roadmap document (scalable-roadmap.md) that explicitly called for stateless frontend proxies and independent storage nodes. The shared-configuration approach contradicted this design. The assistant needed to realign the implementation with the documented architecture.

Third, there was a pedagogical dimension. The message serves as a clear record of what was wrong and how it was fixed. The architecture diagram, the port mapping table, and the usage examples all communicate the corrected design to anyone reading the codebase later. This is not just a fix—it's documentation of the fix.

The thinking visible in the assistant's reasoning traces a clear path: read the current docker-compose.yml, understand the problem (both nodes sharing config, kuri-2's LocalWeb pointing to kuri-1), design the solution (separate configs per node, unique ports, independent URLs), implement the changes (update gen-config.sh, docker-compose.yml, README, start.sh), and validate (test config generation, verify docker-compose validity). Each step is deliberate and verified.

The Decisions Made: From Shared to Independent

Several key decisions are embedded in this message and the surrounding work.

Decision 1: Per-node configuration files. Rather than a single settings.env, the assistant created config/kuri-1/settings.env and config/kuri-2/settings.env. This is the most consequential decision. It means each node can have its own FGW_NODE_ID, its own EXTERNAL_LOCALWEB_URL, and its own EXTERNAL_LOCALWEB_SERVER_PORT. The nodes are no longer interchangeable clones—they have distinct identities.

Decision 2: Unique port allocation. kuri-1's LocalWeb server runs on port 8443, while kuri-2's runs on port 8444. This is a practical necessity for Docker host port mapping, but it also reflects the deeper principle that each node occupies its own network namespace. In production, these would be different hostnames or IP addresses, but for local testing, port differentiation suffices.

Decision 3: Exposing kuri-2's LocalWeb externally. Previously, kuri-2 had no exposed ports—it was effectively a hidden node. The corrected configuration maps port 8444 to the host, making kuri-2's CAR files directly accessible to storage providers. This is essential for the horizontal scaling model to work.

Decision 4: Updating the configuration generator. The gen-config.sh script was rewritten to generate two separate configuration files with distinct settings. The script accepts an external domain parameter and generates appropriate URLs for each node (e.g., http://mydomain.com:8443 and http://mydomain.com:8444). This makes the configuration process repeatable and automated.

Decision 5: Updating startup scripts and documentation. The start.sh script was modified to check for the new config paths (config/kuri-1/settings.env and config/kuri-2/settings.env), and the README was updated with corrected architecture diagrams, port tables, and usage instructions. These changes ensure that the fix is not just a one-time correction but is baked into the project's operational procedures.

Assumptions Made and Mistakes Corrected

The most significant incorrect assumption was that a single configuration could serve multiple nodes. This assumption likely arose from a desire for simplicity—one config file is easier to manage than two—but it conflicted with the architectural requirement for node independence. The assistant had assumed that the EXTERNAL_LOCALWEB_URL setting was a cluster-wide property rather than a per-node property.

A related mistake was the workaround of having kuri-2 point to kuri-1's LocalWeb URL. This attempted to paper over the configuration problem without solving it. It assumed that CAR file staging could be centralized on one node, which would be true only if both nodes stored their CAR files in a shared filesystem—an assumption that was never validated and that contradicted the independent-storage design.

Another assumption worth examining is that kuri-2 could operate as a "hidden" node without its own externally accessible endpoint. This might have been acceptable in a read-only replica scenario, but Kuri nodes are designed to initiate Filecoin deals, which require storage providers to download staged CAR files. A node that cannot serve its staged files is effectively non-functional for deal-making.

The user's question cut through these assumptions with surgical precision. By asking whether each node needs its own HTTP endpoint, the user forced a re-examination of the entire configuration model. The assistant's rapid acknowledgment—"You're absolutely right"—signals that the flaw was immediately recognized once pointed out.

Input Knowledge Required

To understand this message, a reader needs several pieces of context:

  1. The Kuri storage node architecture: Kuri nodes are backend storage servers in a horizontally scalable S3-compatible system. Each node maintains its own RIBS blockstore data and can independently stage CAR files for Filecoin deals.
  2. The LocalWeb server: This is an HTTP server built into each Kuri node that serves staged CAR files to storage providers (SPs) for download and sealing into Filecoin sectors. The EXTERNAL_LOCALWEB_URL setting tells storage providers where to find these files.
  3. The Filecoin deal workflow: When a Kuri node makes a deal with a storage provider, the SP needs to download the CAR file containing the data. This requires a reachable HTTP endpoint. If the endpoint points to the wrong node, the SP cannot retrieve the data.
  4. The Docker Compose test infrastructure: The test cluster runs multiple containers (kuri-1, kuri-2, yugabyte) orchestrated by Docker Compose. Port mapping, volume mounting, and environment variable sourcing are all managed through the compose file.
  5. The gen-config.sh script: This is a configuration generator that creates settings.env files with database connection info, S3 API settings, and LocalWeb configuration. Understanding its role is essential to appreciating the fix.

Output Knowledge Created

This message creates several valuable artifacts of knowledge:

  1. A corrected architectural model: The diagram in the message shows the proper topology where each Kuri node has its own LocalWeb endpoint, both nodes connect to the shared YugabyteDB, and clients can reach either node independently.
  2. A repeatable configuration pattern: The per-node configuration approach establishes a template for scaling to more than two nodes. Adding kuri-3 would follow the same pattern: a new config file, a new port, and a new LocalWeb URL.
  3. Operational documentation: The NAT/reverse proxy routing table at the end of the message provides concrete guidance for production deployment. Anyone setting up a reverse proxy knows exactly which ports to route.
  4. A validation methodology: The assistant's reasoning shows a pattern of testing each change—validating docker-compose config, checking generated files, verifying environment variables. This establishes a quality standard for infrastructure changes.

The Thinking Process: A Window into Debugging Methodology

The assistant's reasoning across messages 327-343 reveals a systematic debugging approach. When the user raised the issue, the assistant didn't immediately start coding. Instead, it:

  1. Analyzed the current state: Read the docker-compose.yml to understand the existing configuration.
  2. Identified the root cause: Both nodes shared the same config, and kuri-2's LocalWeb pointed to kuri-1.
  3. Designed the solution: Separate configs, unique ports, independent URLs.
  4. Implemented incrementally: First the config generator, then the compose file, then the scripts, then the documentation.
  5. Validated at each step: Tested config generation, verified environment variables, checked docker-compose validity. This is a textbook example of disciplined infrastructure engineering. Each change is isolated, tested, and confirmed before moving to the next. The assistant also cleaned up temporary test files at the end (rm -rf /tmp/test-config /tmp/test-config2), demonstrating attention to operational hygiene.

Conclusion

Message 344 is a small message with large implications. It represents the moment when a distributed storage cluster gained its proper identity—not as a monolithic system with interchangeable parts, but as a federation of independent nodes, each with its own name, its own port, and its own reachable endpoint. The fix was technically simple (split one config file into two), but architecturally profound (each node is now a first-class citizen of the cluster).

The message also illustrates a crucial dynamic in collaborative system design: the value of a fresh pair of eyes. The assistant had been deep in implementation, focused on getting things running. The user, stepping back and looking at the architecture holistically, spotted the inconsistency immediately. The best engineering happens when builders and reviewers challenge each other's assumptions, and message 344 is a testament to that collaborative process.

In the end, the corrected test cluster is not just more correct—it is more scalable. Adding a third node now requires only a new config file and a new port, following the established pattern. The architecture can grow horizontally without accumulating technical debt from shared configuration. And that, ultimately, is what horizontal scalability is all about.