The Reversion: Why a Failed Experiment with Host Networking Shaped a More Stable Test Cluster

Introduction

In the middle of a complex debugging session for a horizontally scalable S3 storage architecture built on Kuri storage nodes and YugabyteDB, a single command appears deceptively simple. Message 1319 in the conversation shows the assistant running ./start.sh /data/fgw2 to launch a test cluster. On its surface, this is a routine operation—starting Docker containers that have been configured and built over many prior iterations. But the context surrounding this message reveals something far more significant: it is the culmination of a painful but necessary retreat from an architectural experiment that failed, and the beginning of a path toward a stable, working test environment.

The message reads:

[assistant] [bash] cd /home/theuser/gw/test-cluster && ./start.sh /data/fgw2
========================================
FGW Test Cluster (2 Storage Nodes)
========================================

Data directory: /data/fgw2

Architecture:
  - S3 Proxy: S3 API (:8078) - routes to Kuri nodes
  - kuri-1: LocalWeb (:7001) + Web UI (:9010)
  - kuri-2: LocalWeb (:7002)
  - YugabyteDB: Shared metadata

✅ Docker image fgw:local exists

📁 Initializing data directories...
Initializing test cluster data directories in /data/fgw2
Setting permissions (may require sudo for existing...

This article examines why this message was written, what assumptions and decisions led to it, and what the assistant learned from the preceding failures.

The Context: A Desperate Retreat from Host Networking

To understand why this start.sh invocation matters, one must look at the twenty messages that precede it. The assistant had been chasing a performance bottleneck in the test cluster's Docker networking. The original bridge networking mode required port mappings and introduced a layer of NAT that the assistant suspected was limiting throughput for the S3 load tests. In an attempt to eliminate this overhead, the assistant switched the entire Docker Compose configuration to network_mode: host, which binds containers directly to the host's network stack.

This decision was motivated by a legitimate engineering concern: if the S3 proxy and Kuri storage nodes could communicate without Docker's bridge network translation, latency would decrease and throughput would increase. The assistant's reasoning was sound in isolation—host networking is a well-known technique for reducing network overhead in performance-sensitive Docker deployments.

However, the host networking experiment quickly unraveled. The first sign of trouble came when the YugabyteDB container's default YSQL port (5433) conflicted with a PostgreSQL instance already running on the host. The assistant worked around this by changing the YSQL port to 25433. But this was just the beginning. Next, the Kuri storage nodes crashed because their IPFS gateway tried to bind to port 8080 on the host, which was already occupied by another service. The error message was telling: Error: serveHTTPGateway: manet.Listen(/ip4/127.0.0.1/tcp/8080) failed: listen tcp4 127.0.0.1:8080: bind: address already in use.

At this point, the assistant faced a choice: continue patching port conflicts one by one, or abandon host networking entirely. The assistant chose the latter, running git checkout test-cluster/docker-compose.yml test-cluster/gen-config.sh to revert both files to their bridge-networking versions. This reversion is the immediate precursor to message 1319.

What the Start Script Actually Does

The start.sh script is a convenience wrapper that orchestrates the full cluster lifecycle. It calls gen-config.sh to generate per-node configuration files (settings.env with environment variables for each Kuri node), initializes data directories under /data/fgw2, sets file permissions, and then invokes docker compose up -d to launch the services in detached mode. The architecture it prints is the three-layer design: an S3 frontend proxy on port 8078 that routes requests to Kuri storage nodes, which in turn store metadata in a shared YugabyteDB cluster.

The fact that the script reports "✅ Docker image fgw:local exists" is significant—it confirms that the Docker image built earlier in the session is still available and doesn't need to be rebuilt. This saves time and indicates that the fundamental code artifacts are stable, even if the runtime configuration is in flux.

Assumptions Embedded in This Message

Several assumptions are baked into this seemingly straightforward command:

First, the assumption that reverting to bridge networking will resolve the immediate startup failures. The assistant had just cleaned the data directories (rm -rf /data/yugabyte/* /data/kuri-1/* /data/kuri-2/*) and regenerated configuration files. The expectation was that a clean slate plus the known-good bridge networking configuration would produce a working cluster. This assumption turned out to be partially correct—the port conflicts disappeared—but new problems emerged, as the subsequent messages reveal.

Second, the assumption that the gen-config.sh script (reverted from its host-network-modified version) still produces correct configuration for the current codebase. The assistant had previously modified gen-config.sh to add settings like RIBS_RETRIEVALBLE_REPAIR_THRESHOLD and semicolons in the kuri init command. When reverting via git checkout, those fixes were lost. The very next message (1320) shows kuri-1 failing with RetrievableRepairThreshold greater than MinimumReplicaCount: 3 > 1, proving that the reverted gen-config.sh was missing critical configuration parameters.

Third, the assumption that cleaning data directories is sufficient to reset the database state. The assistant removed /data/fgw2/yugabyte/* and /data/fgw2/kuri-*/*, but the YugabyteDB container's data volume persisted because it was managed separately. This led to "dirty migration" errors where the schema_migrations table in YugabyteDB still recorded previous migration attempts, causing the Kuri nodes to fail when they tried to re-run migrations against tables that already existed.

The Mistakes and Incorrect Assumptions

The most significant mistake was underestimating the complexity of reverting a multi-service Docker Compose configuration. When the assistant ran git checkout on docker-compose.yml and gen-config.sh, it assumed that the Git-tracked versions of these files represented a known-good state. But the session had accumulated numerous uncommitted changes and intermediate fixes that were not captured in the Git index. The reverted gen-config.sh was missing the RIBS_RETRIEVALBLE_REPAIR_THRESHOLD setting, the semicolon fix for the kuri init command, and other adjustments that had been necessary to make the cluster work in earlier iterations.

Another mistake was the incomplete data cleanup. The assistant correctly removed the Kuri data directories and the YugabyteDB file storage, but the YugabyteDB container's internal CQL tables (including schema_migrations) were stored in a Docker volume that persisted across container restarts. When the Kuri nodes started fresh and tried to run their migration logic, they found the schema_migrations table already populated with entries from previous runs. The migration framework's "dirty" flag was set, causing the nodes to refuse to start. This required manual CQL intervention (UPDATE filecoingw_s3.schema_migrations SET dirty = false) in subsequent messages.

Input Knowledge Required

To understand this message, a reader needs to know:

Output Knowledge Created

This message produces a running test cluster (or at least the attempt to start one). The output is visible in the script's progress messages: data directories are initialized, permissions are set, and the Docker Compose services are launched. However, the message itself does not show the final result—it cuts off at "Setting permissions (may require sudo for existing..." because the script is still executing. The reader must look at subsequent messages to learn that the cluster failed to start properly due to the configuration and migration issues described above.

More importantly, this message creates knowledge about the system's fragility. It demonstrates that the test cluster is not yet robust enough to survive a full teardown-and-restart cycle without manual intervention. The assistant learns that reverting to a "known-good" Git state is insufficient because the actual working configuration includes uncommitted fixes. This insight will drive the assistant's subsequent behavior: rather than blindly reverting, the assistant will need to carefully re-apply the missing configuration parameters and manually clean the database state.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the sequence of actions leading up to this message. The decision tree is clear:

  1. Problem identification: Host networking causes port conflicts (8080 for IPFS gateway, 5433 for YSQL).
  2. Trade-off analysis: Continue patching individual ports vs. revert to bridge networking. The assistant judges that the number of conflicts is too large and the fixes too fragile.
  3. Decision: Revert to bridge networking, which is known to work (it was the original configuration).
  4. Execution: Clean data, regenerate config, start cluster. The reasoning is pragmatic but incomplete. The assistant correctly identifies that host networking introduces too many conflicts, but fails to account for the fact that the bridge networking configuration in Git is not the same as the configuration that was actually working before the host networking experiment began. The assistant treats "revert to Git" as equivalent to "revert to last known good state," which is a subtle but critical error.

Conclusion

Message 1319 is a turning point in the debugging session. It represents the moment when the assistant acknowledges that an experimental optimization (host networking) has failed and retreats to a more conservative configuration. But the retreat is not clean—it carries forward the scars of the experiment in the form of missing configuration parameters and a dirty database state. The subsequent messages will show the assistant painstakingly fixing each of these issues, learning the hard way that in complex distributed systems, "reverting" is never as simple as checking out a file from Git. The real lesson is that configuration drift accumulates in uncommitted changes, in database state, and in the subtle interactions between components—and that a truly robust test cluster must be able to survive a full reset without manual intervention.