From Host Network Collapse to Clean Working State: A Distributed S3 Test Cluster's Debugging Odyssey

Introduction

In the course of building a horizontally scalable S3-compatible storage architecture, a single coding session can traverse the full spectrum of engineering experience: from confident architectural decisions to cascading failures, from strategic retreats to methodical root-cause analysis, and finally to the quiet satisfaction of a clean working state. This article examines a segment of such a session — spanning over 200 messages — that captures a complete debugging arc in a distributed systems project built on Kuri storage nodes, YugabyteDB, and a stateless S3 proxy layer.

What began as an attempt to eliminate a Docker networking bottleneck by switching to host network mode unraveled into a cascade of port conflicts, forcing a revert to bridge networking. From that reversion, the session rebuilt toward stability: committing a CQL batcher for high-throughput metadata writes, fixing loadtest diagnostics, discovering and resolving missing configuration variables, and ultimately achieving a clean working single-node cluster. The arc of this session — from over-ambitious optimization to disciplined stabilization — offers a rich case study in how experienced engineers navigate the tension between performance goals and operational reality.

The Opening Gambit: Host Network Mode and Its Hidden Costs

The session's inflection point came when the assistant and user confronted a performance bottleneck. Load tests at high concurrency (100 and 1000 workers) had revealed a troubling pattern: "connection reset by peer" errors that masqueraded as data corruption. The user offered a sharp diagnosis: "Might be docker-proxy issues?" Docker's bridge networking routes traffic through a user-space proxy process (docker-proxy) that can become saturated under high connection counts, introducing latency and connection overhead.

The decision was made to switch the entire test cluster to network_mode: host, which binds containers directly to the host's network interfaces, bypassing Docker's proxy entirely. The assistant rewrote docker-compose.yml, updated gen-config.sh, and modified the README — a significant architectural shift touching every service in the cluster.

But the host network experiment failed catastrophically. When the assistant attempted to restart the cluster, YugabyteDB failed its health check because ports 7000 and 7100 — YugabyteDB's internal communication ports — were already occupied by existing services on the host machine. The assistant discovered this through systematic investigation: running ss -tlnp | grep -E '(7000|7100)' revealed that both ports were bound to 127.0.0.1 by pre-existing processes.

The port conflicts did not stop there. The IPFS gateway embedded in each Kuri node defaulted to port 8080, which was also occupied on the host. The assistant attempted to find configuration options to change the IPFS gateway port, searching the codebase with grep -r "Addresses.Gateway" — but this search returned nothing useful, revealing that the port was buried in the IPFS configuration file inside the container, not exposed through environment variables.

The Strategic Retreat: Reverting to Bridge Networking

After approximately forty messages of debugging port conflicts, dirty migration states, and container startup failures, the assistant reached a critical decision point. The moment of strategic re-evaluation came: "Actually, a cleaner approach is to go back to bridge network mode but configure the S3 proxy to have more connections/less bottleneck. The host network mode is causing too many port conflicts."

This "Actually" is the hinge of the entire session — a shift from tactical thinking ("how do I fix this specific port conflict?") to strategic thinking ("is this whole approach worth the trouble?"). The assistant recognized that the host network experiment was generating an unacceptable amount of friction. Every new container or service that joined the cluster could introduce another port conflict. The IPFS gateway port was just the latest discovery — there could be more.

The revert was executed cleanly: git checkout test-cluster/docker-compose.yml test-cluster/gen-config.sh restored the original bridge-mode configuration files from the repository index. The assistant then stopped the cluster, wiped all data directories, and regenerated configuration files for the now-reverted bridge-mode setup. This was a classic "return to known good" strategy — eliminating all the host-network changes in one atomic operation and ensuring a clean foundation.

The Commit: Consolidating Hard-Won Improvements

Before the user asked the pivotal question about single-node mode, the assistant had accumulated a significant set of uncommitted changes. The user's directive was a single word: "commit."

The resulting commit bundled three categories of changes into a coherent unit:

The CQL Batcher (database/cqldb/batcher.go): A high-throughput batching system for YCQL (YugabyteDB's Cassandra-compatible API) writes. It collects writes from multiple goroutines and flushes when a batch size of 15,000 is reached, an idle timeout of 10ms fires, or a max latency of 30ms is exceeded. An 8-worker pool with exponential backoff retry handles concurrent batch submissions. Critically, Submit() blocks until the batch is committed, ensuring read-after-write consistency — a fundamental requirement for S3 semantics.
Loadtest improvements: The loadtest tool was enhanced to distinguish between verifyErrors (actual checksum mismatches representing real data corruption) and verifyTimeouts (context cancellations or deadline exceedances that occur when the test is ending, not corruption). This distinction had been the subject of significant earlier debugging effort, where what appeared to be data corruption was actually Docker proxy connection resets.
Test-cluster configuration fixes: The RIBS_RETRIEVALBLE_REPAIR_THRESHOLD environment variable was added to gen-config.sh to fix a startup error ("RetrievableRepairThreshold greater than MinimumReplicaCount: 3 > 1"). The kuri init command was changed from && to ; so that the daemon starts even if initialization fails on container restart — a subtle but critical fix for container lifecycle reliability. The commit message itself was a model of technical communication: structured, specific, and explanatory. It documented not just what changed but why — the batch size rationale, the timeout values, the distinction between error types. As one analysis noted, "the commit message is not just a log entry; it is a narrative device. It tells the story of what was learned and what was built."

The Question That Unraveled Everything: "Does ./docker-compose.yml still work for single-node mode?"

With the commit complete and the multi-node test cluster verified working, the user asked a seemingly simple question: "Does ./docker-compose.yml still work for single-node mode?" This seven-word question triggered a chain of discoveries that would consume the next 35 messages.

The question operated on multiple levels simultaneously. On the surface, it was a regression check — had any of the recent changes broken the simpler single-node configuration? But it also revealed the user's architectural priorities: single-node mode had genuine value for development, testing, and demos, and the user wanted assurance that it remained functional.

The assistant's response chain reveals a methodical diagnostic process. First, it inspected the root docker-compose.yml and noted that it already used the semicolon operator (; ./kuri daemon) that had been a recent fix for the test cluster — an optimistic sign. But when the assistant ran docker compose config --quiet to validate the configuration, it received a fatal error: env file /home/theuser/gw/data/config/settings.env not found.

The root docker-compose.yml referenced a settings.env file that did not exist. The data/ directory was empty. This was a gap that had likely existed since the project's inception — the single-node configuration had never been fully set up, or the settings file had been lost during earlier data resets.

Building a Configuration from First Principles

The assistant now faced a knowledge problem: what should this settings.env contain? The answer came from an existing configuration. The multi-node test cluster, which had been debugged and verified earlier in the session, had per-node settings files stored at /data/fgw2/config/kuri-1/settings.env. The assistant read this file to understand the required environment variables and their values.

This was a critical moment of reasoning. Rather than guessing or asking the user for the configuration, the assistant recognized that the test-cluster's working configuration could serve as a template for the single-node setup. The resulting settings.env file contained core settings (data directory, group count, Filecoin API endpoint), database settings (YugabyteDB connection details for both CQL and SQL interfaces), S3 API settings (bind address, authentication disabled), LocalWeb settings, deal settings, and balance manager settings (disabled for test mode).

The assistant made several assumptions in constructing this file. It assumed that the same YugabyteDB connection parameters would work for single-node mode. It assumed that the Filecoin API endpoint was appropriate for testing. It assumed that disabling authentication and the balance manager was acceptable for a test environment. These were reasonable assumptions based on the existing working configuration, but they represented knowledge that was implicitly transferred from the multi-node setup to the single-node one.

With the settings file created, the assistant ran docker compose config --quiet again and received the desired response: "Config valid." The YAML syntax was correct, the file references resolved, and the Docker Compose configuration was structurally sound. But as the next discovery would show, configuration validation is not the same as correctness.

The First Obstacle: Missing EXTERNAL_LOCALWEB_URL

When the assistant started the single-node cluster with the newly created settings.env, the container launched but immediately failed with a cryptic error. The error log showed a dependency injection failure deep in the application's startup chain, ending with a reference to "external offload module."

The assistant's response — "There's a missing config for 'external offload module'" — was remarkable for how much implicit knowledge it encoded. The error message itself did not mention "external offload module." The assistant was interpreting the dependency injection failure through its understanding of the codebase architecture, knowing that the S3 server depends on a block store, which in turn depends on external storage modules for offloading data.

What followed was a methodical code trace. The assistant used grep to search for external.*offload|OFFLOAD|XYZ across the codebase, finding 59 matches. The "XYZ" prefix was a deliberate debugging marker left in the code by the original developer — a distinctive string that made error paths easily searchable. The grep results pointed to four key files: rbstor/rbs.go (storage layer), rbdeal/ribs.go (initialization entry point), rbdeal/group_deal.go (deal lifecycle), and rbdeal/external_s3.go (S3-specific external module).

Reading the source code in rbdeal/ribs.go, the assistant discovered that initExternal() iterates over registered ExternalOffloader modules, including S3OffloadInfo and LocalWebInfo. The LocalWebInfo module's maybeInitExternal() function required EXTERNAL_LOCALWEB_URL to be set. The single-node configuration had EXTERNAL_LOCALWEB_PATH and other LocalWeb settings, but was missing the URL variable entirely.

The fix required adding several LocalWeb-related variables to the settings file:

export EXTERNAL_LOCALWEB_PATH="/root/.ribsdata/cardata"
export EXTERNAL_LOCALWEB_URL="http://localhost:8443"
export EXTERNAL_LOCALWEB_BUILTIN_SERVER="true"
export EXTERNAL_LOCALWEB_SERVER_TLS="false"
export EXTERNAL_LOCALWEB_SERVER_PORT="8443"

While EXTERNAL_LOCALWEB_URL was the critical missing piece — the one that caused the initialization failure — the assistant added several related variables to create a complete, working configuration rather than patching one gap at a time.

After recreating the container with docker compose up -d --force-recreate filecoingw, the container started successfully. The assistant ran a quick sanity test with the loadtest, and the container appeared to be operational. But a new problem immediately surfaced.

The Second Obstacle: Invalid Content SHA256

With the Kuri node now running, the assistant ran the loadtest to verify end-to-end functionality. The loadtest immediately produced write errors, and the server logs revealed the cause: every PUT request was being rejected with "invalid content sha256:".

The assistant's diagnosis was swift and precise. "This is a different issue," the assistant noted — immediately recognizing that this was a protocol-level rejection rather than a configuration failure. The key insight was the comparison with the test-cluster: "The test-cluster proxy doesn't have this check, but the kuri storage node does."

This revealed a fundamental architectural fact about the system. The multi-node test cluster used a three-layer architecture: an S3 frontend proxy (stateless) that sat in front of Kuri storage nodes. The proxy automatically added the x-amz-content-sha256 header required by the storage node's S3 handler. But the single-node docker-compose.yml exposed the Kuri storage node directly on port 8078, bypassing the proxy entirely. The loadtest, which used raw HTTP calls rather than the AWS SDK, never sent this header.

The assistant faced a design decision with two viable paths. Option 1 was to add a proxy layer to the single-node Docker Compose, making it architecturally consistent with the multi-node test cluster but adding complexity. Option 2 was to fix the loadtest to send the x-amz-content-sha256 header, making it compatible with any S3-compatible endpoint. The assistant chose Option 2, reasoning that "the loadtest should work with any S3-compatible endpoint."

The fix required adding x-amz-content-sha256: UNSIGNED-PAYLOAD to the loadtest's HTTP PUT requests. The UNSIGNED-PAYLOAD sentinel value tells the S3 server that the client is not providing a content hash and the server should skip body verification — the simplest valid value that avoids the complexity of computing actual SHA256 hashes.

The Amend: Preserving a Clean Git History

With the loadtest fix verified — the assistant ran the test with "Now let me test again" — the assistant faced a git workflow decision. Rather than creating a new commit for the loadtest header fix, the assistant chose to amend the previous commit:

cd /home/theuser/gw && git add -f integrations/ritool/loadtest.go && git commit --amend --no-edit

This decision revealed several layers of reasoning. The fix was small and directly related to the changes already in the commit — the previous commit included "loadtest improvements," and the content-SHA256 header fix was another loadtest improvement, closely related in both scope and purpose. The commit had not yet been pushed to the remote, so amending before pushing avoided the complications of rewriting published history. And the amendment preserved the commit's identity as the "batcher and loadtest improvements" commit, keeping the logical grouping intact.

The assistant then cleaned up by stopping the single-node cluster with docker compose down and checking git status. The status showed that only test-cluster/README.md remained as a modified but uncommitted file, along with some untracked artifacts (a screenshot and the data/ and s3-proxy directories). The core code changes were all committed.

The Summary: Closing the Loop

The session's final summary captured the entire arc in 67 words:

Single-node ./docker-compose.yml now works: - Created data/config/settings.env with required config (including EXTERNAL_LOCALWEB_URL) - Fixed loadtest to send x-amz-content-sha256: UNSIGNED-PAYLOAD header

Committed changes: - CQL batcher for high-throughput S3 metadata writes - Loadtest improvements (timeout vs corruption distinction + x-amz-content-sha256 header) - Test-cluster fixes (RIBS_RETRIEVALBLE_REPAIR_THRESHOLD, init command semicolon)

This summary, as one analysis noted, "is not just a report of what happened — it is an act of curation, selecting what matters and organizing it into a narrative that the reader can act on." It organized the chaos of 35 messages of investigation, four distinct bugs found and fixed, and three layers of configuration depth into a coherent narrative that the user could immediately understand and act upon.

Lessons from the Arc

This session offers several enduring lessons for distributed systems engineering.

First, performance optimizations should be validated early and reverted quickly if they introduce instability. The host network mode experiment lasted only a handful of messages, but it generated a disproportionate amount of debugging work. The ability to recognize when a strategy is failing and pivot to a simpler approach is more valuable than the ability to force a problematic strategy to work.

Second, configuration validation is not the same as correctness. The assistant ran docker compose config --quiet which validated YAML syntax and file references, but this check could not verify that the runtime environment had all required configuration values. Configuration validation tools check format, not semantic completeness.

Third, the most powerful debugging tool is often not a complex debugger but a simple text search executed with precision and intent. The assistant's grep command for external.*offload|OFFLOAD|XYZ transformed an opaque container startup failure into a navigable map of the codebase's external offload system, turning "I don't know what's wrong" into "I know exactly which four files to read next."

Fourth, default port bindings are a hidden tax on containerized applications. Every service that binds to a default port is a potential conflict waiting to happen, especially when host networking collapses the isolation that bridge networking provides.

Fifth, a well-crafted commit message is a narrative device that tells the story of what was learned and what was built. The assistant's commit message — with structured bullet points, specific parameters, and explanatory rationale — serves as documentation for future developers who might wonder why a particular configuration value was chosen or why a specific code change was made.

Sixth, the ability to recognize when a working configuration can serve as a template for another is a form of architectural reasoning that accelerates debugging. Rather than reconstructing configuration from documentation or source code, the assistant leveraged the existing test-cluster configuration as a proven reference point.

Conclusion

The arc of this session — from the confident decision to adopt host networking, through the cascade of port conflicts, the strategic retreat to bridge networking, the commit of hard-won improvements, and the methodical reconstruction of the single-node configuration — is a microcosm of distributed systems engineering. It demonstrates that progress is not always linear, that the best optimizations are sometimes the ones you don't make, and that the quiet moments of verification — a docker compose config --quiet that returns "Config valid," a loadtest that produces zero errors — are the true milestones of engineering progress.

In the end, the session achieved its goal: a clean working single-node cluster, a committed set of improvements, and a deeper understanding of the system's architecture. The host network experiment was not wasted effort — it confirmed that bridge networking was not the sole cause of connection resets, revealed environmental constraints that will inform future deployment decisions, and produced a more robust configuration that accounts for the realities of a shared development environment. Sometimes the most important optimization is knowing when to stop optimizing and start shipping.

References

[1] "From Host Network Collapse to Clean Working State: A Distributed S3 Test Cluster's Debugging Odyssey" — The chunk article covering this segment's debugging arc, including analysis of the host network revert, the CQL batcher commit, and the single-node configuration fixes.## Deeper Dive: The Code Trace That Uncovered the Missing Configuration

The methodical code trace that uncovered the EXTERNAL_LOCALWEB_URL dependency deserves closer examination, as it exemplifies the kind of systematic debugging that distinguishes experienced engineers. When the container failed with the cryptic "XYZ: trying to initialize external offload" error, the assistant did not simply search for the error message and hope for a configuration snippet. Instead, the assistant constructed a search pattern — external.*offload|OFFLOAD|XYZ — that would capture multiple dimensions of the problem simultaneously.

The "XYZ" prefix was a particularly clever debugging convention. By prefixing certain error paths with a distinctive, easily searchable string, the original developer had created a breadcrumb trail through the codebase. A search for "XYZ" alone would have returned results, but the assistant's compound pattern ensured that even if the XYZ convention was not used everywhere, the broader "external.*offload" pattern would catch related code paths.

The 59 matches were narrowed to four critical files by reading the context around each match. The rbdeal/ribs.go file contained the initExternal() function that was the entry point for the failure. The rbdeal/external.go file showed the list of registered modules — S3OffloadInfo and LocalWebInfo — that initExternal() iterated over. The rbdeal/external_http_path.go file contained the LocalWebInfo implementation, and specifically the maybeInitExternal() function that checked for EXTERNAL_LOCALWEB_URL.

Reading the source code directly — not just the error messages — was the key insight. The error message said "trying to initialize external offload," but the source code revealed which external module was failing and why. The cfg.External.Localweb.Url field was being read from configuration, and if it was empty, the module would fail to initialize. The configuration system used environment variable mapping, so EXTERNAL_LOCALWEB_URL was the environment variable that mapped to this configuration field.

This tracing technique — from error message to entry point, from entry point to module list, from module list to specific module implementation, and from implementation to required configuration — is a generalizable skill for debugging configuration failures in any system that uses dependency injection or modular initialization.

The Architectural Insight: Proxy vs. Direct Storage Node

The "invalid content sha256" error revealed an architectural subtlety that had significant implications for the project's deployment model. The multi-node test cluster used a three-layer architecture: clients connected to an S3 frontend proxy, which in turn forwarded requests to Kuri storage nodes. The proxy was responsible for adding S3-specific headers, handling authentication, and routing requests to the appropriate storage node.

The single-node docker-compose.yml, however, exposed the Kuri storage node directly. This was not a bug but a design choice — for development and testing, a direct connection to the storage node was simpler and eliminated the proxy as a potential point of failure. But this design choice had a hidden dependency: the storage node's S3 handler required the x-amz-content-sha256 header, which the proxy normally supplied.

The assistant's decision to fix the loadtest rather than add a proxy to the single-node configuration was architecturally significant. It meant that the loadtest would be compatible with any S3-compatible endpoint, not just those fronted by the project's proxy. This made the loadtest a more general-purpose tool that could be used for benchmarking against other S3 implementations or for testing storage nodes directly in production-like scenarios.

The fix itself — adding x-amz-content-sha256: UNSIGNED-PAYLOAD — was minimal but correct. The UNSIGNED-PAYLOAD sentinel is a standard S3 convention that tells the server the client is not providing a content hash. This is appropriate for a loadtest where computing SHA256 hashes for every request would add overhead and reduce throughput. The S3 specification allows this value, and the storage node's handler explicitly checked for it in the getBodyReader() function.

The Git Workflow: Amending as a Deliberate Choice

The assistant's decision to amend the previous commit rather than create a new one was a deliberate workflow choice that reveals an understanding of git's capabilities and limitations. Amending a commit that has not been pushed to a remote is safe — it rewrites local history but does not affect any other developer's work. The amendment preserved the logical grouping of the loadtest changes, keeping the commit history clean and meaningful.

The command git add -f integrations/ritool/loadtest.go && git commit --amend --no-edit used the -f flag to force-add the file, which was necessary because the file was already tracked and the assistant wanted to ensure the latest changes were included. The --no-edit flag reused the existing commit message, which already documented the loadtest improvements. This was appropriate because the header fix was a refinement of the loadtest improvements already described in the commit message.

The amendment also meant that the commit hash changed, but since the commit had not been pushed, this had no downstream impact. The assistant's workflow was disciplined: commit early, amend locally, push only when the commit is final. This pattern is common in experienced git workflows and avoids the clutter of many small "fixup" commits that obscure the logical structure of changes.

The Final State: A Clean Working Cluster

The session ended with a clean working state. The git status output showed only test-cluster/README.md as modified but uncommitted — a documentation update that was intentionally left for a separate commit. The untracked files (data/, s3-proxy, and a screenshot) were artifacts of the testing process, not code changes.

The single-node cluster had been verified working with a loadtest that produced zero errors. The multi-node test cluster had been verified earlier in the session. The CQL batcher was committed and integrated into the S3 ObjectIndex. The loadtest could distinguish between timeouts and actual corruption. And the configuration files for both single-node and multi-node deployments were correct and complete.

This final state represented more than just a working system — it represented a deeper understanding of the architecture. The host network experiment, though it failed, had revealed environmental constraints that would inform future deployment decisions. The single-node configuration exercise had uncovered missing dependencies and architectural assumptions that were now documented and resolved. The commit had consolidated weeks of incremental improvements into a coherent, well-documented unit of work.

In distributed systems engineering, a clean working state is always temporary — new features, new bugs, and new requirements will inevitably disturb it. But the discipline of achieving that state, documenting what was learned, and committing the result is what separates professional engineering from hacking. This session was a masterclass in that discipline.