From Dirty Migrations to Distributed Traffic: The Complete Arc of a QA Cluster Deployment

Introduction

In the life of any distributed systems engineer, there are sessions that feel like a single, continuous debugging loop—a stretch of hours where every success is immediately followed by a new failure, and every fix reveals a deeper architectural assumption that had gone unexamined. This article examines one such session: the deployment and validation of a QA test cluster for the Filecoin Gateway (FGW) distributed storage system across three physical nodes. Over the course of dozens of messages spanning multiple hours, the assistant and user navigated a cascade of interconnected problems—dirty database migration states, empty cluster topology APIs, missing environment variables, cross-node read failures, a missing proxy layer, a question about infrastructure automation, and even a silent codebase regression. Each problem was a thread in a larger tapestry, and only by pulling each thread could the cluster finally function as designed.

This article synthesizes the major themes of that session: the dirty migration that blocked service startup, the topology that wouldn't render, the cross-node read architecture gap that demanded a proxy layer, the pivot from ad-hoc SSH to Ansible automation, the accidental code deletion that threatened milestone integrity, and the final validation that both nodes were receiving traffic through the proxy. Together, these episodes tell a story about the nature of distributed systems engineering—where the difference between a working cluster and a broken one is often a single boolean flag, a missing environment variable, or a question asked at the right moment.

The Dirty Migration: When the Database Refuses to Let Go

The session began with what should have been a routine deployment. The assistant had installed YugabyteDB, built the kuri binaries from source, created systemd service files with security hardening, and initialized the database schemas via kuri init. But when the time came to start the kuri daemon as a persistent service, it failed immediately with an exit code of 1 [1].

The failure was traced to a condition known as a "dirty migration" in the YugabyteDB CQL keyspaces. The schema_migrations table—a standard pattern used by database migration frameworks to track which schema versions have been applied—contained a row with dirty = true for migration version 1769890615. This flag signals that a migration was interrupted or failed partway through, and most migration-aware applications refuse to proceed until the flag is cleared, to prevent inconsistent schema states [8].

The root cause was that the test suite had left these dirty flags behind. This is a classic artifact of development and testing workflows: test harnesses often perform schema migrations as part of setup and teardown, and if a test is interrupted or crashes, it can leave the migration state in an ambiguous condition. The dirty flag is a safety mechanism, but in this case it became a deployment blocker [2].

The assistant diagnosed this by checking the logs on kuri1 (journalctl -u kuri), which showed the daemon initializing but failing to proceed past schema validation. Then, using a Python script with the Cassandra driver, the assistant queried the schema_migrations table directly and confirmed the dirty flag. The fix was straightforward: update the flag to false for all affected keyspaces—filecoingw_kuri_01, filecoingw_kuri_02, and the shared S3 keyspace filecoingw_s3 [5].

A notable mistake during this process was the initial failure to check all three keyspaces. The assistant first fixed only filecoingw_kuri_01 and filecoingw_kuri_02, restarted the service, and it still failed. Only then did the assistant check filecoingw_s3 and find the dirty flag there too. This is a classic debugging oversight—fixing the most obvious problem without considering that the same condition might exist in related components [8].

Once all three keyspaces were corrected, both kuri nodes came online successfully. Message 2007 shows kuri1 running with 70.3 MB of memory consumed, 35 tasks, and a clean Active: active (running) status. Message 2008 shows kuri2 following suit with the same clean startup [9]. The cluster had its storage layer operational.

The Topology That Wouldn't Render: A Missing Environment Variable

With both storage nodes running, the next task was to verify the cluster topology. The kuri daemon exposes a Web UI on port 9010 that includes a cluster topology visualization—a React-based dashboard showing the distributed storage cluster's nodes, their health status, and data flow relationships. But when the user checked this interface, the topology was blank [32].

The user's bug report was a model of brevity: ":9010 cluster topology doesn't render" [32]. Seven words, a port number, and a symptom. No greeting, no stack trace, no screenshot. Yet this single sentence set the agenda for the next dozen assistant actions and exposed a critical configuration gap.

The assistant traced the empty topology to the source code. Reading rbstor/diag.go, the assistant found the ClusterTopology() function and its dependency on FGW_BACKEND_NODES—an environment variable that lists the backend storage nodes with their IDs and URLs. This was the root cause: the environment variable was not set during the initial deployment [37].

The assistant configured the variable on both nodes, adding FGW_BACKEND_NODES="kuri_01:http://10.1.232.83:8079,kuri_02:http://10.1.232.84:8079" to the settings files and restarting the kuri services. The subsequent API call returned a full topology with both nodes showing as healthy [38]. The topology rendered, but this fix was cosmetic—it made the Web UI display correctly without actually enabling cross-node data access. The real architectural problem was still lurking.

The Cross-Node Read Problem: When Architecture Meets Reality

The topology fix made the cluster look correct, but the user's next observation revealed the deeper issue. After running a load test against kuri1's direct S3 endpoint, the user noted: "Only kuri1 getting traffic seems wrong" [56]. This observation was a masterclass in concise, high-impact feedback. It was not a complaint about the load test tool or a request for a configuration tweak. It was an architectural observation that revealed a fundamental gap between the deployed system and the intended design.

The assistant investigated and discovered the root cause: when kuri2 received a GET request for an object written to kuri1, it found the object metadata in the shared filecoingw_s3 keyspace (which recorded node_id='kuri_01'), but it could not retrieve the actual block data. The error message was telling: "block was not found locally (offline): ipld: could not find bafkreicq..." [58]. The S3 server on kuri2 was looking for the block in its local blockstore, not asking kuri1 for it.

The assistant attempted a workaround—connecting the two kuri nodes via IPFS swarm to enable bitswap-based block exchange [65]. This succeeded at the network level but did not fix the S3 reads because the S3 layer does not use bitswap for retrieval; it uses a direct local blockstore lookup. The architecture simply did not support cross-node reads at the S3 level [66].

This was the moment of architectural clarity. The system was designed with a three-layer hierarchy: stateless S3 frontend proxies → kuri storage nodes → YugabyteDB. The kuri nodes were never meant to serve S3 requests directly in a multi-node configuration. The proxy layer was the missing piece [70].

Deploying the S3 Proxy: From Manual SSH to Ansible Automation

The correct solution was to deploy the s3-proxy binary on the head node (10.1.232.82), configured with the list of backend kuri nodes and access to the shared S3 CQL keyspace. The proxy would receive all S3 requests, look up the object's node_id in the shared metadata, and route the request to the correct backend node [74].

What followed was a textbook example of ad-hoc infrastructure deployment. Over the course of several messages, the assistant copied the binary using scp, created configuration files by echoing heredocs through SSH, wrote a systemd service file using sudo tee, and started the service—which immediately failed because the environment variable names in the config file didn't match what the binary expected [77][78].

The assistant then discovered, through grep, that the s3-proxy binary expected FGW_YCQL_HOSTS rather than RIBS_S3_CQL_HOSTS. The manual approach had introduced a naming mismatch that would have been caught by the existing Ansible templating system [79].

At this point, the user interjected with a question that changed the trajectory of the entire deployment: "Why are you not doing this with ansible?" [80]. These eight words were a masterclass in effective engineering communication. The question was not a criticism or a technical correction—it was a pure, almost Socratic inquiry that forced a reckoning with a specific kind of engineering failure: the failure to use existing abstractions.

The assistant's response was immediate and unambiguous: "You're right - I should be using ansible properly instead of manually SSHing and creating ad-hoc configs." [81]. The assistant stopped the manual work, read the existing s3_frontend role, examined the Jinja2 templates, and updated the QA inventory to add the head node to the s3_frontend group [85]. The playbook was run, and the s3-proxy was deployed correctly, with the right environment variable names, proper systemd integration, and the full audit trail that infrastructure-as-code provides [86][87].

This pivot was critical. The Ansible roles for the s3_frontend already existed—complete with Jinja2 templates for configuration files, systemd service definitions, and proper variable management. The QA inventory file already had the host groups defined. Everything needed to deploy the s3-proxy through automation was already in place. The assistant had fallen into a common trap: the belief that a "quick" manual deployment would be faster than updating the inventory and running a playbook. But the manual approach had already consumed multiple message turns and produced a broken service [80].

The Moment of Proof: Validating Cross-Node Access

With the s3-proxy deployed via Ansible, the assistant ran a series of curl commands to validate cross-node access. The test script wrote a new object through the proxy, read it back, and then read an older object that had been written directly to kuri1 before the proxy existed. All three operations succeeded [89].

The most important test was the third: reading test.txt (written directly to kuri1) through the proxy at http://10.1.232.82:8078. The proxy returned test-data-123—the correct content. This confirmed that the proxy could discover and retrieve objects regardless of their origin, using the shared metadata in the CQL keyspace [89].

The assistant then escalated to a load test, running ritool loadtest with 10 concurrent workers, 30-second duration, and object sizes between 1KB and 100KB against the proxy endpoint. The test completed successfully with approximately 345 writes per second and 147 reads per second, with zero data corruption—all Read-After-Write verifications passed [90].

The final verification was to confirm that both backend nodes actually received traffic. The assistant SSHed into each node and ran journalctl -u kuri --since '2 min ago' | grep -c 'PUT\|DELETE'. Both nodes returned exactly 3 operations each [91]. This was the empirical confirmation that the proxy was distributing traffic correctly across the multi-node cluster. The architecture was working as designed.

The Silent Regression: When Code Disappears Without Warning

Amidst all the deployment and debugging work, a different kind of crisis emerged. The user noticed something alarming in the git working tree: code related to garbage collection and caching was being removed. The user asked: "Why is some gc/cache code removed in git diff??" [19].

The assistant investigated and discovered that three files had been modified in the working directory, with 397 lines deleted. The assistant had inadvertently removed the GC configuration struct and cache integration code from configuration/config.go, stripped the multi-tier cache import and usage from rbdeal/retr_provider.go, and gutted the GC integration from rbdeal/claim_extender.go [19]. These were not cosmetic changes—they removed the core integration points that made the caching and garbage collection systems functional.

How did this happen? The most likely scenario is that the assistant, while working on the QA deployment configuration, made changes to these files as part of debugging or testing and inadvertently saved modifications. The configuration/config.go file, in particular, contains the RibsConfig struct that controls all system settings—it would be a natural file to touch when adjusting environment variables or deployment parameters. A stray keystroke, an accidental save, or a misapplied search-and-replace could have introduced these deletions without the assistant realizing it [19].

This is a classic version control hazard. When working under pressure—deploying to physical nodes, debugging service failures, managing SSH sessions across three machines—it is easy to make unintended modifications to source files. The assistant was operating in a mode of rapid iteration, running commands, checking logs, and fixing issues as they appeared. The cognitive load of managing the deployment likely overshadowed awareness of the working tree state [19].

The assistant restored the files with git checkout -- configuration/config.go rbdeal/claim_extender.go rbdeal/retr_provider.go and verified the working tree was clean. The code was back to the properly committed state, but the incident served as a powerful reminder that working tree integrity needs active monitoring during long sessions [30][31].

The Verification That Failed: A CQL Query Reveals Database Boundaries

After the load test confirmed both nodes were receiving traffic, the assistant attempted to verify object distribution by querying the shared CQL keyspace directly. The query was straightforward: SELECT node_id, count(*) as cnt FROM s3objects GROUP BY node_id ALLOW FILTERING. But it failed with a SyntaxException because YugabyteDB's YCQL does not support GROUP BY [92].

This failure was far more revealing than any success could have been. It highlighted a fundamental tension in YugabyteDB's design: the YCQL compatibility layer implements the Cassandra wire protocol and query language, but some Cassandra features—like GROUP BY—are not supported by default. The error message itself was informative: "Feature Not Supported. This is not recommended but this error can be suppressed by setting ycql_suppress_group_by_error flag to false." [92]

The assistant's assumption that CQL would support standard aggregation syntax was reasonable—that's the entire point of a compatibility layer. But compatibility layers are never perfect, and GROUP BY is a complex analytical operation that different databases implement differently. The failure wasn't in the storage system itself—the s3-proxy was routing correctly, both nodes were storing data, and the load test passed. The failure was in the verification of the system, which is a fundamentally different problem from building it [92].

Themes and Lessons

Several themes emerge from this session that are broadly applicable to distributed systems engineering.

The hidden cost of test residue. The dirty migration flags were left behind by a test suite that had run earlier. This is a classic "test residue" problem—the test environment's database state doesn't get fully cleaned up, and when you try to deploy for real, the leftover state blocks you. The lesson is that test harnesses must be designed to clean up after themselves, or deployment pipelines must account for the possibility of stale state [2][8].

The gap between cosmetic and functional fixes. The FGW_BACKEND_NODES environment variable made the cluster topology render correctly, but it did not enable cross-node data access. The topology fix was cosmetic—it made the system look correct without making it work correctly. The real fix required the s3-proxy, which was a fundamentally different architectural component. This is a common pattern in distributed systems: making the monitoring work is not the same as making the system work [38][56].

The architecture that was already designed. The three-layer hierarchy (proxy → kuri → database) was specified in the project roadmap, but the assistant initially deployed kuri nodes as direct S3 endpoints. The cross-node read failures were not bugs in the code—they were consequences of deploying the system in a configuration that bypassed its own architecture. The lesson is that architecture documents are not optional reading; they encode design decisions that have real operational consequences [70][74].

The power of a well-timed question. The user's question "Why are you not doing this with ansible?" [80] was more valuable than any technical fix. It realigned the deployment methodology, ensured consistency with the rest of the infrastructure, and reinforced the discipline of infrastructure-as-code. The question worked because the foundation was already laid: the Ansible roles existed, the inventory was structured, and the team had already committed to automation. All that was needed was someone to ask the obvious question at the right moment.

The fragility of long coding sessions. The accidental code deletion incident [19] demonstrated that even experienced engineers can make silent, destructive mistakes when operating under cognitive load. The working tree is a shared resource, and its integrity depends on active monitoring. The user's attentive observation prevented a regression that would have removed critical garbage collection and caching functionality from the codebase.

Verification is its own engineering challenge. The failed CQL GROUP BY query [92] showed that verifying a distributed system requires its own set of skills and knowledge. The database layer has its own quirks and limitations, and the tools used for verification (CQL queries, log greps, API calls) each have their own failure modes. An engineer who can build a working distributed system may still struggle to verify that it is working correctly.

Conclusion

The QA cluster deployment session examined in this article is a microcosm of the challenges inherent in distributed systems engineering. It began with a dirty migration flag that blocked service startup, proceeded through a missing environment variable that broke the topology visualization, encountered a fundamental architectural gap that required a proxy layer to be deployed, navigated a pivot from manual SSH to Ansible automation, survived an accidental code deletion that threatened milestone integrity, and culminated in a successful load test that confirmed both backend nodes were receiving traffic.

Each problem was solvable in isolation, but together they formed a chain of dependencies that demanded sustained attention, systematic debugging, and a willingness to question assumptions. The dirty migration fix required understanding the CQL schema migration system. The topology fix required reading source code to find the environment variable dependency. The cross-node read fix required recognizing that the architecture demanded a proxy layer. The Ansible pivot required acknowledging that manual deployment was the wrong approach. The code restoration required noticing that the working tree had been silently corrupted.

In the end, the cluster worked. Both kuri nodes ran, the s3-proxy routed requests correctly, objects were readable across nodes, and the load test showed balanced traffic. But the journey to that working state was not a straight line—it was a spiral of discovery, where each fix revealed a deeper layer of the system's architecture and each failure taught something about how the system was designed to operate. That is the nature of distributed systems engineering: the work is never just about making things work, but about understanding why they work, and why they sometimes don't.## References

[1] "The Moment of Activation: When Infrastructure Meets Reality" — Message 2000, the initial service start attempt that revealed the dirty migration failure.

[2] "The Moment of Failure: Debugging a Dirty Migration in a Distributed Storage Cluster" — Message 2001, the diagnostic pivot to investigate the service failure.

[5] "The Moment of Incomplete Victory: Debugging a Dirty Migration in a Distributed Storage Cluster" — Message 2004, the discovery that the dirty flag existed in multiple keyspaces.

[8] "The Moment the Daemon Woke: Resolving Dirty Migration State in a Distributed Storage Cluster" — Message 2007, the successful restart of kuri1 after clearing dirty flags.

[9] "The Moment Two Nodes Became One Cluster" — Message 2008, the successful startup of kuri2, completing the two-node storage layer.

[19] "The Six-Word Question That Saved a Codebase" — Message 2018, the user's discovery of uncommitted code deletions in the working tree.

[30] "The Quiet Verification: How a Single git status Confirmed Repository Integrity" — Message 2029, the verification that the working tree was clean after restoration.

[31] "The Silent Reversion: When Uncommitted Changes Threaten Milestone Integrity" — Message 2030, the restoration of GC and cache code via git checkout.

[32] "The Seven-Word Bug Report That Uncovered a Missing Configuration" — Message 2031, the user's report that the cluster topology wasn't rendering.

[37] "The Diagnostic Read That Unlocked Cluster Topology" — Message 2036, the source code analysis that revealed the FGW_BACKEND_NODES dependency.

[38] "The Missing Environment Variable: How a Cluster Topology Bug Revealed Architecture Tensions" — Message 2037, the fix that configured FGW_BACKEND_NODES and restored the topology.

[56] "\"Only kuri1 getting traffic seems wrong\": A User's Observation That Exposed an Architectural Gap" — Message 2055, the user's observation that triggered the cross-node read investigation.

[58] "The Cross-Node Read Problem: Diagnosing a Missing Routing Layer in a Distributed S3 Architecture" — Message 2057, the diagnosis of why kuri2 could not serve objects written to kuri1.

[65] "The Cross-Node Retrieval Problem: A Diagnostic Pivot in Distributed S3 Architecture" — Message 2064, the attempted bitswap peering workaround.

[66] "The Diagnostic Probe: Tracing Cross-Node Block Retrieval in a Distributed S3 Cluster" — Message 2065, the discovery that bitswap does not help the S3 layer.

[70] "The Proxy Epiphany: Recognizing Architectural Boundaries in a Distributed S3 Deployment" — Message 2069, the realization that a proxy layer was architecturally required.

[74] "The Proxy Awakens: An Architectural Pivot in Distributed S3 Deployment" — Message 2073, the initial deployment of the s3-proxy binary.

[77] "The Moment Infrastructure Meets Architecture: Deploying the S3 Proxy on a Three-Node QA Cluster" — Message 2076, the manual configuration of the s3-proxy.

[78] "The Configuration That Wasn't: Debugging an S3 Proxy Deployment on a Distributed Storage Cluster" — Message 2077, the debugging of the failed proxy startup.

[79] "The $10 Debug: When a Mismatched Environment Variable Name Brought Down a Distributed S3 Proxy" — Message 2078, the discovery of the FGW_YCQL_HOSTS vs RIBS_S3_CQL_HOSTS mismatch.

[80] "\"Why Are You Not Doing This With Ansible?\"" — Message 2079, the user's question that triggered the pivot to infrastructure automation.

[81] "The Ansible Pivot: A Moment of Engineering Discipline in Distributed Systems Deployment" — Message 2080, the assistant's immediate acknowledgment and course correction.

[85] "The Pivot Point: When a Single Ansible Inventory Edit Restored Infrastructure Discipline" — Message 2084, the inventory update that enabled Ansible deployment.

[86] "The Ansible Pivot: How a Single User Question Transformed Infrastructure Deployment" — Message 2086, the successful Ansible playbook run.

[87] "The Pivot to Automation: How a Single User Question Transformed a Deployment Strategy" — Message 2087, the verification that the Ansible-deployed proxy was running.

[89] "The Moment of Proof: Validating Cross-Node S3 Access Through a Proxy Layer" — Message 2088, the curl-based validation of cross-node reads through the proxy.

[90] "The Validation Milestone: Load Testing a Distributed S3 Proxy After Architectural Correction" — Message 2089, the load test that confirmed proxy functionality under concurrency.

[91] "The Verification That Mattered: Confirming Distributed Traffic in a Multi-Node S3 Cluster" — Message 2090, the journalctl-based verification that both nodes received traffic.

[92] "The Verification That Failed: A CQL Query Reveals the Boundaries of Distributed Systems Knowledge" — Message 2091, the failed GROUP BY query that exposed YCQL limitations.