The Dirty Migration: A Surgical Fix That Resurrected a Distributed Storage Cluster

The Message

ssh 10.1.232.82 "python3 -c \"
from cassandra.cluster import Cluster
cluster = Cluster(['10.1.232.82'], port=9042)

# Fix for all keyspaces
for ks in ['filecoingw_kuri_01', 'filecoingw_kuri_02']:
    session = cluster.connect(ks)
    try:
        session.execute('UPDATE schema_migrations SET dirty = false WHERE version = 1769890615')
        print(f'{ks}: Fixed dirty migration')
    except Exception as e:
        print(f'{ks}: Error - {e}')

cluster.shutdown()
\""
filecoingw_kuri_01: Fixed dirty migration
filecoingw_kuri_02: Fixed dirty migration

At first glance, this appears to be a mundane database fix—a Python one-liner executed over SSH that flips a boolean column in two rows of a CQL table. But in the context of deploying a distributed storage cluster across three physical machines, this single command was the linchpin that transformed a broken, restart-looping set of services into a functioning multi-node Filecoin Gateway (FGW) system. Understanding why this message exists, what it reveals about the architecture it operates on, and the chain of reasoning that led to it offers a fascinating window into the realities of infrastructure engineering.

The Problem: A Cluster That Would Not Start

The message was written in direct response to a service failure. Moments earlier, the assistant had deployed the kuri daemon—the core storage node process for the FGW distributed S3-compatible system—to two physical nodes (kuri1 at 10.1.232.83 and kuri2 at 10.1.232.84). The deployment had been painstaking: building binaries from source, creating systemd service files with security hardening, setting up restricted-permission configuration directories, and storing the CIDgravity API token in a separate vault file loaded at runtime. Everything looked correct. Yet when the assistant ran sudo systemctl start kuri, the service immediately crashed and entered an auto-restart loop.

The journal logs told a cryptic story. The kuri process printed "Configuration loaded" and then "Initial..." before dying. No stack trace, no obvious error message—just a silent exit. This is the kind of failure that frustrates operators because it provides almost no diagnostic surface. The assistant's first instinct was correct: check the database state. The kuri daemon, upon startup, validates its database schema against the expected migration version. If the migration tracking system reports a problem, the process aborts before it can serve any requests.

Discovering the Dirty Migration

The assistant connected to the YugabyteDB instance running on the head node (10.1.232.82) and queried the schema_migrations table for the filecoingw_kuri_01 keyspace. The result was stark: a single row with version 1769890615 and dirty = True. This is a well-known pattern in database migration frameworks (borrowed from Flyway and similar tools). A "dirty" flag means a migration was started but never completed successfully. The framework sets dirty = true before applying changes and resets it to false upon successful completion. If the process crashes mid-migration, or if someone manually interferes, the flag stays true, and subsequent startup attempts refuse to proceed—they cannot know whether the schema is in a consistent state.

The root cause traced back to earlier testing. The test suite for the FGW system had run against this same YugabyteDB instance, and during those runs, something had interrupted the migration process. Perhaps a test timeout killed the process mid-migration. Perhaps a container restart left the flag dangling. Whatever the cause, the artifact was a boolean column set to true in two keyspaces, and it was blocking the entire cluster from starting.

The Fix: Targeted, Minimal, and Pragmatic

The assistant's response was a model of surgical infrastructure intervention. Rather than attempting to re-run the migrations (which might fail again or produce duplicate objects), rather than dropping and recreating the keyspaces (which would lose data), rather than restarting the entire YugabyteDB node—the assistant chose to directly manipulate the migration tracking table. The Python script connects to the CQL interface (Cassandra Query Language, which YugabyteDB supports for compatibility), iterates over the two affected keyspaces, and executes a simple UPDATE statement to flip dirty from true to false for the specific migration version.

This decision reveals several important assumptions. First, the assistant assumed that the schema itself was fully and correctly applied—that only the tracking flag was wrong. This was a reasonable inference because the kuri init command had run successfully earlier and printed table creation statements (the CQL schema for garbage collection queues, reference counting tables, and other structures). The "ql error -12" that appeared at the end of the init output was initially dismissed as a harmless "table already exists" condition. In hindsight, that error may have been a symptom of the same underlying migration state problem, but the schema objects themselves were present in the database. Second, the assistant assumed that both keyspaces shared the same migration version number (1769890615), which turned out to be correct—the test suite had run the same migration code against both per-node keyspaces. Third, the assistant assumed that simply clearing the dirty flag would allow the daemon to proceed with normal startup, which proved accurate when the services subsequently came online.

What This Message Teaches About Distributed Systems

There is a deeper architectural story embedded in this fix. The FGW system uses per-node keyspaces in YugabyteDB: filecoingw_kuri_01 for the first storage node and filecoingw_kuri_02 for the second. Each kuri daemon owns its own keyspace for its local blockstore metadata, reference counts, and garbage collection state. But they share a third keyspace, filecoingw_s3, which holds the object routing table that maps S3 object keys to the storage node that holds their data. This design is intentional: it allows each node to operate independently for local data while coordinating through the shared keyspace for cross-node object access. The dirty migration had to be fixed in both per-node keyspaces before the cluster could function, but interestingly, the assistant's fix in this message only addressed two of the three keyspaces. The filecoingw_s3 keyspace was handled in a subsequent step, suggesting that the s3-proxy service (which reads from that keyspace) had not yet been deployed at this point, or that its migration state was checked separately.

The Input Knowledge Required

To understand this message, one must grasp several layers of infrastructure knowledge. The Cassandra-compatible CQL interface of YugabyteDB is not the default way to interact with the database—most users interact through PostgreSQL-compatible SQL on port 5433. But the FGW system's internal data structures are modeled as CQL tables (with compound primary keys, clustering columns, and counter columns) because they were originally designed for Apache Cassandra and later ported to YugabyteDB. The schema_migrations table is an internal framework table, not part of the application schema, and its dirty column is a boolean that acts as a gatekeeper for the entire startup sequence. The Python Cassandra driver (cassandra.cluster.Cluster) must be installed on the head node—it was not part of the base OS image, meaning the assistant had already ensured its availability during earlier provisioning steps.

The Output Knowledge Created

The immediate output was two lines confirming that both keyspaces had been fixed. But the real output was invisible: the kuri daemons on both storage nodes, when restarted, would now pass their migration validation check and proceed to bind their internal APIs, open their S3-compatible ports, register themselves in the cluster topology, and begin accepting data. This single database update was the difference between a cluster that spun in a restart loop and a cluster that could serve its first object. The subsequent verification steps—checking the cluster topology API, running a load test with s3cmd, confirming cross-node reads through the proxy—all depended on this fix being in place.

The Thinking Process

The reasoning visible in this message and its immediate predecessors follows a classic diagnostic arc: observe the symptom (service crashes immediately), inspect the logs (truncated output with no clear error), form a hypothesis (database migration state is blocking startup), gather evidence (query schema_migrations, find dirty=true), and apply a targeted remedy (update the flag). The assistant did not over-engineer the solution—it did not write a migration rollback script, did not recreate the keyspaces, did not file a bug report against the test suite. It recognized that in a QA environment, pragmatism trumps perfection. The dirty flag was an artifact of development testing, not a sign of schema corruption. The fastest path to an operational cluster was to clear the flag and move on.

This is the kind of decision that separates infrastructure automation that works in theory from infrastructure that works in practice. The dirty migration fix is not glamorous. It will never appear in a roadmap document or a feature announcement. But without it, the three-node FGW cluster would have remained a collection of machines with software installed but nothing running—a dark cluster, consuming power and network bandwidth but producing nothing. One UPDATE statement, executed over SSH, brought the light back.