The Persistence of Dirty State: A Debugging Moment in Distributed Systems

Introduction

In the middle of a marathon debugging session building a horizontally scalable S3 architecture, a single message captures one of the most frustrating and instructive moments in distributed systems development: the discovery that state persists even when you think you've cleaned everything. Message 1293 in this coding session is brief — barely two lines of reasoning followed by a shell command and its output — but it represents a critical inflection point where the developer realizes that their mental model of system boundaries is wrong. The message reads:

Still dirty migrations. The shared S3 keyspace still has the dirty migration. Let me check the db-init and truncate the migration table: [bash] docker exec test-cluster-yugabyte-1 bin/ycqlsh 127.0.0.1 19042 -e "DESCRIBE KEYSPACES;"

>

filecoingw_kuri2 filecoingw system_schema system filecoingw_kuri1 system_auth filecoingw_s3

This article examines this single message in depth: the reasoning that produced it, the assumptions it reveals, the knowledge it depends on, and the insight it generates. It is a microcosm of distributed systems debugging — a moment where theory meets stubborn reality.

Context: The Architecture Under Construction

To understand why this message was written, one must understand what was being built. The developer was constructing a test cluster for a horizontally scalable S3-compatible storage system called Filecoin Gateway (FGW). The architecture follows a three-layer design:

  1. S3 Frontend Proxy: A stateless proxy layer that accepts S3 API requests on port 8078 and routes them to storage nodes.
  2. Kuri Storage Nodes: Two stateful nodes (kuri-1 and kuri-2) that store data and expose internal S3 APIs on ports 8079 and 8081 respectively.
  3. YugabyteDB: A shared metadata database that stores object metadata, bucket information, and migration state. The system uses a database-per-node isolation strategy, where each Kuri node has its own keyspace (filecoingw_kuri1, filecoingw_kuri2) plus a shared keyspace (filecoingw_s3) for cross-cutting concerns like bucket metadata. The database schema is managed through a migration system that tracks which migrations have been applied — and this is where the trouble begins.

The Immediate Trigger: Failed Container Starts

In the messages immediately preceding this one, the developer had been fighting a series of failures. The Kuri nodes and S3 proxy were starting and immediately exiting with errors. The logs revealed a critical clue: "Configuration load failed: %w invalid log level:" and CQL migration errors. The migration system was reporting "dirty" state — meaning a previous migration had been partially applied and the system refused to proceed without manual intervention.

The developer's first instinct was correct: clean the data directories and start fresh. They ran stop.sh, then used an Alpine container to wipe /data/fgw2/yugabyte/*, /data/fgw2/kuri-1/*, and /data/fgw2/kuri-2/*. They then ran start.sh to bring everything up from a clean state.

But when the containers started, kuri-1 still failed. The migration was still dirty.

The Reasoning: Tracing the Persistence of State

This is where message 1293 enters. The developer's reasoning is captured in the opening line: "Still dirty migrations. The shared S3 keyspace still has the dirty migration."

The word "still" is telling. It reveals an expectation that was violated. The developer assumed that cleaning the Kuri data directories and the YugabyteDB data directory would reset all state. But the migration was still dirty. Why?

The developer then articulates their next hypothesis: "Let me check the db-init and truncate the migration table." This is a pivot from "clean everything" to "inspect and surgically repair." They run a CQL command to list all keyspaces in the YugabyteDB instance.

The output reveals the truth:

filecoingw_kuri2  filecoingw   system_schema  system
filecoingw_kuri1  system_auth  filecoingw_s3

The filecoingw_s3 keyspace still exists. The developer had cleaned the Kuri data directories and the YugabyteDB data directory, but the filecoingw_s3 keyspace — which is the shared S3 metadata keyspace — was recreated by the db-init service during startup. The migration table within that keyspace retained its dirty state because the migration framework stores its state in the database itself, and the db-init service applies migrations on every startup.

The critical insight is this: the developer had cleaned the data files but not the database state. The YugabyteDB data directory wipe removed the physical files, but the db-init container — which runs on every startup — recreated the keyspaces and reapplied the migrations. If the migration scripts themselves were broken or if the migration state was stored in a way that survived a clean start (e.g., if the migration tracking table was created but the migration itself failed partway through), the dirty state would reappear.

Assumptions and Their Violations

This message reveals several assumptions, some correct and some incorrect:

Assumption 1: Cleaning data directories resets all state. This was partially correct — it resets the Kuri node state and the raw YugabyteDB data. But it failed to account for the db-init service's behavior. The db-init container is designed to initialize the database schema on startup. If its migration logic is idempotent (which it should be), cleaning and restarting should work. But if the migration framework detects a dirty state from a previous partial run, it will refuse to proceed even after a clean start, because the dirty flag is stored in the migration table itself.

Assumption 2: The migration state is stored in the Kuri data directories. This was incorrect. The migration state for the shared S3 keyspace is stored in the YugabyteDB database, not in the Kuri nodes' local data. The developer had cleaned the Kuri directories (/data/fgw2/kuri-1/*, /data/fgw2/kuri-2/*) and the YugabyteDB data directory (/data/fgw2/yugabyte/*), but the db-init service recreates the keyspaces from scratch on each startup. If the migration scripts are not perfectly idempotent, the dirty state will be reproduced.

Assumption 3: The filecoingw_s3 keyspace is managed by the Kuri nodes. The keyspace listing reveals that filecoingw_s3 exists alongside filecoingw_kuri1 and filecoingw_kuri2. The per-node keyspaces (kuri1, kuri2) are managed by their respective Kuri nodes, but the shared filecoingw_s3 keyspace is managed by the db-init service or the S3 proxy. The developer's mental model may have conflated these two ownership domains.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains:

CQL (Cassandra Query Language): The command uses ycqlsh, the YugabyteDB CQL shell, which is compatible with Apache Cassandra's query language. Understanding that DESCRIBE KEYSPACES lists all keyspace names is fundamental.

YugabyteDB Architecture: YugabyteDB is a distributed SQL database that supports both PostgreSQL-compatible YSQL and Cassandra-compatible YCQL APIs. Port 19042 is the YCQL port. The database runs inside a Docker container with health checks.

Migration Frameworks: The concept of "dirty migrations" comes from database migration tools (like Flyway, Alembic, or custom implementations). A dirty migration means a migration was applied but didn't complete successfully, leaving the database in an unknown state. The framework refuses to apply new migrations until the dirty state is resolved.

Docker Compose Service Topology: The test cluster uses multiple services — yugabyte, db-init, kuri-1, kuri-2, s3-proxy — each with specific startup dependencies. The db-init service runs before the Kuri nodes to ensure the database schema exists.

Keyspace Segregation Strategy: The architecture uses separate keyspaces per node (filecoingw_kuri1, filecoingw_kuri2) plus a shared keyspace (filecoingw_s3). This is a deliberate design choice for isolation and scalability.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

Evidence of the dirty migration's location: The keyspace listing confirms that filecoingw_s3 exists and contains the dirty migration state. This narrows the debugging target from "somewhere in the system" to "the shared S3 keyspace."

Confirmation of the db-init service's role: The fact that filecoingw_s3 exists after a clean data wipe proves that the db-init service recreates it on startup. This is important for understanding the system's lifecycle.

A refined debugging strategy: The developer's stated next step — "check the db-init and truncate the migration table" — represents a shift from brute-force cleanup to targeted surgical intervention. Instead of wiping everything again, they will inspect the migration table within filecoingw_s3 and manually reset its state.

A lesson in system boundaries: The persistence of the dirty migration teaches that in distributed systems, state can live in unexpected places. Cleaning data directories is not the same as resetting database state, especially when initialization services recreate schemas from scripts.

The Thinking Process: A Window into Debugging Methodology

The reasoning visible in this message reveals a structured debugging approach:

  1. Observe the symptom: Containers fail with dirty migration errors.
  2. Form a hypothesis: The data directories contain old state; cleaning them will fix it.
  3. Test the hypothesis: Stop containers, clean data directories, restart.
  4. Observe the result: The symptom persists.
  5. Refine the hypothesis: The dirty state must be in the shared S3 keyspace, not in the Kuri data.
  6. Gather evidence: List keyspaces to confirm the shared keyspace exists.
  7. Plan the next intervention: Inspect the db-init logic and truncate the migration table. This is textbook debugging: form a hypothesis, test it, observe the outcome, and refine. The developer does not jump to conclusions or randomly change variables. Each step is deliberate and evidence-driven. The phrase "Let me check the db-init and truncate the migration table" is particularly telling. It shows that the developer has already formed a mental model of the fix: the db-init service is responsible for the migration state, and truncating the migration table will reset the dirty flag. This is a specific, targeted intervention rather than another broad cleanup.

Technical Significance

This message, while brief, captures a fundamental challenge in distributed systems development: state management across service boundaries. In a monolithic application, cleaning state is straightforward — delete the database file and restart. But in a distributed system with multiple services, initialization scripts, and migration frameworks, state is created and managed by different components at different times.

The filecoingw_s3 keyspace is created by the db-init service, but its migration state is consumed by the Kuri nodes. If the migration framework in the Kuri nodes marks a migration as dirty, the db-init service cannot clean it on restart because the dirty flag is stored in the database itself. This creates a circular dependency: the migration state can only be reset by directly manipulating the database, not by restarting services.

This is why the developer's next step is to "truncate the migration table" — a direct database operation that bypasses the service layer entirely. It's a pragmatic solution, but it also highlights a design weakness: the migration system should be resilient to partial failures, or the initialization scripts should check for and repair dirty state automatically.

Conclusion

Message 1293 is a small but revealing moment in a complex debugging session. It shows a developer moving from a failed hypothesis to a refined one, from broad cleanup to targeted intervention, and from assuming that state follows obvious boundaries to discovering that it persists in unexpected places. The message is a reminder that in distributed systems, state is never truly gone until you understand every component that creates, stores, and consumes it.

The lesson extends beyond this specific session: when debugging distributed systems, always verify your assumptions about where state lives. Cleaning data directories is not the same as resetting a database. Restarting services is not the same as clearing migration state. And the most persistent bugs are often the ones that survive your best attempts to wipe them away — because you were looking in the wrong place.