The Duplicate Object Trap: A Lesson in Shared-State Migration Management in Distributed S3 Storage

Introduction

In the trenches of distributed systems debugging, the smallest oversight can cascade into a confounding failure. Message 1299 from this coding session captures one such moment—a seemingly straightforward restart of a storage node that reveals a subtle but critical flaw in how database migrations are managed across shared and per-node keyspaces. The assistant, deep in the process of stabilizing a horizontally scalable S3 test cluster, encounters a "Duplicate Object" error that exposes the tension between shared infrastructure and node-local state.

The Message in Full

The message reads:

Good, kuri1 is clean. Now restart kuri-1:

>

```bash docker start test-cluster-kuri-1-1 && sleep 5 && docker logs test-cluster-kuri-1-1 2>&1 | tail -15 test-cluster-kuri-1-1 (details: Duplicate Object. Object 'filecoingw_s3.multipartuploads' already exists create table MultipartUploads ^^^^^^^^^^^^^^^^ ( upload_id text, bucket text, key text, node_id text, created_at timestamp, expires_at timestamp, status text, primary key (upload_id) );

>

(ql error -202)) ```

At first glance, this is a routine debugging step: verify a fix, restart a container, check the logs. But the error it surfaces tells a deeper story about the challenges of managing shared database state in a multi-node distributed system.

Context and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the debugging journey that led to it. The session centers on building and debugging a test cluster for a horizontally scalable S3 architecture. The architecture has three layers: stateless S3 frontend proxies that route client requests, Kuri storage nodes that hold the actual data, and a shared YugabyteDB cluster that stores metadata.

The assistant had been wrestling with a cascade of issues. A port conflict had emerged because the YugabyteDB web UI was binding to port 15433—the same port the assistant had configured for YSQL access. This was resolved by moving YSQL to port 25433. Data directories had to be cleaned with Alpine containers because they were owned by root. Container health checks were failing. And crucially, the database migration system had left the shared filecoingw_s3 keyspace in a "dirty" state—a flag that prevents migrations from running again after a failed or interrupted attempt.

The assistant had just manually fixed that dirty flag by executing a YCQL query:

UPDATE filecoingw_s3.schema_migrations SET dirty = false WHERE version = 1754293669;

After verifying that the per-node keyspace filecoingw_kuri1 was clean (its migration was already marked dirty = false), the assistant restarted kuri-1 to see if the fix would allow it to boot successfully. This message is that verification step—the moment of truth after a manual database repair.## The Error: Duplicate Object and What It Reveals

The error itself is instructive. The kuri-1 node crashes during startup because it attempts to create the MultipartUploads table in the filecoingw_s3 keyspace, but that table already exists. The migration framework is supposed to track which migrations have been applied via the schema_migrations table, but something has gone wrong.

The key insight is that the filecoingw_s3 keyspace is shared across all Kuri nodes. Unlike the per-node keyspaces (filecoingw_kuri1, filecoingw_kuri2), which are isolated to individual storage nodes, the S3 keyspace holds metadata that must be visible to all nodes—things like multipart upload records, S3 object metadata, and bucket information. This shared nature creates a coordination problem: if one node applies a migration, the schema_migrations table records it, but another node starting up may not see that record correctly, or the migration logic may not check properly before attempting to create tables.

The assistant had manually set dirty = false on the shared keyspace's migration record, but this didn't address the root cause. The migration system likely checks the dirty flag to decide whether to run pending migrations. By clearing the dirty flag, the assistant expected the node to skip the already-applied migration. But the error shows that the node is still trying to create the MultipartUploads table—suggesting that the migration version tracking and the actual table creation are decoupled, or that the migration logic doesn't check for table existence before executing CREATE TABLE.

Assumptions Made

This message reveals several assumptions—some explicit, some implicit:

  1. The dirty flag fix would be sufficient. The assistant assumed that resetting the dirty flag to false would allow the migration system to recognize that the migration had already been applied and skip it. The error proved this assumption wrong.
  2. Per-node keyspace cleanliness implies shared keyspace readiness. The assistant checked filecoingw_kuri1.schema_migrations and found it clean, then concluded "kuri1 is clean." But the node also depends on the shared filecoingw_s3 keyspace, which had a different problem.
  3. The migration system uses the dirty flag as its sole gate. The error suggests that the migration framework may use a different mechanism—perhaps checking a version number or a list of applied migrations—rather than simply relying on the dirty flag. The dirty flag might only indicate whether a migration is in progress, not whether it has been completed.
  4. Restarting the container would be a clean test. The assistant expected that restarting kuri-1 after the database fix would produce a clean startup. Instead, it revealed a deeper issue with how the migration system interacts with shared state.## The Thinking Process Visible in the Reasoning The assistant's reasoning, visible across the surrounding messages, follows a systematic debugging pattern. First, identify the symptom: kuri-1 exits immediately after starting. Second, inspect the logs: the error points to a CQL migration failure. Third, trace the cause: the schema_migrations table shows dirty = true. Fourth, apply a fix: manually set dirty = false. Fifth, verify: check that the per-node keyspace is also clean. Sixth, test: restart the container and observe. This is textbook debugging methodology—isolate, diagnose, fix, verify. But the error that surfaces in message 1299 reveals that the diagnosis was incomplete. The assistant correctly identified that the dirty flag was blocking the migration, but incorrectly assumed that clearing it would be sufficient. The migration system's behavior is more nuanced: even with dirty = false, the node still attempts to create tables that already exist. The thinking here reflects a deeper tension in distributed systems debugging: the gap between what the database state shows and what the application logic expects. The assistant is reasoning about the system at the database level (manipulating rows in schema_migrations), but the application's migration logic operates at a different level of abstraction—it may compare version numbers, check table existence, or use other heuristics that aren't visible in the raw database state.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. Evidence that the dirty flag fix is insufficient. The error proves that manually resetting schema_migrations.dirty does not prevent the migration from re-executing. This is a concrete finding about the behavior of the migration framework.
  2. A bug report in real-time. The error documents that the migration system attempts to create tables without checking for their existence, or that the version tracking doesn't prevent re-execution of already-applied migrations.
  3. A debugging milestone. The message marks the transition from one phase of debugging (fixing the dirty flag) to the next (understanding why the migration still runs). It's a pivot point in the troubleshooting process.
  4. Documentation of shared-state fragility. The error demonstrates that shared keyspaces are more vulnerable to migration issues than per-node keyspaces, because multiple nodes interact with the same migration state.## Mistakes and Incorrect Assumptions The primary mistake in this message is not the error itself—the error is a legitimate bug in the system—but the assumption that the fix was complete. The assistant's reasoning went: "The dirty flag is set → clearing it will allow the node to start." This is a reasonable assumption based on how many migration frameworks work, but it proved incorrect. A secondary mistake is the framing of "kuri1 is clean." The assistant checked only the per-node keyspace (filecoingw_kuri1) and concluded the node was ready to restart. But the node also depends on the shared filecoingw_s3 keyspace, which had a different migration version and a different problem. The per-node keyspace had version 1756300000 with dirty = false, while the shared keyspace had version 1754293669 with dirty = false (after the fix). These are different migration versions, and the shared keyspace's migration had been applied but its dirty flag had been manually cleared—leaving the system in an inconsistent state that the migration framework couldn't handle. The deeper mistake is a failure to fully understand the migration framework's logic before manipulating its state. In production systems, manually editing migration tables is a last resort, and it requires understanding exactly what each column means and how the framework uses it. The dirty flag might mean "a migration is currently running" rather than "a migration failed." Setting it to false might tell the framework "the migration completed successfully," but if the framework also checks a checksum, a version history, or a list of applied migrations, the manual edit won't be sufficient.

Broader Implications

This message, while small, illustrates a fundamental challenge in distributed systems development: the tension between shared state and node-local autonomy. In this architecture, the S3 metadata keyspace is shared across all Kuri nodes, which means migration state must be carefully coordinated. If two nodes start simultaneously and both try to run the same migration, the system must handle the race condition. If one node crashes mid-migration, the dirty flag must be resolved before any node can proceed.

The error also highlights the importance of idempotent migrations. A well-designed migration framework should handle the case where a table already exists—either by skipping the CREATE TABLE statement or by using IF NOT EXISTS clauses. The error message shows that the migration is using a plain CREATE TABLE without such safeguards, making it vulnerable to exactly this kind of duplicate-object error.

For the assistant, this message represents a learning moment. The debugging process will now need to shift from "fix the dirty flag" to "understand the migration framework's logic and either fix it to be idempotent or properly coordinate shared-state migrations." This is a more complex problem, but one that must be solved for the distributed architecture to work reliably.

Conclusion

Message 1299 captures a pivotal moment in a debugging session—the instant when a seemingly simple fix reveals a deeper, more interesting problem. The assistant's attempt to restart a storage node after manually clearing a migration dirty flag exposes the gap between database state and application logic in distributed systems. It's a reminder that in complex architectures, the state we see in the database is only one layer of truth; the application's interpretation of that state, mediated by framework logic, caching, and version tracking, is equally important. The duplicate object error is not just a bug—it's a signal that the migration system needs to be more robust in the face of shared-state coordination, and that manual database surgery is rarely a complete fix.