Debugging Distributed Database Initialization in a Multi-Node S3 Cluster

Introduction

In distributed systems, few problems are as frustrating as the "one node works, another doesn't" scenario. When two instances of the same software, configured similarly and pointed at the same database cluster, behave differently, the root cause is rarely obvious. Message 825 of this coding session captures exactly such a moment: an assistant, having just rebuilt and redeployed a Docker-based test cluster for a horizontally scalable S3 storage system, discovers that one of two Kuri storage nodes has failed to start while the other runs perfectly. The message is a single step in a debugging chain, but it reveals the layered complexity of multi-node database initialization, the importance of understanding database-per-node architecture decisions, and the disciplined process of isolating configuration failures in distributed systems.

The Context: A Three-Layer S3 Architecture Under Test

The broader project is an ambitious horizontally scalable S3-compatible storage system for the Filecoin Gateway. The architecture follows a clean three-layer design: stateless S3 frontend proxies accept client requests and route them to independent Kuri storage nodes, which in turn share object routing metadata through a common YugabyteDB cluster. Each Kuri node operates with its own isolated RIBS keyspace—filecoingw_kuri1 for node 1, filecoingw_kuri2 for node 2—while the S3 routing layer uses a shared filecoingw_s3 keyspace. This design enables horizontal scaling: add more Kuri nodes with their own keyspaces, and the frontend proxy can distribute load across them.

The test cluster, orchestrated via Docker Compose, includes YugabyteDB, two Kuri nodes, the S3 frontend proxy, and an nginx-based web UI for monitoring. Ports are carefully allocated: 8078 for the S3 API, 9010 and 9011 for the respective node web UIs, and 7001/7002 for CAR file access. The cluster had been through multiple iterations of debugging—HTTP route conflicts, JSON case mismatches between frontend and backend, missing database tables—and had reached a state where the monitoring dashboard was displaying live metrics. The user had just requested cosmetic fixes (renaming "SLA" to "SLO" at 350ms on the latency chart), and the assistant had rebuilt the React frontend, rebuilt the Docker image, and restarted the cluster.

The Problem: Asymmetric Node Behavior

After restarting the cluster with docker compose up -d --force-recreate, the assistant ran docker compose ps and immediately noticed a problem: kuri-2 was missing from the running services. Only kuri-1 and the S3 proxy were listed. Checking the logs for kuri-2 revealed a cryptic error: sp_deal_stats_tmp table was missing. Yet kuri-1 had started without issue. Both nodes were running the same Docker image, connecting to the same YugabyteDB cluster, and configured through the same gen-config.sh script. Why would one node fail and the other succeed?

The assistant's initial hypothesis was that the database initialization step—a db-init container that runs SQL scripts against YugabyteDB—had not created the necessary tables for kuri-2. But the db-init logs showed "Databases initialized successfully," and restarting kuri-2 individually did not resolve the issue. The assistant then searched the codebase for references to sp_deal_stats_tmp and deal_stats_tmp, finding nothing—suggesting the table might be created dynamically by the RIBS storage layer at runtime, not by the initialization script.

The Subject Message: Verifying Database Existence

Message 825 is the assistant's next logical step in the debugging process. Having confirmed that kuri-1 is running and that the table reference is not in the initialization code, the assistant turns to the database layer itself. The command executed is:

cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec yugabyte bin/ysqlsh -h yugabyte -U yugabyte -c '\l'

This runs ysqlsh—the YugabyteDB SQL shell—inside the yugabyte container, listing all databases with the \l meta-command. The output reveals five databases:

filecoingw
filecoingw_kuri1
filecoingw_kuri2
postgres
...

The critical finding is that both filecoingw_kuri1 and filecoingw_kuri2 databases exist. This confirms that the database-per-node initialization is working correctly—the db-init container has created the expected keyspaces for both nodes. The problem is not at the database creation level.

The Thinking Process: What This Message Reveals

This message is a textbook example of systematic debugging in distributed systems. The assistant's reasoning, visible through the sequence of actions, follows a clear pattern:

  1. Observe the symptom: kuri-2 is not running; kuri-1 is running.
  2. Gather diagnostic data: Check container status, then check logs for the failing node.
  3. Form a hypothesis: The sp_deal_stats_tmp table is missing from kuri-2's database.
  4. Test the hypothesis: Search the codebase for where this table is created. Find nothing.
  5. Refine the hypothesis: Perhaps the database itself wasn't created for kuri-2.
  6. Execute the diagnostic: List all databases in YugabyteDB to verify creation. The message captures step 6. The assistant is ruling out a specific failure mode: that the db-init container failed to create the filecoingw_kuri2 database. By confirming both databases exist, the assistant narrows the problem space. The issue must be elsewhere—perhaps in the table creation within the database, perhaps in the configuration that kuri-2 is loading, or perhaps in the order of operations during startup.

Assumptions and Their Implications

Several assumptions underpin this debugging step. First, the assistant assumes that database creation is a prerequisite for table creation—a reasonable assumption, but not one that guarantees table creation succeeded. The db-init container might have created the databases but failed to run subsequent SQL statements for kuri-2 specifically. Second, the assistant assumes that both nodes should have identical database schemas, which follows from the architecture but might not hold if the initialization script has conditional logic. Third, the assistant assumes that the sp_deal_stats_tmp table is expected to exist at startup, which might be incorrect—it could be a table that RIBS creates lazily on first access, and the error might be a red herring masking a different configuration problem.

The message also reveals an implicit assumption about the debugging workflow: that checking database existence is a quick, low-cost diagnostic that can eliminate a major class of failures. This is sound engineering practice—start with the simplest checks before diving into complex configuration analysis.

Input Knowledge Required

To fully understand this message, a reader needs several pieces of context. They need to know that the system uses YugabyteDB as its distributed database layer, with per-node keyspaces for isolation. They need to understand the Docker Compose orchestration—that db-init is a one-shot container that runs initialization scripts before the application containers start. They need familiarity with the ysqlsh command-line tool and the \l meta-command for listing databases. They also need to understand the broader architecture: that kuri-1 and kuri-2 are independent storage nodes, each with their own database, sharing only the S3 routing metadata.

Output Knowledge Created

This message produces several valuable pieces of knowledge. It confirms that the database-per-node initialization strategy is working correctly—both filecoingw_kuri1 and filecoingw_kuri2 exist. It eliminates database creation as the root cause of kuri-2's failure, forcing the investigation to move to other areas: table-level initialization, configuration parameters, or startup ordering. It also documents the state of the YugabyteDB cluster at a specific point in time, which could be useful for comparing against future states.

The Broader Significance

What makes this message interesting is not the command itself, but what it represents: the disciplined narrowing of a failure space in a complex distributed system. The assistant could have jumped to conclusions—blaming the configuration generator, the Docker networking, or the YugabyteDB cluster health. Instead, it methodically checked each layer, starting with the most fundamental: does the database exist? This is the database equivalent of "is it plugged in?"—and it's often the question that saves hours of debugging.

The message also highlights a common challenge in multi-node distributed systems: symmetric configuration does not guarantee symmetric behavior. Both Kuri nodes use the same Docker image, the same initialization pipeline, and the same database cluster. Yet one starts and one doesn't. The asymmetry forces the debugger to look beyond the obvious and examine the specific state of each node's dependencies.

Conclusion

Message 825 is a small but revealing window into the process of debugging distributed database initialization. It shows a developer (or in this case, an AI assistant) systematically verifying assumptions, ruling out failure modes, and gathering diagnostic data. The command itself is simple—list databases in YugabyteDB—but the reasoning behind it is sophisticated: a chain of inferences about how the system should work, what could go wrong, and how to efficiently test each hypothesis. For anyone building or operating multi-node storage systems, this message is a reminder that the most effective debugging often starts with the most basic questions.