The Database That Wasn't There: Debugging Startup Order in a Distributed S3 Test Cluster

Introduction

In the middle of building a horizontally scalable S3-compatible storage system for the Filecoin Gateway, a seemingly simple error brought the entire test cluster to a halt. Both Kuri storage nodes crashed immediately on startup with the same cryptic message buried deep in a chain of dependency injection failures: pq: database "filecoingw" does not exist. The message that became the subject of this analysis — message index 266 in the conversation — is the assistant's concluding summary after diagnosing and fixing this issue. It is a short, retrospective message that catalogs four changes made across the test cluster's infrastructure: adding a db-init service to Docker Compose, wiring up dependency chains, updating the startup script, and documenting the fix. But behind this brief summary lies a rich debugging session that reveals how assumptions about database provisioning, container orchestration, and application initialization interact in distributed systems.

The Subject Message

The message itself reads as follows (quoted exactly from the conversation):

Fixed! Added database initialization to solve the "database does not exist" error.

Changes made:

1. Added db-init service to docker-compose.yml: - Runs after YugabyteDB is healthy - Creates PostgreSQL database: filecoingw - Creates CQL keyspace: filecoingw - Exits successfully when done

2. Updated Kuri dependencies: - Both kuri-1 and kuri-2 now wait for db-init to complete - Uses condition: service_completed_successfully

3. Updated start.sh: - Added explicit wait for db-init to complete - Shows progress while waiting - Warns if initialization times out

4. Updated README: - Added troubleshooting section for database creation issues - Included manual database creation commands

To test the fix: ``bash cd test-cluster ./start.sh /data/fgw2 ``

The cluster will now: 1. Start YugabyteDB 2. Wait for it to be healthy 3. Run db-init to create databases 4. Start Kuri nodes (only after DB is ready)

This is a retrospective summary, not an action message. It contains no new code edits, no tool calls, no further debugging. It is the assistant's way of closing the loop — telling the user what was done, why, and what the expected behavior will be going forward.

The Problem: A Missing Database in a Distributed Stack

To understand why this message was written, we need to trace back to the error that triggered it. The user had run ./start.sh /data/fgw2 to spin up the test cluster, which consists of two Kuri storage nodes backed by a shared YugabyteDB instance. The startup script reported both Kuri nodes as "not running." When the user inspected the logs with docker-compose logs kuri-1, the error was unmistakable:

create postgres migrations driver: pq: database "filecoingw" does not exist

This error was nested inside a long chain of dependency injection failures — the application's fx dependency framework was trying to construct the S3 server, which needed a blockstore, which needed a RIBS instance, which needed an RBS database, which needed a SQL database connection, which finally failed because the PostgreSQL database filecoingw simply did not exist in the YugabyteDB cluster.

The critical detail is that the Docker Compose configuration had set environment variables YSQL_DB: filecoingw and YCQL_KEYSPACE: filecoingw on the YugabyteDB container. The assistant had assumed — reasonably — that these environment variables would cause YugabyteDB to automatically create the specified database and keyspace on startup. This assumption turned out to be incorrect. YugabyteDB's YSQL_DB and YCQL_KEYSPACE environment variables do not auto-provision databases; they are metadata hints used by certain client tools and configuration helpers. The actual database creation must be performed explicitly via SQL commands after the database node is running.## The Reasoning Process: From Assumption to Correction

The assistant's reasoning, visible in the preceding messages, shows a clear diagnostic arc. In message 251, the assistant correctly identifies the root cause: "The issue is that the YugabyteDB database 'filecoingw' doesn't exist. The docker-compose.yml sets the environment variables but doesn't actually create the database." This is the key insight. The assistant then considers three possible solutions: adding an init script to the YugabyteDB container, using yugabyted command flags to auto-create the database, or adding a separate initialization service.

The first attempt (message 252) tries to embed the database creation directly into the YugabyteDB service definition. But the assistant quickly realizes this approach is fragile — if the creation command runs in the background, the health check might pass before the database is actually ready. This is a subtle but important point about container orchestration: health checks and readiness are not the same thing. A container can be "healthy" (its process is running and responding to pings) without the application-level state (a specific database) being initialized.

The assistant then pivots to a cleaner architectural pattern: a dedicated db-init service that runs as a one-shot container after YugabyteDB is healthy, creates the database and keyspace, and then exits. This is a common pattern in Docker Compose setups — using ephemeral "init" containers to perform bootstrapping that the main application containers depend on. However, the assistant initially misunderstands how Docker Compose handles dependencies on one-shot containers. In message 255, the reasoning reveals: "in Docker Compose, you can't depend on a one-shot container completing successfully — you can only depend on it being started." This is actually incorrect for modern Docker Compose. The depends_on directive with condition: service_completed_successfully does wait for a one-shot container to exit with code 0 before starting dependent services. The assistant eventually arrives at the correct approach, using this exact condition.

The Assumptions That Needed Correction

Several assumptions were at play in this debugging session, and recognizing them is essential to understanding why the fix took the shape it did.

Assumption 1: Environment variables provision databases. The most significant incorrect assumption was that YugabyteDB's YSQL_DB and YCQL_KEYSPACE environment variables would automatically create the specified database and keyspace. This is a reasonable assumption — many database containers (PostgreSQL's official image, for example) do auto-create databases from environment variables. But YugabyteDB, being a distributed SQL database with both PostgreSQL-compatible (YSQL) and Cassandra-compatible (YCQL) APIs, does not. The environment variables are advisory, not provisioning directives.

Assumption 2: Docker Compose health checks imply application readiness. The assistant initially tried to embed database creation into the YugabyteDB service definition, but correctly recognized that this could create a race condition where the health check passes before the database is created. This assumption about the relationship between container health and application state is a common pitfall in containerized deployments.

Assumption 3: One-shot containers cannot be used as dependencies. The assistant initially believed that Docker Compose could not wait for a one-shot container to complete before starting dependent services. This was a mistaken understanding of Docker Compose's capabilities — condition: service_completed_successfully exists precisely for this use case. The assistant corrected this during the reasoning process.

Assumption 4: The startup script's status check logic was sufficient. The original start.sh checked for running containers using docker-compose ps, which by default only shows running containers. The assistant later discovered (in a previous chunk) that exited containers don't appear in the default output, requiring docker-compose ps -a to detect completed one-shot services. This assumption about tool behavior had to be corrected across multiple iterations.

Input Knowledge Required

To fully understand this message, several pieces of context are necessary. First, one must understand the architecture of the test cluster: two Kuri storage nodes that implement the S3 API, backed by a shared YugabyteDB instance that stores metadata and object placement information. The Kuri nodes use dependency injection (via the fx framework in Go) to wire together their components — the S3 server depends on a blockstore, which depends on a RIBS (Remote Indexed Block Store) layer, which depends on an RBS database, which depends on a SQL database connection. This chain of dependencies is why a missing database at the bottom of the stack produces a towering error trace.

Second, one must understand Docker Compose's service dependency model, including depends_on with condition: service_healthy and condition: service_completed_successfully. These are Docker Compose v2+ features that allow orchestrating startup order beyond simple "start after" semantics.

Third, familiarity with YugabyteDB's dual API surface is important. YugabyteDB exposes both a PostgreSQL-compatible endpoint (YSQL, on port 5433) and a Cassandra-compatible endpoint (YCQL, on port 9042). The Kuri nodes use both: the CQL interface for object index queries and the SQL interface for relational metadata. The db-init service must create resources on both interfaces.

Output Knowledge Created

This message, and the changes it summarizes, created several important outputs for the project:

A reusable initialization pattern. The db-init service establishes a pattern that can be extended for future bootstrapping needs — creating users, setting up replication, initializing schema migrations, or seeding test data.
Idempotent database initialization. The SQL commands in the db-init service use CREATE DATABASE IF NOT EXISTS and CREATE KEYSPACE IF NOT EXISTS patterns (or their equivalent with error suppression), ensuring that the initialization can be run repeatedly without failure. This is critical for development workflows where containers are frequently restarted.
Explicit startup ordering. The dependency chain yugabyte → db-init → kuri-1/kuri-2 is now explicitly encoded in the Docker Compose configuration, making the startup order clear to anyone reading the file.
Documented troubleshooting guidance. The README update provides manual database creation commands, which serve both as documentation and as a debugging aid for situations where the automated initialization fails.

The Broader Significance

This message represents a classic "last mile" debugging scenario in distributed systems. The infrastructure was largely correct — the Docker images were built, the network was configured, the environment variables were set — but a single missing piece (the database not being created) brought the entire system down. The error message was deeply nested inside a dependency injection chain, making it initially difficult to parse. The assistant had to read through layers of failed to build errors to find the root cause at the bottom: pq: database "filecoingw" does not exist.

The fix itself is elegant in its simplicity: a dedicated init container that runs exactly once, creates the required database resources, and exits. This pattern is well-established in container orchestration — Kubernetes uses init containers for exactly this purpose — and applying it to the Docker Compose test cluster brings production-grade initialization semantics to the development environment.

Moreover, this debugging session highlights an important principle in distributed systems engineering: startup order is a correctness concern, not just a performance concern. When services have hard dependencies on shared state (like a database that must exist before connections can be made), the orchestration layer must enforce ordering guarantees. Docker Compose's depends_on with conditions provides this, but only if the developer explicitly configures it. The default behavior — starting all services simultaneously — would have caused the Kuri nodes to crash-loop until the database happened to be created, which is fragile and unpredictable.

Conclusion

Message 266 is a retrospective summary, but it encapsulates a debugging journey that touched on assumptions about database provisioning, container orchestration semantics, dependency injection chains, and startup ordering. The fix — adding a db-init service with proper dependency conditions — transformed the test cluster from a crash-looping failure into a reliably ordered startup sequence. For anyone building distributed systems with Docker Compose, this pattern of using dedicated initialization containers with service_completed_successfully conditions is a valuable addition to the operational toolkit. The message may be short, but the reasoning behind it reveals the depth of consideration required to make a multi-service distributed system start reliably every time.