The Critical Dependency: Orchestrating Service Startup in a Distributed S3 Test Cluster

Introduction

In the world of distributed systems, few problems are as deceptively simple as service startup ordering. When a system comprises multiple interdependent components—a database, storage nodes, and proxy layers—the sequence in which they come online can determine success or failure. Message 254 of this coding session captures a pivotal moment in building a horizontally scalable S3-compatible storage architecture: the seemingly mundane act of adding Docker Compose dependency directives between services. Yet beneath this routine operation lies a cascade of architectural reasoning, debugging, and system design decisions that illuminate the challenges of distributed system orchestration.

This article examines a single message from an opencode coding session where an assistant reads a docker-compose.yml file with the stated intention: "Now I need to make kuri-1 and kuri-2 depend on db-init so they start after the database is created." What appears to be a simple configuration change is, in reality, the culmination of a debugging journey that exposed fundamental truths about the test cluster's architecture, the behavior of its components, and the assumptions embedded in its design.

The Context: A Cluster That Wouldn't Start

To understand why this message was written, we must first understand the failure that preceded it. The user had executed ./start.sh /data/fgw2 to launch a test cluster consisting of two Kuri storage nodes (kuri-1, kuri-2) sharing a YugabyteDB instance. The startup script reported both nodes as "not running." Examination of their logs revealed a consistent error:

pq: database "filecoingw" does not exist

This error was the root cause. The Kuri nodes, upon initialization, attempted to connect to a PostgreSQL-compatible database called "filecoingw" within YugabyteDB. That database had never been created. The docker-compose.yml had set environment variables like YSQL_DB: filecoingw on the YugabyteDB service, but these variables were merely configuration hints—they did not actually provision the database. The YugabyteDB container started, passed its health check, and declared itself ready, but the required database simply did not exist.

The assistant's response to this failure was to add a db-init service to the Docker Compose configuration. This service would run after YugabyteDB became healthy, executing SQL commands to create the "filecoingw" database and the associated CQL keyspace. This was a sound architectural decision: separate the database server's lifecycle from the database initialization step, ensuring that initialization happens deterministically.

However, adding the db-init service alone was insufficient. Without explicit dependency declarations, Docker Compose would start all services—YugabyteDB, db-init, kuri-1, and kuri-2—in parallel. The Kuri nodes would still attempt to connect to the database before db-init had a chance to run, reproducing the exact same failure. This is the problem that message 254 sets out to solve.

The Message: Reading Before Writing

The message itself is brief and contains two elements: a reasoning section and a file read operation.

## Agent Reasoning
Now I need to make kuri-1 and kuri-2 depend on db-init so they start after the database is created.
[read] /home/theuser/gw/test-cluster/docker-compose.yml

The assistant then reads lines 51-64 of the docker-compose.yml, which show the kuri-1 service definition: its image, port mappings, and environment variables including FGW_NODE_TYPE=storage, FGW_NODE_ID=kuri-1, and the YugabyteDB connection parameters.

This is a planning message. The assistant has identified the missing piece—the dependency chain—and is reading the current file to understand the exact structure before making edits. The reasoning is clear and direct: the Kuri nodes must wait for database initialization to complete. The solution is Docker Compose's depends_on directive.

The Dependency Problem in Distributed Systems

The challenge the assistant faces is a classic one in container orchestration. Docker Compose, by default, starts all services simultaneously. While this maximizes startup parallelism, it assumes that services are independent or can handle the absence of their dependencies gracefully. In practice, many applications fail immediately if their database isn't available, with no retry logic or graceful degradation.

The depends_on directive addresses this by establishing startup order. When kuri-1 declares depends_on: db-init, Docker Compose ensures that the db-init container starts before kuri-1 and, critically, waits for it to exit successfully before proceeding. Since db-init is a one-shot container that creates the database and then terminates, this creates a reliable initialization sequence:

YugabyteDB starts and becomes healthy
db-init starts, connects to YugabyteDB, creates the "filecoingw" database and CQL keyspace, then exits
kuri-1 and kuri-2 start, find the database already exists, and initialize successfully This dependency chain is the backbone of reliable service startup. Without it, the cluster's behavior is non-deterministic—sometimes the database initialization might complete before the Kuri nodes connect (if startup timing happens to align), but more often it would fail.

Assumptions Embedded in the Design

The assistant's approach makes several assumptions, each worth examining.

First, it assumes that depends_on with a one-shot init container is sufficient for correctness. Docker Compose's depends_on (without the condition: service_healthy option) only waits for a container to start, not for it to complete its work. However, because db-init is designed to run a SQL script and then exit, the assistant is implicitly relying on the container's exit code—Docker Compose will wait for the container to finish before starting dependent services. This is correct behavior for this use case.

Second, the assistant assumes that the db-init service itself will succeed. If the SQL script fails (due to a syntax error, a permissions issue, or a network timeout connecting to YugabyteDB), the entire cluster startup will stall. The assistant does not appear to have added retry logic or error handling to db-init at this point.

Third, there is an assumption about idempotency. The db-init service creates the database "filecoingw." But what happens if the cluster is restarted? The database already exists, and the SQL CREATE DATABASE command will fail. This was actually a problem encountered later in the session (as noted in the analyzer summary for chunk 2: "the db-init container was failing with 'database already exists' on subsequent runs"). The assistant would later need to add error suppression to handle this case.

Fourth, the assistant assumes that both Kuri nodes share the same database. This is architecturally correct—the whole point of the shared YugabyteDB is to coordinate object placement across storage nodes. But it means that a single db-init service is sufficient for all nodes, which is efficient.

The Broader Architecture

This message sits within a larger architectural story. The test cluster is implementing a horizontally scalable S3-compatible storage system where:

YugabyteDB serves as the shared metadata store, tracking which objects are stored on which Kuri node
Kuri nodes are independent storage backends, each with its own RIBS blockstore data
S3 frontend proxies (planned but not yet implemented at this stage) would route requests to the appropriate Kuri node The database initialization problem is particularly acute in this architecture because the database is the single source of truth for object placement. If a Kuri node starts before the database is initialized, it cannot register itself, cannot query object locations, and effectively cannot function. The dependency chain is not merely a convenience—it is a correctness requirement.

What This Message Reveals About the Development Process

Message 254 is interesting not for what it contains but for what it represents: the moment when a developer transitions from diagnosing a problem to implementing a solution. The assistant has already identified the root cause (missing database), added the init service, and is now completing the fix by wiring up the dependency graph.

The reasoning section is notably concise: "Now I need to make kuri-1 and kuri-2 depend on db-init so they start after the database is created." This brevity masks the complexity of the reasoning that led to this point. The assistant had to:

Parse the error message from the Kuri node logs
Trace the error through the dependency injection chain (S3Server → S3Server → Blockstore → RIBS → RBS → RbsDB → Database)
Identify that the database "filecoingw" didn't exist
Understand that YugabyteDB's YSQL_DB environment variable doesn't automatically create databases
Design the db-init service as a solution
Recognize that without depends_on, the init service and Kuri nodes would start simultaneously Each of these steps required specific knowledge: familiarity with the YugabyteDB container's behavior, understanding of Docker Compose's startup model, and knowledge of the application's dependency injection framework.

The Input and Output Knowledge

To fully understand this message, a reader needs:

Docker Compose fundamentals: Understanding of services, ports, environment variables, and the depends_on directive
YugabyteDB/PostgreSQL basics: Knowledge that databases must be explicitly created before they can be connected to
The test cluster architecture: Awareness that kuri-1 and kuri-2 are storage nodes that share a YugabyteDB instance
The error context: Knowledge that the Kuri nodes failed with "database does not exist" The message creates new knowledge in the form of a plan: the assistant will edit the docker-compose.yml to add depends_on: db-init to both Kuri node services. This plan will be executed in subsequent messages, ultimately resulting in a working test cluster where services start in the correct order.

Conclusion

Message 254 captures a deceptively simple moment in a complex debugging session. The act of reading a configuration file and planning a dependency directive is, on its surface, unremarkable. But when viewed in context, it represents the intersection of multiple debugging threads: a database that wasn't created, an init service that was added but not wired in, and a cluster that couldn't start because its components didn't wait for their dependencies.

The message is a testament to the fact that in distributed systems, startup ordering is not an afterthought—it is a fundamental design concern. The assistant's reasoning, while expressed in a single sentence, encapsulates a deep understanding of the system's architecture, the failure modes of its components, and the Docker Compose mechanisms available to control service lifecycle. It is a small but critical step toward a working distributed S3 cluster.