The Dependency Dilemma: Orchestrating Container Startup in a Distributed S3 Test Cluster

Introduction

In the course of building a horizontally scalable S3-compatible storage architecture for the Filecoin Gateway, a seemingly mundane technical decision—how to order container startup in a Docker Compose file—became a microcosm of the deeper challenges inherent in distributed systems design. The message at index 255 of this coding session captures a moment of technical deliberation that reveals much about the nature of infrastructure debugging: the gap between what we expect containers to do and what orchestration tools actually guarantee. This article examines that single message in detail, unpacking the reasoning, assumptions, and architectural thinking that went into a five-paragraph deliberation about a depends_on directive.

The Message in Full

The subject message, written by the AI assistant, reads as follows:

## Agent Reasoning I need to update the depends_on for kuri-1 and kuri-2 to also wait for db-init. But db-init is a one-shot container (restart: "no"), so I need to make sure the depends_on works correctly. Actually, in Docker Compose, you can't depend on a one-shot container completing successfully - you can only depend on it being started.

Let me think about this differently. A better approach might be to: 1. Have kuri-1 and kuri-2 depend on yugabyte being healthy 2. Add a retry mechanism in the application or just let it crash and restart until the DB is ready

Actually, looking at the error again, the issue is that the database doesn't exist. The simplest fix is to just have the Kuri nodes retry on startup. But that's harder to implement.

Another approach: Add the db-init as a dependency but also add a small delay or healthcheck to ensure it completes. But that's complex.

Simplest fix: Just update the start.sh script to create the database after YugabyteDB is healthy, before starting the Kuri nodes. But that won't work well with docker-compose up -d.

Best approach for now: Modify the docker-compose to have Kuri depend on db-init, and make db-init exit successfully after creating the DB. The issue is that Docker Compose doesn't wait for the container to complete, just for it to be created.

Actually, let me just add db-init to the depends_on and hope that the timing works out. The db-init should run quickly after yugabyte is healthy. [edit] /home/theuser/gw/test-cluster/docker-compose.yml Edit applied successfully.

This message is notable not for its final decision—which is essentially a pragmatic hack—but for the reasoning process it exposes. The assistant walks through no fewer than five distinct approaches to solving the startup ordering problem, evaluating each against criteria of simplicity, reliability, and compatibility with the existing Docker Compose workflow.

Context: The Problem That Preceded This Message

To understand why this message was written, we must trace the chain of events that led to it. The user had been testing a Docker Compose-based test cluster for the Filecoin Gateway's distributed S3 storage system. The cluster consisted of two Kuri storage nodes (kuri-1 and kuri-2) sharing a YugabyteDB database for metadata coordination. When the user ran ./start.sh /data/fgw2, both Kuri nodes failed to start. The error logs revealed the root cause:

create postgres migrations driver: pq: database "filecoingw" does not exist

The Kuri nodes were attempting to connect to a PostgreSQL database called "filecoingw" inside YugabyteDB, but that database had never been created. The YugabyteDB container had environment variables like YSQL_DB: filecoingw and YCQL_KEYSPACE: filecoingw, but these did not automatically provision the database—they merely configured the YugabyteDB server to expect those names. The actual database creation required an explicit SQL command: CREATE DATABASE filecoingw; or its equivalent.

This is a classic infrastructure pitfall: the assumption that configuration directives automatically perform the actions they describe. Setting YSQL_DB: filecoingw in a Docker Compose environment block feels like it should create the database, much the same way that setting POSTGRES_DB: mydb in a standard PostgreSQL container does. But YugabyteDB's yugabyted startup process does not interpret YSQL_DB as a creation directive—it is simply a default database name for client connections, and if that database doesn't exist, connections fail.

The assistant's first response (message 251) correctly identified the issue and began exploring solutions. An initial attempt modified the YugabyteDB service definition to include a startup command that would create the database. But the assistant quickly realized this approach had a flaw: the healthcheck might pass before the database creation command completed, creating a race condition. This led to the second attempt (message 253): creating a separate db-init service—a one-shot container that would run after YugabyteDB was healthy, execute the database creation SQL, and then exit.

The Reasoning Process: A Taxonomy of Approaches

The subject message (255) is the third iteration of this debugging cycle. The assistant has already added the db-init service to the Docker Compose file in message 253, but now faces a new problem: how to ensure the Kuri nodes wait for db-init to finish before they start.

The assistant's reasoning enumerates five approaches, each with distinct trade-offs:

Approach 1: Docker Compose depends_on with condition: service_completed_successfully. The assistant correctly notes that Docker Compose's depends_on directive, when applied to a one-shot container (one with restart: "no"), does not wait for the container to complete successfully. It only waits for the container to be created—a subtle but critical distinction. Docker Compose's condition: service_started is the default, and condition: service_healthy requires a healthcheck. There is no built-in condition: service_completed_successfully for one-shot containers in standard Docker Compose (though it exists in Compose v2.20+ with depends_on conditions). This means adding db-init to the depends_on list would not actually solve the problem.

Approach 2: Retry mechanism in the application. The assistant considers having the Kuri nodes simply crash and restart until the database exists. This is a common pattern in container orchestration—the "crash-loop backoff" approach where containers are expected to fail during initialization and be restarted by the orchestrator until dependencies are available. The assistant dismisses this as "harder to implement," which is a pragmatic assessment: modifying the Kuri application code to add retry logic would require changes to the Go source code, a new build, and additional testing.

Approach 3: Healthcheck with delay. Adding a healthcheck to db-init that only passes after the database is created, then having Kuri nodes depend on that healthcheck. The assistant notes this is "complex," likely because healthchecks on one-shot containers are unusual and Docker Compose's healthcheck polling may not work cleanly with containers that exit after completing their work.

Approach 4: Script-based sequencing. Moving the database creation logic into start.sh, so the shell script creates the database after YugabyteDB is healthy and before starting the Kuri nodes. The assistant identifies a key problem: "that won't work well with docker-compose up -d." The -d (detached) mode starts all containers simultaneously, so a script that runs before docker-compose up would have to start YugabyteDB, wait for it, create the database, and then start the remaining containers—a multi-step process that diverges from the simple docker-compose up -d workflow.

Approach 5 (selected): Hope for the best. The assistant ultimately chooses to add db-init to the depends_on list despite knowing it won't guarantee completion, relying on the assumption that "db-init should run quickly after yugabyte is healthy." This is a deliberate trade-off of correctness for simplicity—a "good enough" solution that will work in practice even if it doesn't provide a formal guarantee.

Assumptions and Their Validity

This message is built on several assumptions, some explicit and some implicit:

Assumption 1: Docker Compose depends_on does not wait for one-shot container completion. This is correct for the version of Docker Compose being used (v2.x). The depends_on directive with default settings only ensures that the depended-upon container has been started, not that it has completed its work. This is a well-known limitation of Docker Compose that often surprises developers transitioning from single-host orchestration to more sophisticated systems like Kubernetes, where init containers provide exactly this guarantee.

Assumption 2: The db-init container will execute quickly. This is a reasonable assumption—creating a single database in YugabyteDB is a fast operation, typically taking less than a second once the database server is ready. However, it's an assumption about timing that could be violated under conditions like resource contention, slow disk I/O, or network latency.

Assumption 3: The race condition window is small enough to ignore. The assistant implicitly assumes that even if Kuri nodes start before db-init completes, the timing gap will be negligible. This is the weakest assumption, as it depends on factors outside the assistant's control: Docker Compose's startup ordering, the speed of container initialization, and the specific timing of database creation.

Assumption 4: A crashed-and-restarted Kuri node is acceptable. By not implementing retry logic, the assistant implicitly accepts that Kuri nodes may crash once on the first startup attempt and then succeed on restart. This is a reasonable assumption in a test cluster context, where operational tolerance for transient failures is high.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

Docker Compose semantics. The distinction between depends_on with condition: service_started (the default) and condition: service_healthy is crucial. Without understanding that Docker Compose does not natively support waiting for one-shot container completion, the assistant's reasoning appears confused rather than pragmatic.

YugabyteDB initialization behavior. Understanding that YSQL_DB and YCQL_KEYSPACE environment variables do not automatically create databases is essential. Many developers coming from the standard PostgreSQL Docker image (where POSTGRES_DB does create the database) would make the same mistaken assumption.

Container lifecycle management. The concept of "one-shot containers" (containers with restart: "no" that perform a single task and exit) versus long-running services is fundamental to the reasoning. The assistant correctly identifies that these two container types have different lifecycle semantics that complicate dependency management.

The application architecture. The Kuri nodes are Go applications that use dependency injection (via the fx framework) to build their component graph. The error trace shows a deep chain of dependency resolution: StartS3Server → MakeS3Server → ribsBlockstore → makeRibs → Open → NewRibsDB → makeSqlDb, where the failure at the bottom (database doesn't exist) cascades upward. Understanding this architecture helps explain why the assistant considers modifying the application code as "harder to implement"—it would require changes at the deepest layer of the dependency graph.

Output Knowledge Created

This message creates several forms of output knowledge:

A documented design decision. The Docker Compose file now includes a db-init service with depends_on linking it to both Kuri nodes. While imperfect, this represents a deliberate architectural choice that future developers can examine, critique, and improve.

An explicit trade-off analysis. The assistant's reasoning documents why the "simplest" approach was chosen over more robust alternatives. This is valuable knowledge for anyone who later encounters startup ordering issues in this cluster—they can see that the trade-off was conscious rather than accidental.

A pattern for database initialization. The db-init service pattern—a one-shot container that runs after the database is healthy but before application containers start—is a reusable pattern that can be applied to other services in the architecture.

A boundary condition in Docker Compose usage. The message implicitly documents a limitation of Docker Compose: the inability to reliably sequence one-shot containers before dependent services. This is knowledge that may inform future decisions about whether to use Docker Compose or a more sophisticated orchestrator for production deployments.

The Thinking Process: A Window into Debugging Methodology

The most valuable aspect of this message is the thinking process it exposes. The assistant does not simply implement a solution; it iterates through a design space, evaluating options against multiple criteria:

Correctness: Does the approach guarantee the database exists before Kuri nodes start?
Simplicity: How many changes are required, and how complex are they?
Compatibility: Does the approach work with the existing docker-compose up -d workflow?
Maintainability: Would the approach introduce fragile timing dependencies? This multi-criteria evaluation is characteristic of experienced infrastructure engineers. The assistant recognizes that no single approach optimizes all criteria simultaneously, so it must make a judgment call based on the context: this is a test cluster, not a production deployment. Reliability requirements are lower, and simplicity is prioritized. The final decision—"hope that the timing works out"—is not ignorance or laziness. It is a calculated risk based on the understanding that: - The db-init container starts after YugabyteDB is healthy (due to depends_on) - Database creation is a fast operation - Even if Kuri nodes start slightly early, they will crash and be restarted by Docker Compose - The test cluster can tolerate transient startup failures This kind of pragmatic decision-making is often invisible in final code changes, which show only the clean result. The reasoning message preserves the decision process, making it available for future review and critique.

Conclusion

The message at index 255 is a small moment in a larger debugging session, but it encapsulates themes that resonate throughout distributed systems engineering: the gap between declarative intent and actual behavior, the challenge of container startup ordering, and the art of making pragmatic trade-offs under uncertainty. The assistant's five attempted approaches to a single depends_on problem illustrate that even "simple" infrastructure decisions require deep knowledge of tool semantics, application architecture, and operational context. The final choice—imperfect but practical—reflects the reality that most infrastructure work is not about finding perfect solutions but about finding solutions that are good enough for their context. This message, preserved in the conversation log, serves as a documentation artifact that future developers can learn from, critique, and ultimately improve upon when the test cluster evolves into a production system.