The Art of Orchestration: Refining Service Startup Order in a Distributed S3 Test Cluster

Introduction

In the course of building a horizontally scalable S3 storage architecture with three layers—stateless frontend proxies, Kuri storage nodes, and a shared YugabyteDB backend—a seemingly small but critically important message appears. The assistant writes:

## Agent Reasoning The start.sh script is now starting services sequentially, which is good. But the checks for YugabyteDB and db-init are now happening after the services have already been started and waited for. Let me reorganize the script to check services in the right order. [edit] /home/theuser/gw/test-cluster/start.sh Edit applied successfully.

This message, index 413 in the conversation, is a moment of meta-cognitive clarity. It is not about adding a new feature, fixing a compilation error, or designing an architectural component. It is about orchestration hygiene—the realization that a script's control flow has become logically inconsistent, and that the sequence of actions must match the sequence of dependencies. To understand why this matters, we must trace the cascade of failures that led here, the assumptions that were made, and the subtle art of writing startup scripts for distributed systems.

The Cascade of Failures That Preceded This Message

The user had just run ./start.sh /data/fgw2 and encountered a cluster that was only partially operational. The logs told a story of multiple failures:

  1. kuri-2 crashed with a YugabyteDB migration deadlock. Both Kuri nodes had been started simultaneously, and each tried to run database migrations at the same time, colliding on a migration lock table. YugabyteDB, being a distributed SQL database, detected the deadlock and refused one of the connections.
  2. s3-proxy failed with "invalid log level: info." The configuration system expected log levels in component=level format (e.g., *=info or ribs:.*=debug), but the docker-compose file had been set to the bare value info, which failed validation.
  3. Port 9010 was inaccessible despite the Web UI container running and Docker reporting the port as listening. These were three distinct failure modes, but they shared a common root: the startup script had been written in an earlier iteration of the architecture, and as the system grew more complex, the script had not kept pace. The assistant's first response (message 409) correctly identified all three issues. The second response (message 410) dug into the log level parsing code to understand the format requirement. Message 411 applied two edits: fixing the log level in docker-compose.yml and beginning to restructure start.sh for sequential startup. Message 412 read the partially-edited script and noted that the service checks were now misplaced. And then came message 413—the subject of this article.

What This Message Actually Accomplishes

On its surface, message 413 is almost trivial: the assistant reads its own prior edit, notices an inconsistency, and applies a fix. The edit is described in a single line: "Let me reorganize the script to check services in the right order." The tool output confirms success: "Edit applied successfully."

But the significance lies in what the assistant recognized. The prior edit (message 411) had restructured the startup sequence to be:

docker-compose up -d yugabyte db-init
sleep 10
docker-compose up -d kuri-1
# ... presumably kuri-2 next

However, the health check logic—the code that polls YugabyteDB for readiness, waits for database initialization to complete, and verifies that Kuri nodes are running—was still positioned at the bottom of the script, after all services had been started. This meant the script would:

  1. Start YugabyteDB and db-init
  2. Wait 10 seconds
  3. Start kuri-1
  4. Start kuri-2 (implicitly)
  5. Then check if YugabyteDB is healthy
  6. Then check if database initialization succeeded
  7. Then check if Kuri nodes are running This ordering is logically backwards. You cannot meaningfully check whether database initialization succeeded after you have already started the storage nodes that depend on that initialization. If db-init had failed, the Kuri nodes would have been launched into a broken environment, and the health checks would only report the failure after the fact—potentially leaving containers running in a degraded state. The assistant recognized this flaw and reorganized the checks to occur between the service startups, creating a proper dependency chain:
  8. Start YugabyteDB
  9. Wait and check that YugabyteDB is healthy
  10. Start db-init
  11. Wait and check that database initialization completed
  12. Start kuri-1
  13. Wait and check that kuri-1 is running
  14. Start kuri-2
  15. Wait and check that kuri-2 is running
  16. Start s3-proxy and webui
  17. Final readiness summary This is the difference between a script that launches containers and a script that orchestrates a distributed system. The former fires and forgets; the latter respects the dependency graph.

The Thinking Process: What the Assistant Realized

The assistant's reasoning, as recorded in the message, reveals a specific cognitive step: "The start.sh script is now starting services sequentially, which is good. But the checks for YugabyteDB and db-init are now happening after the services have already been started and waited for."

The phrase "which is good" indicates approval of the sequential startup direction. The "but" signals the discovery of a residual inconsistency. The assistant is performing a mental simulation of the script's execution flow—tracing through the lines in order and asking, at each step, whether the preconditions for that step have been verified.

This is a form of control flow analysis that experienced developers do almost unconsciously. The assistant is effectively asking:

Assumptions Made and Implicit Knowledge

Several assumptions underlie this message:

Assumption 1: Sequential startup prevents migration deadlocks. The assistant assumes that starting Kuri nodes one at a time, rather than simultaneously, will avoid the YugabyteDB migration lock conflict that killed kuri-2. This is a reasonable assumption given that the deadlock occurred on a migrations_locks table—a table designed for exclusive locking during schema migrations. If only one node runs migrations at a time, there can be no deadlock. However, this assumption depends on the migration code correctly releasing the lock after completion, and on the second node recognizing that migrations have already been applied rather than trying to re-acquire the lock.

Assumption 2: Health checks are reliable indicators of readiness. The assistant assumes that checking "is YugabyteDB healthy?" and "did db-init exit successfully?" are sufficient preconditions for starting Kuri nodes. In practice, database readiness is more nuanced—the node might be healthy but still replicating data, or the connection pool might not be fully initialized. The script uses a combination of docker inspect health status and exit code checking, which is reasonable for a test cluster but might not catch all edge cases.

Assumption 3: The user's data directory is reusable. The script includes logic to skip permission updates if data directories already exist, and the db-init container reports "Database may already exist" as a non-fatal message. The assistant assumes that reusing existing data directories is safe, which is true for test clusters but could mask corruption in production.

Assumption 4: The set -e shell option is sufficient error handling. The script uses set -e to exit on any command failure. The assistant assumes that this, combined with explicit health checks, provides adequate error handling. However, set -e has well-known subtleties (it does not apply inside command substitutions, pipeline components other than the last, or conditions in if/while statements), and a production script would need more robust error management.

Input Knowledge Required to Understand This Message

To fully grasp what is happening in message 413, a reader needs:

  1. Knowledge of the architecture: The three-layer hierarchy of S3 proxy → Kuri nodes → YugabyteDB, and the fact that Kuri nodes share a database but need isolated keyspaces.
  2. Knowledge of the failure modes: The migration deadlock that occurs when two Kuri nodes initialize simultaneously, and the log level format error that prevents s3-proxy from starting.
  3. Knowledge of Docker Compose orchestration: How docker-compose up -d works, how health checks are configured, and how container dependencies are managed.
  4. Knowledge of shell scripting patterns: The difference between starting services and checking their readiness, the use of sleep for timing, and the importance of ordering in startup scripts.
  5. Knowledge of the specific codebase: The configureLogLevels() function in configuration/config.go that requires component=level format, and the gen-config.sh script that generates per-node settings.

Output Knowledge Created by This Message

Message 413 produces a concrete artifact: an improved start.sh script with properly ordered service startup and health checks. But it also creates several forms of knowledge that extend beyond the file:

  1. A pattern for sequential dependency resolution: The script now demonstrates a general approach to starting multi-service distributed systems—start infrastructure, verify, start dependent services, verify, repeat. This pattern can be applied to other test clusters and production deployments.
  2. A debugging heuristic: The realization that "checks are happening after services have already started" is a reusable insight. When debugging startup failures, one of the first things to examine is whether preconditions are verified before dependent services are launched.
  3. Documentation of the startup order: The script, through its structure, now documents the correct dependency order: YugabyteDB → db-init → kuri-1 → kuri-2 → s3-proxy/webui. This is valuable for anyone who needs to understand or modify the cluster.
  4. A regression test for the architecture: The fact that sequential startup is required reveals something about the architecture—namely, that the migration system does not support concurrent initialization. This is an architectural constraint that might need to be addressed in the future if the system needs to scale to many nodes starting simultaneously.

Mistakes and Incorrect Assumptions

While the assistant's analysis is largely correct, there are some potential issues worth examining:

The sleep-based timing is fragile. The script uses sleep 10 to wait for YugabyteDB to be ready after starting it. This is a common but brittle pattern—on a loaded system or with different hardware, 10 seconds might be too little or unnecessarily long. A more robust approach would poll the health endpoint in a loop with a timeout, but the assistant did not implement this.

The sequential startup solves one problem but may create others. Starting kuri-1 and kuri-2 sequentially ensures they don't collide on migrations, but it also means the second node starts with a warm database cache (since the first node already ran queries). This asymmetry could mask performance issues that only appear when both nodes start simultaneously in production.

The health check for db-init relies on exit codes. The db-init container is configured to run once and exit. The script checks that it exited successfully (status 0). However, the container's log shows "Database may already exist" as a normal message, which means the script cannot distinguish between "initialization succeeded" and "initialization was skipped because the database already exists." For a test cluster this is acceptable, but it means the script would not catch a corrupted but existing database.

The port 9010 issue was never fully diagnosed. The user reported that port 9010 was listening but inaccessible in the browser. The assistant noted this but did not resolve it in the messages leading up to message 413. The fix for the log level and the sequential startup were prioritized, but the Web UI accessibility issue remained open.

The Deeper Lesson: Scripts as Executable Documentation

Message 413 is a small moment in a long coding session, but it illustrates a profound truth about distributed systems development: the startup script is not just a convenience—it is executable documentation of the system's dependency graph.

When an architect designs a three-layer system with stateless proxies, storage nodes, and a shared database, they create a mental model of how these components relate. The startup script translates that mental model into a sequence of concrete actions. If the script gets the order wrong, it reveals either a bug in the script or a misunderstanding of the architecture.

In this case, the assistant had correctly understood the architecture but had written the script in a way that separated the "starting" phase from the "checking" phase. The edit in message 413 reunites these phases, creating a script that mirrors the actual dependency chain: database first, then storage nodes, then proxies.

This is the kind of refactoring that is easy to overlook in the heat of debugging. When kuri-2 is crashing, the s3-proxy is failing to configure, and the Web UI is unreachable, the temptation is to fix each symptom individually. But the assistant took a step back, looked at the script's overall structure, and recognized a logical flaw that would have caused subtle issues even after all the individual bugs were fixed.

Conclusion

Message 413 is a testament to the value of meta-cognitive awareness in software development. The assistant did not just fix bugs—it recognized that the structure of the fix was incomplete, and that the ordering of operations in a startup script must respect the dependency graph of the system it orchestrates.

The edit itself is small: reorganizing health checks to occur between service startups rather than after all services have been launched. But the reasoning behind it touches on fundamental principles of distributed systems, shell scripting, and operational hygiene. It is a reminder that in complex systems, the order of operations is not a detail—it is the architecture made visible.