The Database That Wasn't There: Debugging a Missing PostgreSQL Database in a Distributed S3 Test Cluster

Introduction

In the messy, iterative process of building distributed systems, few moments are as instructive as the first time you try to actually run what you've built. The architecture diagrams look clean, the code compiles, the configuration files are in place—and then the containers start, the logs fill with red, and the whole thing collapses into a heap of "database does not exist" errors. This is the story of one such moment: a single user message that exposed a critical gap between configuration and reality in a horizontally scalable S3-compatible storage system.

The message in question is a straightforward bug report. The user—the architect and primary developer of a Filecoin Gateway project—runs a test cluster startup script, watches both storage nodes fail to initialize, and pastes the terminal output into the conversation. On its surface, it is a simple operational failure: the PostgreSQL database that the storage nodes need to connect to hasn't been created yet. But beneath that surface lies a rich story about assumptions in infrastructure configuration, the gap between environment variables and actual database provisioning, the cascading nature of dependency injection failures, and the delicate dance between development and operations in containerized systems.

The Message: A Bug Report in Terminal Form

The message opens with the user executing the test cluster startup script:

Need small fix: theuser@biryani  ~/gw/test-cluster  pgf-port ● ? ↑2  ./start.sh /data/fgw2

What follows is the complete output of the startup process. The script prints a banner announcing "FGW Test Cluster (2 Storage Nodes)," confirms that the Docker image exists, initializes data directories, and launches the cluster via docker-compose up -d. The Docker Compose engine reports four containers created: a network, a YugabyteDB instance (which reaches healthy status), and two Kuri storage nodes (which are created but never actually start running). The script's health check loop finds both Kuri nodes in a non-running state and reports failure.

The user then immediately checks the logs for each node:

docker-compose logs kuri-1
docker-compose logs kuri-2

Both logs tell the same story. After initializing IPFS node identities, each Kuri node encounters a catastrophic build failure. The error message is a masterpiece of nested dependency injection tracebacks:

could not build arguments for function "github.com/CIDgravity/filecoin-gateway/server/s3".StartS3Server
  → failed to build *s3.S3Server
    → failed to build *ribsbstore.Blockstore
      → failed to build iface.RIBS
        → failed to build iface.RBS
          → failed to build *rbstor.RbsDB
            → failed to build sqldb.Database
              → received non-nil error from function "github.com/CIDgravity/filecoin-gateway/integrations/kuri/ribsplugin".makeSqlDb
                → create postgres migrations driver: pq: database "filecoingw" does not exist

At the very bottom of this chain, the root cause: pq: database "filecoingw" does not exist. The Kuri nodes are trying to connect to a PostgreSQL database called "filecoingw" via YugabyteDB's PostgreSQL-compatible interface, but that database hasn't been created yet.

The Context: What Led to This Message

To understand why this message matters, we need to understand the architecture being built. The Filecoin Gateway project is implementing a horizontally scalable S3-compatible storage system. The architecture follows a three-layer design:

  1. Stateless S3 frontend proxies that handle request routing and load balancing
  2. Kuri storage nodes that maintain independent RIBS blockstore data
  3. A shared YugabyteDB that tracks object placement across nodes The Kuri nodes are not simple storage servers. They are wrappers around IPFS Kubo (the Go implementation of IPFS), extended with a plugin system that provides S3 API compatibility, a web UI for monitoring, and a sophisticated blockstore backed by the RIBS (Redundant Independent Block Store) system. Each Kuri node connects to a shared YugabyteDB instance that serves as both a PostgreSQL-compatible SQL database and a Cassandra-compatible YCQL (Yugabyte CQL) database for different parts of the system. The test cluster was built incrementally through an iterative debugging process. Earlier in the session, the assistant had made a significant architectural error by configuring Kuri nodes as direct S3 endpoints with a single shared configuration. The user corrected this, pointing to the architecture roadmap that specified separate stateless frontend proxy nodes. The assistant then redesigned the test cluster to use a proper three-layer hierarchy, but the frontend proxy binary wasn't built yet, so the immediate test setup was simplified to two Kuri nodes connecting directly to YugabyteDB. This simplification set the stage for the database creation problem. The Docker Compose configuration included environment variables for the YugabyteDB container:
environment:
  YSQL_DB: filecoingw
  YCQL_KEYSPACE: filecoingw

The assumption was that these environment variables would cause YugabyteDB to automatically create the specified database and keyspace on startup. This assumption turned out to be incorrect.

The Root Cause: A Gap Between Configuration and Reality

The error message tells us exactly what went wrong, but understanding why requires knowledge of how YugabyteDB handles database creation. YugabyteDB is a distributed SQL database that is compatible with both PostgreSQL (via the YSQL API) and Cassandra (via the YCQL API). When you set environment variables like YSQL_DB and YCQL_KEYSPACE, these are not universally supported across all YugabyteDB Docker images and configurations.

In some YugabyteDB deployments, these environment variables are used by the yugabyted process to automatically create databases on first startup. However, in the specific Docker image being used, or with the specific configuration applied, these variables were not triggering database creation. The YugabyteDB container started successfully and passed its health check, but the "filecoingw" database simply didn't exist inside it.

This is a classic infrastructure pitfall: environment variables that seem like they should cause an action to happen, but actually only serve as configuration hints that the application may or may not honor. The YSQL_DB variable might be intended for use by an initialization script that wasn't included in the container, or it might be read by a different version of the YugabyteDB startup process. Without explicit database creation logic, the variable is just a string sitting in the container's environment, doing nothing.

The Cascade of Failure

What makes this error particularly instructive is the way it propagates through the Kuri node's dependency injection system. The Kuri node uses Uber's fx dependency injection framework (a fork of the popular dig library) to wire together its components at startup. The dependency chain is:

  1. Database connection (sqldb.Database) — the lowest-level dependency, connecting to YugabyteDB's PostgreSQL interface
  2. RIBS database (*rbstor.RbsDB) — wraps the SQL database connection
  3. RIBS blockstore (iface.RBS) — the block-level storage interface
  4. Full RIBS interface (iface.RIBS) — the complete storage abstraction
  5. Blockstore wrapper (*ribsbstore.Blockstore) — adapts RIBS to IPFS blockstore conventions
  6. S3 server (*s3.S3Server) — the S3-compatible API server When the database doesn't exist, the first link in this chain fails. But the error message doesn't just say "database not found" and stop. Because fx builds the entire dependency graph before starting any component, the failure propagates upward through every layer, producing a deeply nested error that looks far more catastrophic than it actually is. This is a common pattern in systems that use dependency injection frameworks: a simple, fixable problem at the foundation produces a terrifying error message at the top. The S3 server can't start because the blockstore can't start because the RIBS interface can't start because the database can't connect because the database doesn't exist. Each layer adds another level of nesting, making the error look like a cascading system failure rather than a single missing database.

The Assumptions Embedded in the Infrastructure

The message reveals several assumptions that were baked into the test cluster configuration:

Assumption 1: Environment variables auto-create databases. The most critical assumption was that YSQL_DB: filecoingw would cause YugabyteDB to create the database automatically. This assumption was reasonable—many database containers do support such auto-creation—but it was not validated. The YugabyteDB documentation and behavior around this feature are nuanced, and the specific container configuration didn't support it.

Assumption 2: The health check guarantees full readiness. The Docker Compose configuration included a health check for YugabyteDB, and the startup script waited for the health check to pass before proceeding. However, the health check only verified that the YugabyteDB process was running and accepting connections—it didn't verify that the specific database "filecoingw" existed. The container passed its health check while still being unable to serve the application's needs.

Assumption 3: The Kuri nodes would retry or handle missing databases gracefully. The Kuri nodes were configured to fail immediately if the database didn't exist, with no retry logic. This is actually a reasonable design choice—fail fast is better than hanging indefinitely—but it meant that any timing issue between database creation and node startup would cause a hard failure.

Assumption 4: The startup order in Docker Compose would be sufficient. The depends_on configuration in Docker Compose controlled the order in which containers were started, but Docker Compose's default behavior only ensures that dependent containers are started after their dependencies—it doesn't wait for the dependency to be fully ready. The health check configuration added readiness checking for YugabyteDB, but as noted, the health check wasn't specific enough.

The Input Knowledge Required to Understand This Message

To fully grasp what this message is communicating, a reader needs knowledge spanning several domains:

Docker Compose orchestration: Understanding how depends_on, health checks, and container lifecycle interact. The fact that containers can be "created" but not "running," and that Docker Compose's default dependency behavior doesn't guarantee readiness.

YugabyteDB and PostgreSQL: Understanding that YugabyteDB provides a PostgreSQL-compatible SQL interface, that databases must be explicitly created before they can be connected to, and that environment variables like YSQL_DB are not universally supported for auto-creation.

The Kuri node architecture: Understanding that Kuri is not a standalone storage server but an IPFS Kubo wrapper with a plugin system, that it uses dependency injection to wire together its components, and that the S3 API, web UI, and blockstore are all started as part of a single process.

Go dependency injection patterns: Understanding the fx framework and how it builds dependency graphs, propagates errors, and produces nested error messages that can obscure the root cause.

The Filecoin Gateway project's roadmap: Understanding that the test cluster is part of a larger effort to build a horizontally scalable S3 architecture, that frontend proxies are a separate component from storage nodes, and that the current test setup is a simplified version of the final architecture.

The Output Knowledge Created by This Message

This message, despite being a simple bug report, creates significant knowledge:

The database must be explicitly created. The immediate output is the knowledge that YugabyteDB, in this configuration, does not auto-create databases from environment variables. An explicit database initialization step is required.

The dependency injection chain is documented. The nested error message serves as a map of the Kuri node's dependency graph, showing exactly which components depend on which, and how failures propagate. This is valuable documentation for anyone working on the system.

The startup script has a gap. The start.sh script's health check loop only checks YugabyteDB's generic readiness, not the specific database existence. This is a gap that needs to be filled.

The test cluster is not yet functional. The most practical output is the knowledge that the test cluster, as configured, cannot start. This triggers the next iteration of debugging and fixes.

The infrastructure needs an initialization service. The solution that emerges from this message (in the subsequent conversation) is the addition of a db-init service that explicitly creates the database and keyspace before the Kuri nodes attempt to connect. This becomes a permanent part of the test cluster infrastructure.

The Thinking Process Visible in the Message

The message itself doesn't contain explicit reasoning—it's a terminal output paste—but the user's actions reveal their thinking process:

Step 1: Execute the startup script. The user runs ./start.sh /data/fgw2, expecting the cluster to start successfully. This is a test of the infrastructure the assistant built.

Step 2: Observe the failure. The script reports that both Kuri nodes are not running. The user doesn't stop at this surface-level failure—they immediately dig deeper.

Step 3: Check the logs. The user runs docker-compose logs kuri-1 and docker-compose logs kuri-2 to see what actually went wrong. This shows a systematic debugging approach: don't just report that it failed, find out why it failed.

Step 4: Present the evidence. The user pastes the complete output, including the full error tracebacks, into the conversation. This is not just a complaint—it's a carefully documented bug report that gives the assistant everything needed to diagnose and fix the problem.

Step 5: Frame the request. The message begins with "Need small fix," which is both a request and a judgment. The user has already determined that this is a small, fixable problem, not a fundamental architectural issue. This framing shapes the assistant's response, directing it toward a targeted fix rather than a redesign.

The Broader Significance

This message is a microcosm of the challenges in building distributed systems. It illustrates several universal truths:

Configuration is not execution. Setting an environment variable is not the same as running a command. The gap between what we expect a configuration to do and what it actually does is a constant source of bugs.

Health checks are only as good as their criteria. A health check that verifies "is the process running?" is very different from one that verifies "is the required resource available?" Choosing the right health check criteria is a design decision with real consequences.

Dependency injection magnifies simple failures. A missing database at the bottom of a dependency chain produces a spectacular error at the top. This can make simple problems look catastrophic and obscure the actual root cause.

Test infrastructure is production infrastructure. The test cluster, despite being "just for testing," needs the same rigor as production infrastructure. Database initialization, startup ordering, and health checking are not optional—they are essential for a working system, even in development.

The user's debugging approach is exemplary. Rather than reporting "the cluster doesn't work" and stopping, the user checked the logs, identified the specific error, and presented the complete evidence. This is the kind of debugging behavior that makes collaborative development productive.

Conclusion

The message "Need small fix" is deceptively simple. On its surface, it's a user reporting that a test cluster failed to start because a database didn't exist. But in its context—as part of an iterative process of building a distributed S3 storage system—it reveals the gap between infrastructure assumptions and reality, the cascading nature of dependency failures, and the importance of explicit database provisioning in containerized environments.

The fix that follows—adding a db-init service to explicitly create the database before the Kuri nodes start—is straightforward. But the knowledge created by this message extends far beyond that single fix. It documents the dependency chain of the Kuri node, validates (or invalidates) assumptions about YugabyteDB's behavior, and establishes a pattern for database initialization that will be reused throughout the project.

In the end, this message is a reminder that in distributed systems, the simplest things—like a database that doesn't exist—can bring the entire architecture to its knees. And the only way to find these gaps is to actually run the system, watch it fail, and read the logs.