The Two-Bug Fix: When Log Level Format and Database Deadlock Collide in a Distributed S3 Cluster

Introduction

In the course of building a horizontally scalable S3-compatible storage architecture, every deployment attempt reveals hidden assumptions. The message at index 411 of this coding session is a compact but revealing example: a single assistant response that fixes two unrelated bugs—one a trivial configuration string format, the other a subtle distributed systems race condition—that together had brought a two-node test cluster to its knees. The message reads:

[assistant] ## Agent Reasoning The log level format requires component=level. Just info is invalid. Let me fix the docker-compose.yml to use the correct format or remove it entirely. I'll also update start.sh to start Kuri nodes sequentially. [edit] /home/theuser/gw/test-cluster/docker-compose.yml Edit applied successfully. [edit] /home/theuser/gw/test-cluster/start.sh Edit applied successfully.

This short message, barely a dozen lines of reasoning and two file edits, encapsulates a debugging session that required tracing through Docker Compose configurations, Go source code, distributed database migration internals, and shell scripting. It is a microcosm of the kind of systems-level debugging that defines distributed infrastructure work.

The Context: A Cluster That Wouldn't Start

The user had just attempted to launch a test cluster for the Filecoin Gateway's horizontally scalable S3 architecture. The architecture follows a three-layer design: stateless S3 frontend proxies on port 8078, independent Kuri storage nodes (each with isolated RIBS keyspaces), and a shared YugabyteDB cluster for metadata. The test cluster was configured with two Kuri nodes, a shared YugabyteDB instance, a web UI, and an S3 proxy—all orchestrated via Docker Compose.

The startup attempt (message 408) revealed two critical failures. First, the s3-proxy container logged: Failed to load configuration: invalid log level: info. Second, kuri-2 failed to start entirely, with a deeply nested error trace culminating in a YugabyteDB deadlock: pq: deadlock detected (query layer retry isn't possible because data was already sent...). Meanwhile, kuri-1 had started successfully but also showed a warning about Configuration load failed: %w MinimunRetriveable count greater than MinimumReplica: 5 > 1—a separate configuration issue that was non-fatal.

The user's terminal output showed the cluster status: only kuri-1 and the web UI were running. kuri-2 and the s3-proxy were absent from the docker-compose ps output. The assistant had to diagnose both failures from the log output alone.

The First Bug: A Single Word That Broke Configuration Loading

The s3-proxy failure was the more straightforward of the two. The error message invalid log level: info pointed directly to the RIBS_LOGLEVEL environment variable. In the docker-compose.yml, the S3 proxy service had been configured with RIBS_LOGLEVEL=info—a seemingly innocent setting.

But the assistant's reasoning reveals the crucial detail: the log level configuration parser expects a component=level format, not a bare level name. Tracing through the source code in configuration/config.go, the configureLogLevels() function splits the LogLevel string by commas, then splits each segment by =. It checks if len(s) != 2 and returns an error if the format is wrong. A bare info produces a single-element slice after splitting by =, triggering the validation error.

The correct format would be *=info (apply to all components) or ribs=info (apply only to RIBS components). The assistant's reasoning considered two options: removing the variable entirely (letting the default apply) or fixing the format. The actual edit likely replaced RIBS_LOGLEVEL=info with a valid format or removed it.

This is a classic configuration API design issue. The developer who wrote configureLogLevels() chose a compound format (component=level,component2=level2) that allows fine-grained control over individual subsystem log levels. But this design creates a trap: a user who writes info instead of *=info gets a hard failure rather than a sensible default. The error message is clear—invalid log level: info—but it doesn't tell the user what the correct format should be. The assistant had to read the source code to understand the expected format.

The Second Bug: The Race That Killed kuri-2

The kuri-2 failure was far more interesting. The error log showed a chain of dependency injection failures spanning six layers of abstraction:

StartS3Server → MakeS3Server → ribsBlockstore → makeRibs → Open → NewRibsDB → makeSqlDb

At the bottom of this chain was a database migration deadlock: failed to set migration lock in line 0: INSERT INTO migrations_locks (lock_id) VALUES ($1) (details: pq: deadlock detected...).

The root cause was architectural: both Kuri nodes were configured to use the same YugabyteDB keyspace for their RIBS data. When both started simultaneously, each tried to acquire the migration lock on the shared migrations_locks table. YugabyteDB, being a distributed SQL database with serializable snapshot isolation, detected a deadlock and aborted one of the transactions. kuri-2 lost the race and failed to initialize.

The assistant's reasoning correctly identified this as a "migration race." The fix was to start the Kuri nodes sequentially rather than in parallel. The updated start.sh would first start the infrastructure (YugabyteDB and db-init), wait for it to be ready, then start kuri-1, wait for its migrations to complete, and only then start kuri-2. This eliminates the race condition by ensuring only one node attempts database migrations at a time.

This fix, however, is a workaround rather than a true solution. The deeper architectural issue—that both nodes share a database keyspace and compete for migration locks—was later addressed in subsequent work by segregating keyspaces per node. But for the immediate goal of getting a test cluster running, sequential startup was the pragmatic choice.

The Reasoning Process: How the Assistant Connected the Dots

What makes this message instructive is the reasoning process visible in the preceding messages. In message 409, the assistant listed three observed issues: kuri-2's deadlock, the s3-proxy's log level error, and the user's inability to access port 9010 in the browser. It then formulated a plan: investigate the migration deadlock, check the log level configuration, and diagnose the port issue.

Message 410 shows the assistant diving deeper. It read start.sh to understand the startup sequence, then grepped for RIBS_LOGLEVEL and LogLevel across the codebase. It found the configureLogLevels() function and immediately recognized the format mismatch. The reasoning shows the assistant considering two options: remove the variable or fix the format. It also noted the need to fix the startup sequence.

By message 411, the assistant had enough understanding to act. The reasoning is concise but shows the key insight: "The log level format requires component=level. Just info is invalid." This single sentence represents the synthesis of reading error logs, tracing through configuration code, understanding the parser's expectations, and formulating a fix.

Assumptions and Their Consequences

The assistant made several assumptions in this message. First, it assumed that removing the RIBS_LOGLEVEL variable (or fixing its format) would resolve the s3-proxy failure without side effects. This was reasonable—the default log level is presumably sensible—but it meant the S3 proxy would lose any custom logging configuration that might have been intended.

Second, the assistant assumed that sequential startup was sufficient to prevent migration deadlocks. This was correct for the immediate problem, but it didn't address the underlying keyspace sharing issue. The assistant implicitly assumed that the database schema and migration system could handle multiple nodes sharing a keyspace as long as they didn't migrate simultaneously. This assumption held for the test cluster but would need revisiting for production deployments.

Third, the assistant assumed that the user's inability to access port 9010 was a separate issue (possibly a browser or firewall problem) rather than something it needed to fix. The reasoning in message 409 noted "The port issue might be a firewall or browser issue, but the port is listening" and did not include a fix for it. This was a correct triage decision—not every reported symptom requires a code change.

Input Knowledge Required

To understand this message, a reader needs knowledge of several domains. First, Docker Compose configuration: understanding how environment variables are passed to containers and how service dependencies work. Second, Go configuration patterns: the envconfig library convention and the component=level logging format. Third, distributed database migration mechanics: how migration locks work, what a deadlock looks like in YugabyteDB, and why concurrent migrations fail. Fourth, shell scripting: how docker-compose up -d works and how to orchestrate sequential service startup.

The message also assumes familiarity with the broader architecture: that Kuri nodes are storage nodes that connect to YugabyteDB, that they run database migrations on startup, and that the S3 proxy is a separate stateless service that also loads RIBS configuration.

Output Knowledge Created

This message produced two concrete outputs. First, a corrected docker-compose.yml with a valid log level configuration (or the variable removed). Second, an updated start.sh that starts services sequentially: infrastructure first, then kuri-1, then kuri-2, with appropriate waits between each step.

But the message also produced intangible knowledge: a documented debugging path from error symptom to root cause. The assistant's reasoning shows how to read a multi-layer error trace, how to grep for configuration variables across a codebase, and how to distinguish between a configuration format error and a distributed systems race condition.

Conclusion

Message 411 is a masterclass in pragmatic debugging. It demonstrates how two completely unrelated bugs—a string format error in a configuration parser and a database migration race condition—can surface simultaneously in a distributed system startup. The assistant's response is minimal but effective: identify both root causes, apply the minimal fix for each, and move on. The log level fix is a one-line configuration change; the sequential startup fix is a restructuring of the boot sequence. Neither is glamorous, but both are necessary.

The deeper lesson is about the nature of distributed systems debugging. Error messages like invalid log level: info and deadlock detected appear unrelated, but they both manifest as "container won't start." The skill lies in reading the logs, tracing through the code, understanding the architecture, and applying the right fix for each. This message shows that skill in action.