Debugging a Distributed S3 Cluster: The Moment Two Failures Collide

In the life of a distributed systems engineer, few moments are as revealing as the first time you start up a new cluster and watch it fail. This message, message 579 from an extended coding session building a horizontally scalable S3-compatible storage system for the Filecoin Gateway, captures exactly that moment. The user has reported two broken endpoints—a web UI at http://127.0.0.1:9010/ returning connection resets, and an S3 API at http://localhost:8078/ returning internal server errors—and the assistant responds with a focused investigation that uncovers the root causes of both failures.

The Message in Full

The assistant's response is concise but dense with diagnostic information:

I'll investigate both issues. Let me start by checking the container logs and status.

`` $ docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | grep -E "test-cluster|NAMES" NAMES STATUS PORTS test-cluster-webui-1 Up 47 seconds 0.0.0.0:9010->9010/tcp, [::]:9010->9010/tcp test-cluster-s3-proxy-1 Up 47 seconds 0.0.0.0:8078->8078/tcp, [::]:8078->8078/tcp test-cluster-kuri-2-1 Exited (2) 53 seconds ago test-cluster-kuri-1-1 Exited (1) 47 seconds ago test-cluster-db-init-1 Exited (0) 47 seconds ago test-cluster-yugabyte-1 Up About a minute (healthy) 5433/tcp, 6... ``

`` $ docker logs test-cluster-s3-proxy-1 2>&1 | tail -50 S3 Frontend Proxy started on :8078 (node: proxy-1) Backend nodes: 2 configured 2026-01-31T00:51:43.930Z ERROR gw/s3frontend/backend s3frontend/server.go:90 Failed to lookup object {"error": "failed to lookup object: code=2200; message=Undefined Column. Column doesn't exist\nSELECT node_id FROM S3Objects WHERE bucket = ? AND key = ?\n ^^^^^^^\n (ql error -203)", "bucket": "", "key": ""} 2026-01-31T00:51:44.018Z ERROR gw/s3frontend/backend s3frontend/server.go:90 Failed to lookup object {"er... ``

`` $ docker logs test-cluster-webui-1 2>&1 | tail -20 Web UI runs on kuri-1 - access via http://localhost:9010 ``

Context and Motivation: Why This Message Was Written

This message exists because a complex, multi-layered system had been assembled over the course of dozens of prior commits and configuration changes, and it was time to see if it actually worked. The architecture under test was ambitious: a three-tier distributed S3 storage system consisting of stateless S3 frontend proxies (port 8078), multiple Kuri storage nodes (kuri-1 on port 7001, kuri-2 on port 7002), and a shared YugabyteDB database for object routing metadata. The entire system was orchestrated via Docker Compose in a test-cluster/ directory that had been carefully constructed over the preceding hours.

The user's report of two simultaneous failures—connection reset on the web UI and internal server errors on the S3 API—was the first real validation test of this infrastructure. The assistant needed to determine whether these were separate issues with independent causes, or symptoms of a deeper systemic problem. The motivation was straightforward: diagnose both failures efficiently so the cluster could be brought online.

But there was also a deeper motivation at play. The assistant had just completed a major architectural correction in the preceding segment, restructuring the system to properly separate stateless S3 frontend proxies from Kuri storage nodes with independent configurations. This debugging session was the first test of that corrected architecture. If the fundamentals were wrong—if the database schema was incorrect, if the containers couldn't start, if the proxy couldn't route requests—then the entire architectural effort would need revisiting.

The Debugging Methodology: Systematic Triage

The assistant's approach to debugging reveals a clear methodology: start with the broadest possible view, then drill into specifics. The first command, docker ps -a, provides a complete inventory of the test cluster's state. This is the distributed systems equivalent of taking vital signs before ordering specific tests.

The output is immediately revealing. The database (test-cluster-yugabyte-1) is up and healthy. The database initialization container (test-cluster-db-init-1) exited with code 0, meaning the schema migrations ran successfully. But both Kuri storage nodes have crashed—kuri-1 with exit code 1, kuri-2 with exit code 2. The S3 proxy and web UI containers are running, but that doesn't mean they're functional.

This single docker ps output tells a story: the database layer works, but the storage nodes are failing. The proxy and web UI are running but may be failing because their backends (the Kuri nodes) are down.

The assistant then checks the S3 proxy logs. The error is unmistakable: Undefined Column. Column doesn't exist\nSELECT node_id FROM S3Objects. The node_id column, which was added in migration 1754293669_init_s3_object_index.up.cql as part of commit c3fff2a, is missing from the actual database table. This is a classic schema drift problem—the migration file exists, but for some reason the column wasn't created in the running database.

Finally, the web UI log reveals a different kind of problem. The output Web UI runs on kuri-1 - access via http://localhost:9010 is not an error message—it's a placeholder. The web UI container appears to be running a stub script that prints this message and then either exits or sleeps, rather than actually serving the web application. This explains the "connection reset" error: the container is listening on port 9010 but has no real HTTP server running.

Assumptions and Blind Spots

Every debugging session carries assumptions, and this one has several worth examining.

Assumption 1: The database migration was applied correctly. The assistant assumed that because db-init exited with code 0, the S3Objects table had the node_id column. In reality, either the migration file wasn't included in the db-init script, or the migration was written to create the table without the column and the column was added later in a separate migration that wasn't applied. The error message clearly shows the column doesn't exist.

Assumption 2: The web UI container was properly configured. The assistant had previously noted that the web UI container was "just a placeholder (sleep infinity)" in the summary at message 575, yet the user's report of connection resets prompted a fresh investigation. The log output confirms the placeholder nature—the container prints a message and doesn't actually serve the UI.

Assumption 3: Both failures were independent. The assistant's approach treats the web UI and S3 proxy as separate investigations, which is correct—they are independent services with different root causes. But there's an implicit assumption that fixing each in isolation will resolve both issues, which may not hold if there's a shared dependency (like the database schema) that affects both.

Assumption 4: The Kuri nodes crashed for reasons unrelated to the S3 proxy errors. The assistant doesn't immediately investigate the Kuri node crashes (exit codes 1 and 2), focusing instead on the proxy and web UI logs. This is a reasonable triage decision—the proxy and web UI are the user-facing services—but the Kuri node failures may be the deeper cause of the S3 proxy's inability to route requests.

Input Knowledge Required

To understand this message, a reader needs knowledge spanning several domains:

Docker and container orchestration: Understanding docker ps -a output, container status codes (Exited vs Up), port mapping syntax, and the concept of container health checks.

CQL and distributed databases: Knowledge of CQL (Cassandra Query Language), schema migrations, column definitions, and error codes like ql error -203 for undefined columns.

S3 protocol and HTTP: Understanding of S3 object storage semantics, HTTP status codes (connection reset vs internal server error), and the proxy routing pattern.

The project's architecture: Familiarity with the three-layer design—S3 frontend proxy, Kuri storage nodes, YugabyteDB—and the concept of per-node keyspaces for horizontal scaling.

Go programming and logging: Understanding structured logging output with fields like {"error": "...", "bucket": "", "key": ""} and the stack trace format from Go's standard logging libraries.

Output Knowledge Created

This message generates several critical pieces of knowledge that shape the subsequent debugging effort:

The database schema is out of sync. The node_id column doesn't exist in the S3Objects table, even though the migration file was supposedly applied. This requires either re-running the migration manually or fixing the db-init process.
The web UI container is a stub. It doesn't serve the actual web application. This needs a proper Nginx reverse proxy or a direct link to the Kuri node's web interface.
Both Kuri nodes are crashing on startup. Exit codes 1 and 2 suggest different failure modes (exit code 1 is typically a general error, exit code 2 often indicates misconfiguration or missing dependencies). These need separate investigation.
The S3 proxy starts successfully but fails on the first request. The proxy's initialization succeeds (it prints "S3 Frontend Proxy started" and "Backend nodes: 2 configured"), but every request fails because the database query for object routing hits the missing column.
The system has multiple independent failure modes. The web UI failure is a configuration/container definition issue, while the S3 proxy failure is a database schema issue. They require different fixes and don't share a single root cause.

The Thinking Process Visible in the Commands

The assistant's reasoning is embedded in the choice and sequence of commands. The first command is a formatted docker ps that filters for test-cluster containers and includes status and ports. The --format flag with table template and grep filtering shows deliberate attention to readability—the assistant wants a clean summary, not the full verbose output.

The second command checks the S3 proxy logs with tail -50, showing the last 50 lines. The assistant is looking for recent errors, not the full log history. The error message is truncated with {"er... at the end, suggesting the log line was cut off by the tail output.

The third command checks the web UI logs, again with tail -20. The output is surprisingly short—just two lines. This brevity is itself a signal: the container isn't doing much, which confirms the placeholder suspicion.

Notably absent from this message is any investigation of the Kuri node logs. The assistant sees that both nodes have exited but doesn't immediately check their logs. This is a deliberate triage decision: the user reported two specific failures (web UI and S3 API), and the assistant is addressing those directly. The Kuri node crashes are secondary—they may be causing the S3 proxy errors, but the proxy's own logs reveal a more immediate issue (missing column) that would persist even if the Kuri nodes were running.

Conclusion: The Diagnostic Pivot Point

Message 579 represents a diagnostic pivot point in the coding session. Before this message, the assistant had been building and assembling—creating files, writing migrations, configuring containers. After this message, the focus shifts to debugging and fixing. The three commands in this message transform the problem from "the system doesn't work" to two specific, actionable problems: "the S3Objects table is missing the node_id column" and "the web UI container is a placeholder."

The message also reveals the nature of distributed systems debugging: failures cascade and compound. The S3 proxy's internal server error is not a bug in the proxy code—it's a consequence of a database schema that wasn't properly applied. The web UI's connection reset is not a network issue—it's a container that was never configured to serve a real application. Both failures trace back to gaps between the intended configuration and the actual deployed state.

For the engineer reading this message, the lesson is clear: when a complex distributed system fails on first startup, look first at the boundaries between components. The database schema boundary (migration files vs actual tables) and the container configuration boundary (Dockerfile vs running process) are where the assistant found both failures. These are the seams where assumptions break down and where the difference between "this should work" and "this does work" becomes visible.